Methodology
Exact definitions for every metric shown on the leaderboard. The goal is a transparent, reproducible benchmark that any operator can independently verify.
Overview
SuiScope is a continuous, neutral benchmarking platform for public Sui infrastructure endpoints. Probes run as stateless agents deployed across multiple geographic regions and fire on a fixed 60-second schedule. Every probe cycle produces one measurement event per provider per endpoint type; results are stored in ClickHouse and exposed through a read-only public API.
SuiScope measures what a client actually experiences: the round-trip cost of a cold TCP connection, TLS handshake, request encoding, network transit, server processing, and time to first response byte. It does not simulate internal server behaviour or measure capacity under load.
Probe cycle
Each probe cycle for a given provider and endpoint type follows these steps in strict order:
- 1DNS resolution — the target hostname is resolved once before the timing window opens. DNS time is excluded from all latency measurements.
- 2Open a new TCP connection — no connection pooling, no keep-alive reuse. Each probe cycle establishes a fresh socket from scratch.
- 3TLS handshake — for TLS endpoints the full handshake is included in the latency measurement.
- 4Send request — for gRPC endpoints a GetServiceInfo unary call (sui.rpc.v2.LedgerService); for GraphQL endpoints an HTTP POST with query { checkpoint { sequenceNumber } }.
- 5Record time-to-first-response-byte (TTFB) — the timer starts at the moment the first byte of the request is sent over the wire and stops when the first byte of the response is received. This is latency_ms.
- 6Parse response — extract the provider's latest checkpoint sequence number.
- 7Fetch chain head — a separate cold-connection call to a trusted Mysten Labs reference endpoint (fullnode.mainnet.sui.io) retrieves the current chain head checkpoint. This call is made inside the same probe cycle and is not cached across cycles.
- 8Compute freshness — freshness_checkpoints = chain_head − provider_latest_checkpoint. A value of 0 means the provider is at the head of the chain.
- 9Record outcome — success or failure, with a structured error code if failed.
- 10Close connection — the socket is closed. No state is retained between probe cycles.
Probe timing
| Measurement | Interval | Derivation |
|---|---|---|
| Latency + freshness | 60 s | Direct measurement per cycle |
| Error rate | Derived | Failed / total over 5-minute rolling window |
| Uptime | Derived | Successful / total over 1-hour rolling window |
Endpoint types
Each provider may expose gRPC, GraphQL, and/or Archival endpoints. Probes run independently for each endpoint type. If a provider exposes multiple types, each appears as a separate row in the leaderboard filtered by endpoint type.
Archival endpoints expose the same LedgerService gRPC API as full nodes but serve historical data beyond the full-node retention window. See the Archival probe section for the specific probe mechanics.
Archival probe
Archival nodes serve historical Sui data beyond what standard full nodes retain. The Sui Archival Service exposes the same sui.rpc.v2.LedgerService gRPC API as a full node. What differs is the probe intent: instead of checking freshness, the archival probe verifies that the node can actually serve deep historical data.
What is probed
- 1DNS resolution — excluded from latency, same as gRPC/GraphQL probes.
- 2Open a new cold TCP+TLS connection to the archival endpoint.
- 3Compute a target checkpoint: chain_head − 2 592 000 (approximately 30 days behind the current chain head, at ~1 checkpoint per second).
- 4Call GetCheckpoint with the target sequence number.
- 5Record latency_ms (TTFB) and success. A NOT_FOUND response is a failure — the node lacks the requested depth.
- 6Close the connection.
Why 30 days
The 30-day threshold (2\u202f592\u202f000 checkpoints) is well beyond any known full-node retention window. A successful response at this depth confirms that the node is not merely a full node masquerading as an archival service.
What is not measured for archival
Freshness (freshness_checkpoints) is intentionally not measured for archival endpoints. Archival nodes are designed to lag the chain head — reporting a large freshness gap would be misleading rather than informative.
Probe target is deterministic
The target checkpoint is always chain_head − 2\u202f592\u202f000, computed fresh each cycle. There is no per-cycle randomness. Caching at the archival service layer is considered unlikely given the Bigtable-backed architecture of the reference implementation.
Metric definitions
These definitions are canonical. The dashboard, API, and probes all use exactly these semantics. Any deviation requires an Architecture Decision Record.
| Metric | Definition | Unit |
|---|---|---|
| latency_ms | Cold TCP connect + TLS handshake + request write + time to first response byte. DNS pre-resolved and excluded. | milliseconds |
| freshness_checkpoints | chain_head_checkpoint − provider_latest_checkpoint. Integer lag; 0 means at chain head. Lower is better. | checkpoints |
| error_rate | Failed probes ÷ total probes over a 5-minute rolling window. A probe fails if no valid response is received within 10 seconds. | ratio (0–1) |
| uptime | Successful probes ÷ total probes over a 1-hour rolling window. | ratio (0–1) |
Tier thresholds
Each metric value is classified into a display tier for the leaderboard. Thresholds reflect real-world client expectations for production Sui infrastructure.
| Metric | Good | Degraded | Poor |
|---|---|---|---|
| latency_ms | < 100 ms | 100 – 300 ms | ≥ 300 ms |
| freshness_checkpoints | ≤ 2 | 3 – 10 | > 10 |
| uptime | ≥ 99.5 % | 98 – 99.5 % | < 98 % |
| error_rate | < 0.5 % | 0.5 – 5 % | ≥ 5 % |
Anti-gaming approach
A benchmarking platform is only useful if its results cannot be artificially inflated. The following design choices make it structurally difficult for a provider to perform well on SuiScope without actually providing a fast, reliable service to real clients.
Cold TCP connections
Every probe opens a fresh TCP socket. There is no connection reuse, no keep-alive, and no HTTP/2 or gRPC multiplexing across cycles. This means a provider cannot benefit from a warm connection pool — each measurement reflects the full connection-establishment cost a cold client would pay.
Opaque probe timing
Probe cycles fire on a fixed 60-second schedule, but the exact wall-clock start time within that window is not published. A provider that attempted to pre-warm connections or cache responses for expected probe traffic would need to do so continuously — indistinguishable from simply running a well-maintained service.
Transparent User-Agent
All probe requests carry the header User-Agent: SuiScope-Probe/<version>. Providers can see probe traffic in their own logs. This is intentional: SuiScope is a neutral observer, not a secret auditor. However, a provider that routes SuiScope traffic to faster infrastructure must apply those same optimisations globally — there is no practical way to identify only probe requests in production traffic without also improving service for real clients.
DNS excluded; network path not controlled
DNS resolution is excluded from latency to avoid penalising providers for slow DNS propagation. However, the network path from each probe region to the provider endpoint is not controlled or disclosed. Providers cannot selectively optimise routing for probe source IPs without deploying equivalent improvements for all traffic from those regions.
Independent chain-head reference
Freshness is calculated against the chain head observed from Mysten Labs’ reference endpoint (fullnode.mainnet.sui.io) within the same probe cycle. This reference is fetched via its own cold connection and is not shared across providers or cached. A provider cannot influence its freshness score by manipulating the reference point.
Public methodology
This page is the authoritative specification. Any change to a measurement definition requires an Architecture Decision Record in the public repository. Providers who believe a measurement is incorrect can file an issue or submit a PR — the methodology is open to review.
Important
SuiScope measures publicly accessible endpoints only. Providers with private or auth-gated infrastructure are not included in the leaderboard unless they expose a freely accessible public endpoint.
Public endpoints
SuiScope benchmarks both public and private endpoints. Public endpoints are freely accessible — no API keys or authentication required. Because they are open to the internet, providers apply standard rate limiting and request throttling to protect their infrastructure.
Probe traffic is deliberately low-volume: one request per provider per endpoint type every 60 seconds. This is well within the fair-use limits of any public node. If you use the same endpoints in your own application, be aware that heavier usage patterns may trigger provider-side rate limiting that is not reflected in SuiScope measurements.
Rate limiting notice
Public endpoints are subject to the rate-limiting policies of each respective provider. SuiScope does not control or publish individual rate limits — consult the provider’s documentation for details. Private endpoints are auth-gated and not subject to the same public throttling constraints.
What is not measured
Understanding the limits of SuiScope is as important as understanding what it measures.
- –Throughput or capacity — SuiScope fires one probe per provider per 60 seconds. It does not load-test endpoints.
- –Geographic fairness — probe regions are fixed. A provider optimised for US-East will score better from the iad region than from ap-southeast. Filter by region to compare on a level footing.
- –Internal server performance — SuiScope observes external behaviour only. It cannot distinguish between a slow network path and a slow server.
- –Transaction submission success — probes use read-only queries. Write-path reliability (transaction throughput, finality latency) is not measured.
- –Stream stability (Phase 2) — gRPC subscription metrics (stream_uptime_pct, stream_checkpoint_gap, stream_disconnects_per_hour) are defined but not yet deployed. They will be added in a future milestone.
- –Composite scores — no weighted aggregate score is published. Raw metrics only, sortable by the user. See ADR-004.