A managed LLM gateway adds a hop between your code and the provider. The honest question is: how much does that hop cost in latency and reliability? This page is our methodology for measuring it — and a commitment to publish real numbers, not a fabricated leaderboard.
What this page measures
Read this first. The latency figures below are SLA targets we hold ourselves to — not the results of a published study. A full reproducible gateway-latency report is on the roadmap. We would rather state that plainly than print numbers we cannot defend. For live, real-time health see the status page.
Gateway time on top of the provider call
The slow tail we hold ourselves to
Contractual — see the legal SLA
Switch models with a string, no re-integration
Targets, not measurements. p50 mirrors SLA_LATENCY_MS from the live stats endpoint; p95 is the tail bound we commit to.
Routing a request does not change the model's answer. So NemoRouter benchmarks the things routing actually affects: added latency, failover, and throughput. Model quality stays where it belongs — with the model provider and independent evaluators.
The single number that is genuinely ours: how many milliseconds NemoRouter adds on top of the provider round-trip. Auth, credit reserve, guardrail evaluation, routing, and the cost-header read all happen here.
When a provider 5xxs, times out, or rate-limits us, the routing engine retries the next model in the fallback chain. We measure how often that happens and how much latency the retry adds.
Sustained requests-per-minute and tokens-per-minute the gateway holds before queueing. This is the number behind the per-tier RPM/TPM guarantees on the pricing page.
Model quality — MMLU, HumanEval, GPQA, reasoning scores — is the model provider’s to measure and publish. NemoRouter does not run a model leaderboard, because routing a request does not change the model’s answer.
Every request flows Client → Frontend → Nemo Backend → in-process routing engine → provider. Everything before the provider is overhead we own. We break it into stages so the number is auditable, not a black box.
Overhead budget
A pass-through request — no guardrails enabled — is the floor. Each capability you switch on adds a known, measured increment. Nothing is hidden inside an average.
Where the overhead budget goes (illustrative target split)
The routing engine runs in-process inside Nemo Backend — there is no separate network service to add a second hop. Browse the live catalog on the models page.
A single provider has bad minutes — a 503, a timeout, a regional blip. The value of a gateway is that one bad provider does not become your bad request. We measure how often failover fires and what it costs.
Failover behavior
When a model call fails, the routing engine moves to the next model in the fallback chain. Circuit breakers keep a struggling provider out of rotation until it recovers. The benchmark question is how much latency a failover hop adds — and we report it separately, never blended into the happy-path p50.
Failover invariants
A benchmark is only worth printing if someone else can reproduce it. Every study we publish follows the same four rules — and ships the config so you can run it yourself.
Run against the zero-cost mock provider so provider latency is constant. Whatever moves is gateway overhead — nothing else.
Discard cold-start requests, then collect a large fixed sample at steady state. Cold starts are reported separately, never blended into p50.
Publish p50, p95, and p99 — never just the average. A gateway that looks fast at p50 and stalls at p99 is a slow gateway.
Every run ships its config: region, instance size, guardrails enabled, concurrency, sample size, and date. A benchmark you cannot reproduce is marketing.
The honest version
Many gateways publish a single average latency number with no config, no tail percentiles, and no way to reproduce it. We treat that as marketing, not measurement. Until our reproducible study is published, the figures on this page are stated as SLA targets — the bar we commit to — and the live status page shows real component health right now.
The 99.9% uptime SLA is a contractual commitment in the legal SLA — not a figure we picked because it looked good on a slide.
Uptime SLA
The gateway runs on managed Cloud Run autoscaling and managed Supabase Postgres. Component health — gateway, database, settlement queue — is published live, and incident history is public rather than buried behind a support login.
Published component health
See real-time component health on the status page, and the full availability terms in the legal SLA.
When the first reproducible study lands, here is exactly what ships with it — so there is no ambiguity about what was tested or how.
Each study is timestamped and versioned. Older reports stay live so you can see the trend, not just today’s headline.
Region, instance size, concurrency, guardrails enabled, sample size, and provider mix — everything needed to reproduce the run.
Every latency claim ships all three percentiles. A single average is not a benchmark; it is a hope.
Reliability and speed are reported together. A gateway that is fast only when nothing fails is not actually fast.
Want the methodology in detail, or to run a benchmark against your own workload before committing? Email sales@nemorouter.ai.
Measure it yourself
Point a real workload at NemoRouter and watch the overhead in your own dashboard. No fabricated leaderboard to trust — just the gateway, your requests, and the numbers.
Live component health is always on the status page.