A coding tool mixes fast autocomplete with deep reasoning. Nemo Router lets you pick the model per task from one catalog, route latency-sensitive calls to the quickest endpoint, and cap spend per developer seat.
One catalog, the right model per task
models on Google Vertex AI
p50 — LLM inference dominates
One key per developer, capped
Model, latency, and cost per request
A coding assistant has two competing needs — speed for completion, depth for reasoning — and cost that scales with every seat. Nemo Router handles all three behind one key.
Inline autocomplete wants a fast model; a multi-file refactor or a test-suite generation wants a strong reasoning model. The catalog exposes every model behind one key — set it per request.
A developer is waiting on every keystroke. Latency-based routing steers each completion request to the quickest healthy endpoint; least-busy avoids the most loaded deployment.
A coding assistant scales cost with headcount. Issue one virtual key per developer or per team and set a per-key budget — spend is visible and capped per seat.
When a suggestion is wrong or slow, you want the request. Every completion lands in the log with the model, latency, token counts, and real cost.
Each developer carries a seat key. Completion and reasoning requests route to the model the task needs, stream back token-by-token, and land in the log attributed to that seat.
Code-assistant request flow
Editor request
completion or chat
A keystroke completion or a refactor prompt from the IDE.
Seat key
sk-nemo-... · per developer
Budget and rate limit scope to the developer’s key.
Model + latency route
catalog · least-busy
Fast model for completion, strong model for reasoning.
Stream back
token-by-token
Streaming proxied transparently to the editor.
Logged per seat
request log
Model, latency, tokens, cost — attributed to the seat.
The gateway adds about 95 ms at p50 — LLM inference is the dominant latency factor. Streaming is proxied with no hot-path buffering.
Per-seat budgets
A coding assistant’s bill grows with every engineer you add. Give each developer their own virtual key with a per-key budget. The request log, rate limits, and spend all scope to that key, so cost-per-seat is a number, not a guess — and a 402 is the ceiling, not an overage.
Spend per seat
A coding assistant just sets the model field per call — fast for completion, strong for reasoning. These snippets come from the same SDK examples the playground uses; change the model string and the catalog does the rest.
pip install openai| 1 | # Cache: enabled (org default). Pass nemo_cache: false to skip. |
| 2 | from openai import OpenAI |
| 3 | import os |
| 4 | |
| 5 | client = OpenAI( |
| 6 | api_key=os.environ["NEMOROUTER_API_KEY"], |
| 7 | base_url="https://api.nemorouter.ai/v1", |
| 8 | ) |
| 9 | |
| 10 | response = client.chat.completions.create( |
| 11 | model="gemini-2.5-flash", |
| 12 | temperature=1, |
| 13 | max_tokens=1024, |
| 14 | top_p=1, |
| 15 | messages=[ |
| 16 | {"role": "user", "content": "Hello! What models do you support?"}, |
| 17 | ], |
| 18 | extra_body={ |
| 19 | # "nemo_cache": False, # Uncomment to skip cache |
| 20 | }, |
| 21 | ) |
| 22 | |
| 23 | print(response.choices[0].message.content) |
One key reaches every model in the catalog — no per-model provider account to manage.
Model choice, low latency, per-seat budgets
Pick the model per task, route for latency, and cap spend per developer — all unlocked on every plan.