LLM Gateway & Routing
Why LLM systems need a centralized operational layer, and how routing, reliability, and observability work inside one
LLM Gateway & Routing
TL;DR
LLM-backed systems face operational problems a single traditional model deployment doesn't: multiple providers with different pricing and reliability, highly variable per-request cost, and streaming responses. A gateway centralizes routing, reliability (rate limiting, retries, circuit breaking), and observability (cost, latency, provider) in one place, instead of every calling service reinventing them badly.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~20 minutes |
| Prerequisites | A/B Testing & Experimentation |
| You will understand | Why LLM systems need a gateway, and how routing, reliability, and observability work inside one |
Why LLM Systems Need This and a Single Model Deployment Didn't
A traditional ML deployment serves one model, with one latency profile and one, roughly predictable, cost per request. An LLM-backed system typically calls out to several models across several providers, each with its own pricing, rate limits, latency characteristics, and uptime — and the "right" model for a given request can change based on task difficulty, cost budget, or which provider happens to be degraded right now.
Without a gateway, this logic gets duplicated, inconsistently, across every service that calls an LLM. One service retries on failure, another doesn't. One tracks cost, another doesn't notice its spend until the bill arrives. A gateway exists to make this logic exist exactly once, correctly, for everything built on top of it.
What a gateway centralizes
Routing
Reliability
Observability
This page is the direct setup for this track's Project Ideas brief — building a modest version of exactly this system is the fastest way to feel why each layer earns its place.
Routing
Rule-based routing
The simplest router picks a model based on the request's declared task or required capability — a request tagged code-generation goes to a model known to be strong at code; a request needing a 100K-token context goes only to models that support it.
Cost-based routing
A more active strategy: route to the cheapest model that clears a quality bar for this request type, escalating only when needed.
def route(request: LLMRequest) -> str:
if request.requires_tool_use and not CHEAP_MODEL_SUPPORTS_TOOLS:
return "expensive-model"
if request.estimated_complexity == "low":
return "cheap-model"
if request.estimated_complexity == "high":
return "expensive-model"
return "cheap-model" # default to cheap; escalate on failure/low-confidence, not by defaultThis is the same escalation idea as the Small Language Models track's routing pattern, generalized: a small/cheap model by default, escalation for the minority of requests that genuinely need more. A gateway is where that pattern gets implemented once, for every calling service, instead of copy-pasted into each one.
Fallback chains
Routing isn't just about picking the best option — it's about having a next option when the first one fails.
FALLBACK_CHAIN = ["primary-provider", "secondary-provider", "tertiary-provider"]
async def call_with_fallback(request: LLMRequest):
for provider in FALLBACK_CHAIN:
try:
return await call_provider(provider, request)
except ProviderError:
continue
raise AllProvidersFailedError(request)A gateway with one provider and no tested fallback path is barely different from calling that provider directly. The value is specifically in what happens when the primary route fails or rate-limits — and that path needs to be tested deliberately (e.g. simulating a provider outage), not assumed to work because the code exists.
This is the practical operational counterpart to Inference Providers from the HuggingFace track — a routed, multi-provider API is exactly the kind of upstream a gateway is built to sit in front of, adding the reliability and cost-control layer a raw API client doesn't give for free.
Reliability
Rate limiting
Every provider imposes limits — requests per minute, tokens per minute, concurrent requests. A gateway enforces these before a request leaves your system, rather than discovering the limit from a provider's error response after the fact, which wastes the request and risks penalty for repeated violations.
Retries with backoff
async def call_with_retry(provider: str, request: LLMRequest, max_attempts: int = 3):
for attempt in range(max_attempts):
try:
return await call_provider(provider, request)
except TransientError:
if attempt == max_attempts - 1:
raise
await asyncio.sleep(2 ** attempt) # 1s, 2s, 4sExponential backoff spaces retries out increasingly, so a brief blip gets retried almost immediately while a sustained problem doesn't hammer an already-struggling provider with rapid-fire retries.
Circuit breaking
A circuit breaker's states
Closed
Normal operation — requests flow through to the provider
N consecutive failures
Threshold crossed
Open
Requests are failed immediately (or routed elsewhere) without calling the failing provider, for a cooldown period
Half-open, after cooldown
Allow a small number of test requests through
A circuit breaker protects both sides. It stops your system from wasting time and money on calls to a provider that's already failing, and it stops piling additional load onto a provider that's already struggling — retrying aggressively against a degraded provider can make its outage worse, for everyone calling it, not just you.
Observability
Every other layer in this page depends on this one existing first: log cost, latency, and provider/model for every request, from day one.
log_request(
provider=chosen_provider,
model=chosen_model,
input_tokens=usage.input_tokens,
output_tokens=usage.output_tokens,
cost_usd=compute_cost(usage, chosen_model),
latency_ms=elapsed_ms,
cache_hit=was_cached,
)Without this, you cannot answer the question a gateway exists to answer: did routing, caching, or the fallback chain actually help? A cost-based router that isn't measurably cheaper than always calling the expensive model is a router that's adding complexity for nothing.
Concept Checks
Check yourself
Why does duplicated per-service LLM-calling logic (retries, cost tracking) tend to go wrong, compared to centralizing it in a gateway?
Because each service implements it independently and inconsistently — one retries on failure and another doesn't, one tracks cost and another only notices at the bill. A gateway makes the logic exist exactly once, correctly, so every consumer gets the same reliability and visibility guarantees rather than whatever each team happened to build.
Why is a fallback chain that's never been tested against a simulated provider outage not actually providing the reliability it appears to?
Because the code existing doesn't guarantee the failure path works correctly — error handling code is exactly the kind of code that's rarely exercised in normal operation and often has bugs that only surface when it actually needs to run. Without deliberately simulating a primary-provider failure, you don't know whether the fallback chain activates correctly until a real outage tests it for you, at the worst possible time.
Why does exponential backoff matter for retries against a struggling provider, rather than retrying immediately every time?
Because immediate, rapid retries against a provider that's already failing add more load to a system that's already struggling, potentially making the outage worse for everyone calling it. Spacing retries out increasingly (1s, 2s, 4s...) lets a brief blip recover almost immediately while a sustained problem doesn't get hammered by repeated rapid-fire requests.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Why a gateway exists | LLM systems need per-request routing, reliability, and cost tracking a single-model deployment didn't |
| Rule-based routing | Route by declared task or required capability |
| Cost-based routing | Cheapest model that clears a quality bar, escalate on need — the SLM escalation pattern, generalized |
| Fallback chains | Must be tested deliberately, not assumed to work because the code exists |
| Rate limiting | Enforced before a request leaves your system, not discovered from a provider error |
| Exponential backoff | Spaces retries out so a blip recovers fast without hammering a struggling provider |
| Circuit breaker | Stops calling a failing provider for a cooldown period, protecting both sides |
| Observability first | Cost/latency/provider logging is the foundation everything else is measured against |
Next
Routing an LLM request correctly depends on a gateway having good information. The next page covers a related but distinct problem for traditional ML: making sure a model sees the same data at training time and serving time. Feature Stores & Data Pipelines.
A/B Testing & Experimentation
Why deploying a new model isn't proof it's better, traffic splitting mechanics, statistical significance, and when a bandit beats a fixed split
Feature Stores & Data Pipelines
Training-serving skew explained in full, and the online/offline feature store architecture that exists specifically to prevent it