Serving systems
Routing, load balancing and scheduling
LLM-aware gateways: KV-cache-aware routing, token-based rate limits, priorities and fairness.
Classic load balancing assumes cheap, short, mostly uniform requests. LLM inference is none of those. The gateway should enforce token budgets, priorities and cancellation, then pick a replica from prefix affinity, KV headroom, queue depth and adapter locality. See why round robin fails for the 100× cost gap and the prefix-cache miss that turns into a multi-second prefill.
Engine-side scheduling (continuous batching, chunked prefill) optimizes a single replica's GPU. Gateway-side routing optimizes the fleet. You need both. Putting all policy in the engine does not help when the wrong replica received the request.
In incidents, check reject rate and queue time before assuming the model "got slower." Often the router went dumb after a metrics scrape break. If gateway queue time rises while engines look idle, the picker is wrong or metrics are stale. If engines are pegged and rejects rise, you need capacity or shedding — not a smarter hash.
Concepts in this section
| Step | Concept | What it answers |
|---|---|---|
| 1 | Why round robin fails | Why equal request counts do not equal equal GPU work? |
| 2 | Replica scoring | Which signals and weights beat random? |
| 3 | Token-based rate limits | How do you budget tokens, priorities and cancels? |
| 4 | Admission control | What do you shed when the fleet is full? |
Prefix affinity in more depth lives in cache-aware routing. Kubernetes is converging on Gateway API Inference Extension and llm-d so an Inference Gateway can call an Endpoint Picker using model-server metrics, not only kube readiness. Treat those projects as the signal contract even if you are not ready to adopt the whole stack.
Minimum practical path: expose KV utilization, waiting queue and prefix hit rate; pick affinity_bonus - queue_penalty - kv_pressure; hard admission limits per pool; dashboards that show hit rate and reject rate next to GPU duty cycle.
Map model names to pools (8B vs 70B vs embeddings) and pin version aliases so a tag move cannot fragment caches. KV-aware routing rarely crosses regions — the cache is not there. Replicate configuration and adapter storage, not live KV.
Common mistakes
- Request-count balancing on a fleet whose prompts differ by 100× in GPU-seconds.
- Sticky sessions by client IP instead of prefix hash or adapter id.
- RPM limits only, so one 100k-token prompt equals a thousand short chats.
- Retries without jitter after a 503, which oscillates a full fleet.
- A 30 s Prometheus scrape as the live routing signal.
Key takeaways
- LLM requests are unequal, cache-local and long-lived. Round robin optimizes the wrong thing.
- Route on prefix affinity, KV headroom, queue depth and adapter locality, then fall back when saturated.
- Rate-limit tokens, not just requests. Enforce priorities, fairness and cancellation at the gateway.
- Admit with a queue-time budget. Shed load deliberately instead of OOMing replicas.
- Prefer standardized inference gateway patterns as the fleet grows.
Further learning
llm-dAchieve state of the art inference performance with modern accelerators on Kubernetes - llm-d/llm-dGitHubRelated
Why round robin fails for LLMs
Two HTTP requests can differ by 100× in GPU-seconds. Equal request counts do not equal equal load when cost, cache and streams are uneven.
Load balancing and proxies
How a load balancer chooses which server gets each request, and how a badly set up balancer can cause the outage it was meant to prevent.
Prefix caching
Reusing KV across requests that share prompts. Block hashing, RadixAttention, cache-aware routing, and why cached input tokens are billed cheaply.