Why round robin fails for LLMs

Round-robin across LLM replicas wasted more money than almost any other "simple" default in early GenAI platforms. Classic load balancing assumes cheap, short, mostly uniform requests. LLM inference is none of those.

Two requests that look identical at the HTTP layer can differ by 100× in GPU-seconds. A request that would have been a free prefix cache hit on replica A becomes a multi-second prefill on replica B. A single long generation pins a batch slot while short chats pile up.

Why the usual balancer breaks

• Variable cost. Prefill FLOPs scale with input length. Decode time scales with output length and batch contention. Balancing request count does not balance GPU work.
• Prefix locality. KV caches are per replica (or per disaggregated pool). Sticky routing by prefix hash raises hit rate. Random routing destroys it.
• Long-lived streams. A chat completion may stream for minutes. Connection-level balancing done at accept time cannot rebalance mid-stream. Endpoint choice at request start dominates.
• Adapter locality. Multi-LoRA replicas only keep a subset of adapters hot in GPU memory. Routing a cold adapter everywhere causes thrash.

TCP or HTTP session stickiness is not enough. One user session may issue many completions with different models and adapters. Each request needs a fresh routing decision. Stickiness helps only when you key it on the right affinity (prefix hash, adapter id), not on client IP. Agents that fan out parallel tool calls should send a shared routing key so sibling calls land where the conversation prefix is warm.

The big idea

A post office that gives every clerk the same number of envelopes still fails if one envelope is a piano. The picture breaks when you want the piano on a special clerk: prefix affinity is that clerk, and you still need a fallback when their dock is full.

Walking through a shared system prompt

Fleet of 10 replicas, shared 2k-token system prompt, unique 500-token user turns. Random routing: each replica sees ~10% of repeats → low hit rate after cache churn. Prefix-hash routing to 2 primary replicas for that system prompt: those two stay warm. TTFT for steady traffic drops from ~800 ms to ~200 ms on hits, and the other eight absorb cold or different templates. Under a spike, the gateway overflows to secondaries on purpose and accepts colder TTFT rather than OOM.

See cache-aware routing for affinity strategies.

Common mistakes

• Balancing connections and calling the GPU fleet "even."
• Sticky sessions by client IP. Agents and NATs smash that key.
• Affinity without overflow. The hot replica OOMs while cold ones sit idle.

Check yourself

1. Why can two /v1/chat/completions calls cost 100× different GPU-seconds? Answer: prompt length, output length, cache hit vs miss, and adapter locality.
2. What should overflow do when the warm replica is full? Answer: send the request to a colder replica on purpose, accepting worse TTFT rather than OOM.

Key takeaways

• Request count is the wrong load unit for LLMs.
• Cache, KV headroom and adapters are per replica, so routing has to see them.
• Pick the endpoint at request start. Streams are too long to rebalance later.
• Overflow to a cold replica beats pinning a bomb on a full GPU.

Further learning

Next, turn those signals into a replica score.