Guide

Serving systems

Replica scoring

Beat round robin with an explicit score from prefix affinity, queue depth, KV headroom and adapter locality. Hard filters first, then pick the max.

105 of 138 · 3 min read

You do not need ML to beat round robin. Collect a few signals from each engine and pick the replica with the best explicit score. Tune the weights against a benchmark that includes shared prefixes.

SignalUse
Queue depth / waiting requestsAvoid replicas already backing up
GPU KV utilization / free blocksPrefer replicas with room for this prompt's KV
Prefix cache scoreHash prompt prefix; prefer replicas likely to hit
Active tokens / running seqsRough load beyond request count
LoRA adapters loadedPin adapter-sticky traffic
Recent TTFT/TPOTSoft feedback when metrics lag
Model ID / capabilityRoute to the right pool

Walking through prefix affinity and KV headroom

Prefix affinity. Hash the first N tokens (or a stable system-prompt ID your app sends as a header). Route to the replica that owns that hash bucket, with fallback if that replica is saturated. N should cover the shared template but not the unique user tail — otherwise every request looks unique and affinity buys nothing.

KV headroom. Even with a perfect prefix match, a replica at 95% KV utilization cannot admit a 32k prefill. Combine affinity with a reservation estimate: kv_free >= kv_needed(prompt_len) (plus safety margin). If no replica fits, queue at the gateway with a deadline or reject with 429/503. Do not randomly park a bomb on the fullest GPU.

score(replica) =
    W_aff * prefix_affinity(replica, request)   # 0..1 estimated hit
  - W_q   * normalized_queue_depth(replica)
  - W_kv  * kv_pressure(replica, request)       # 0 if fits with margin, else large
  - W_lora * adapter_miss_penalty(replica)      # 0 if adapter hot

Pick the max score among replicas that pass hard filters (right model, not cordoned, KV fits). If all fail hard filters, queue or shed — do not assign anyway. Log the winner and the top features so you can debug "why did this go to B?" without guessing.

Example

W_aff=10, W_q=3, W_kv=100 (effectively a gate), W_lora=5. Replica A has affinity 0.9 and queue 0.2; B has affinity 0.1 and queue 0.0. Scores: A ≈ 9.4, B ≈ 1.0 → A wins. When A's KV cannot fit a 32k prompt, W_kv forces B even with a cold prefix — correct under memory constraint. After a week, plot TTFT vs affinity bucket. If cold routes are rare but expensive, increase warm replica count for that hash rather than raising W_aff blindly.

Scrape intervals matter. A 30 s Prometheus scrape is slow relative to GPU batches measured in milliseconds. Prefer push or high-frequency sidecars for KV and queue depth used in live routing. Use slower scrapes for dashboards.

If gateway queue time rises while engines look idle, the picker is wrong or metrics are stale. If engines are pegged and rejects rise, you need capacity or shedding — not a smarter hash.

The big idea

A hospital triage board does not send the next patient to whichever room has seen fewer patients today. It sends them to a free room that already has the right kit. The picture breaks when the board is 30 seconds stale: then you walk people into full rooms. Live KV and queue signals have to be fresher than a dashboard scrape.

Common mistakes

  • Hashing the whole prompt, so every user tail looks unique and affinity is noise.
  • Scoring without a hard KV filter, so the "best" replica OOMs.
  • Routing from 30-second scrapes and wondering why the picker disagrees with reality.

Check yourself

  1. Replica A scores 9.4 and cannot fit the prompt. What should happen? Answer: the KV hard filter (or W_kv) must eliminate A; pick a replica that fits or shed.
  2. How should you choose N for a prefix hash? Answer: long enough to cover the shared template, short enough to ignore the unique user tail.

Key takeaways

  • Score affinity minus queue, KV pressure and adapter miss.
  • Hard-filter model, health and KV fit before the score.
  • Hash the shared prefix, not the whole prompt.
  • Live routing needs fresher KV and queue signals than a 30 s scrape.

Further learning

Next, put token-based rate limits in front of the picker.

Routing, load balancing and scheduling

LLM-aware gateways: KV-cache-aware routing, token-based rate limits, priorities and fairness.

Cache-aware routing

Prefix cache lives on one replica. A load balancer that ignores cache state sprays requests and destroys hit rate. Send each request toward the replica that already holds its prefix.

Benchmarking inference

Measuring honestly: workloads, load generators, concurrency sweeps and the mistakes that make numbers meaningless.