Guide

Core optimizations

Cache-aware routing

Prefix cache lives on one replica. A load balancer that ignores cache state sprays requests and destroys hit rate. Send each request toward the replica that already holds its prefix.

73 of 138 · 2 min read

Prefix cache is per replica, or per cache shard. A load balancer that ignores cache state sprays requests across pods and destroys hit rate. Each replica sees a random subset of prefixes and thrash-evicts.

Cache-aware routing sends a request toward a replica that already holds its prefix, or a long matching trunk. Without affinity, you can disable prefix caching and barely notice. With affinity, TTFT histograms change shape.

Concept

Affinity means a request prefers the replica that already has its state. Here the state is cached KV for a prefix or session, not a user session cookie alone.

The big idea

Think of a library with several branches. If you return to a random branch, your reserved books are never there. If you return to the same branch, pickup is instant.

The picture breaks when one branch is flooded. Then you need a fallback, not a longer queue.

Walking through the strategies

Strategies include:

  • Hash the system-prompt id or session id to a sticky replica.
  • Query a control plane for "who has this prefix hash."
  • Soft affinity: prefer the sticky replica but overflow when overloaded.

This ties directly to routing and scheduling. A fleet-wide average hit rate can hide one sticky replica with 90% hits and five cold replicas with 5%. Dashboard per pod.

Warning

Sticky sessions without a drain plan create hotspots. Combine affinity with admission limits and a fallback to cold prefill rather than queueing forever on one popular pod.

Example

10M requests/day, a 2,000-token static prefix, 10% hit rate: you still recompute 90% of prefixes. Move hit rate to 80% via sticky routing and an immutable system prompt: you eliminate prefill for 0.7 × 10M × 2000 tokens/day. At fleet scale that is whole GPUs you do not need to buy.

The cost model to track:

effective_prefill_tokens = new_tokens + (miss_rate × reusable_prefix_tokens)
hit_rate = cached_prompt_tokens / total_prompt_tokens

Raise hit rate by stabilizing system prompts, putting static content first, and keeping multi-turn history on the same replica.

Common mistakes

  • Round-robin in front of a prefix cache. Hit rate collapses and the cache looks useless.
  • Sticky routing with no overflow. One popular prefix pins one pod.
  • Trusting fleet-average hit rate. Per-replica charts show whether affinity is working.

Check yourself

  1. You enable APC and keep round-robin across 8 replicas. Why is hit rate still low? Answer: each replica sees a random eighth of prefixes and evicts them. The next request for the same prefix lands elsewhere.
  2. What is the safe fallback when the sticky replica is overloaded? Answer: send the request to another replica and pay cold prefill, rather than queueing forever.

Key takeaways

  • Prefix cache is per replica. Routing must know that.
  • Hash or look up the prefix, prefer that replica, overflow when it is full.
  • Measure hit rate per replica, not only fleet-wide.
  • Affinity without a drain plan creates hotspots.

Further learning

See routing and scheduling for the gateway side. Next, multi-tier prefix cache.

Prefix caching

Reusing KV across requests that share prompts. Block hashing, RadixAttention, cache-aware routing, and why cached input tokens are billed cheaply.

Routing, load balancing and scheduling

LLM-aware gateways: KV-cache-aware routing, token-based rate limits, priorities and fairness.

Data parallelism for inference

Independent replicas of an already-sharded model. Each replica has its own KV pool. Throughput scales; single-request latency does not.