Prefix caching

Every token of prefill writes KV cache and burns compute. In production, a huge fraction of those tokens are identical across requests. Prefix caching keeps KV for shared token prefixes and reuses it on the next request that starts the same way. First-token latency drops, prefill FLOPs drop, and GPU capacity stretches.

Providers bill cached input tokens at a steep discount, often on the order of ~10% of uncached input price as of when this was written, because their marginal cost really did fall. This section is the serving-engine half of the story. Product-level prompt caching also appears in LLM caching.

Decode cost scales with output length. Prefill cost scales with prompt length. Agent and RAG systems often send thousands of input tokens for tens of output tokens. Shaving repeated prefill is then the dominant cost lever.

Concepts in this section

Economics and what to measure

When TTFT is bad but GPUs look idle on decode:

1. Log prefix cache hit rate and mean cached tokens.
2. Put static prompt bytes first. Confirm tokenization is stable.
3. Enable APC or a radix cache. Size GPU reserved for cache vs active KV.
4. Add session affinity or cache-aware routing.
5. Only then invest in L2/L3 offload.

Compare notes with application-level LLM caching. Prefix KV reuse still runs the model for the uncached suffix and for decode. Response caches skip the model when the full answer is reusable. Use both, at different layers.

Common mistakes

• Randomly reordering tools or injecting a UUID at the front. The first blocks miss and the cache looks broken.
• Round-robin across replicas. Each pod sees a random subset of prefixes.
• Judging by request hit rate alone. A hit on a 20-token header is not the same as a hit on a 2,000-token tool list.

Key takeaways

• Shared system prompts, tools, few-shots and history make KV reuse one of the highest-ROI serving optimizations.
• vLLM hashes paged blocks. SGLang stores prefixes in a radix tree.
• Cache-aware routing is required in multi-replica deployments, or hit rate collapses.
• Providers discount cached input because prefill work disappeared. Measure hit rate in your own fleet too.

Further learning

SGLang: Efficient Execution of Structured Language Model ProgramsLarge language models (LLMs) are increasingly used for complex tasks that require multiple generation calls, advanced prompting techniques, control flow, and structured inputs/outputs. However, efficient systems are l…arXiv.org Fast and Expressive LLM Inference with RadixAttention and SGLang (LMSYS)Large Language Models (LLMs) are increasingly utilized for complex tasks that require multiple chained generation calls, advanced prompting techniques, control flow, and interaction with external envi...lmsys.org vLLM documentationdocs.vllm.ai