Core optimizations
Shared prompt prefixes
System prompts, tools, few-shots and chat history repeat across requests. Those shared heads are the tokens prefix caching can skip.
Every token of prefill writes KV cache and burns compute. In production, a huge fraction of those tokens are identical across requests: the same system prompt, the same tool schemas, the same RAG preamble, the same agent header, yesterday's multi-turn history still sitting on the session. Recomputing that KV every time is waste.
Prefix caching keeps KV for shared token prefixes and reuses it on the next request that starts the same way. This page is where those prefixes come from, and why putting static bytes first matters.
Concept
A prefix is the leading span of tokens that two prompts share. Caches match from the start. A difference at token 10 throws away everything after it.
The big idea
Think of a form letter. The letterhead is printed once. Only the name at the bottom changes. Prefix caching is that letterhead, stored as KV.
The picture breaks if you put the name at the top. Then every letter looks unique, and the press reprints the whole page.
Walking through the patterns
| Pattern | Shared prefix |
|---|---|
| System prompts | Long policy, persona, or safety text |
| Few-shot | Same exemplars before each user question |
| Multi-turn chat | History up to the last turn |
| RAG | Identical instructions, sometimes identical retrieved docs |
| Agents | Tool definitions, ReAct headers, shared planner prompts |
| Code assistants | Repo rules, file preambles, linter context |
If two prompts share only a short header and diverge immediately, gains are small. If 2k tokens of tools and system are stable all day, gains are enormous.
Example
System plus tools = 3,000 tokens; user question = 100 tokens. Without cache, every request prefills 3,100. With a hot prefix cache, you prefill about 100. At equal decode length, TTFT can fall by an order of magnitude, and the GPU absorbs far more QPS before prefill saturates.
Exact response caching (LLM caching) skips even more when the entire answer repeats. It cannot help when the user question changes while the tools and system stay fixed. That is prefix KV's sweet spot.
Rules that raise hit rate:
- Put immutable content first: policy, tools, schema, few-shots. Put user-specific and retrieved text last.
- Do not regenerate tool JSON with shuffled key order if the serializer is nondeterministic.
- Prefer a small set of system-prompt versions over per-tenant micro-edits that explode the tree.
- For RAG, cache the instruction prefix. If retrieved chunks differ every time, only that prefix hits.
- For agents, cache the tool hub. Do not put timestamps or request ids before large static blobs.
Tip
If you can hash the first 2k tokens of production traffic and find thousands of unique prefixes that differ only by a UUID in position 10, move the UUID. That single edit can turn a cold cache into a hot trunk.
Common mistakes
- Paraphrasing the system prompt each request. Meaning matches. Tokens do not. The cache misses.
- Putting a request id or timestamp first. The longest common prefix becomes a few tokens.
- Expecting RAG chunks to hit. They usually do not. The instructions before them can.
Check yourself
- A 3,000-token tool list sits after a per-request UUID. What does the cache see? Answer: a unique prefix from token 1. The 3,000 tool tokens never match.
- Why can prefix KV help when an exact-response cache cannot? Answer: the user question changed, so the full answer is new, but the system and tools are the same.
Key takeaways
- Shared system prompts, tools, few-shots and history are the tokens you can skip.
- Caches match bytes from the start. Static content goes first.
- Tiny unique headers destroy hit rate.
- Prefix KV still runs the model on the uncached suffix. Response caches skip the model entirely.
Further learning
Next, how engines store those prefixes: automatic prefix caching.
Related
Prefix caching
Reusing KV across requests that share prompts. Block hashing, RadixAttention, cache-aware routing, and why cached input tokens are billed cheaply.
The KV cache
What keys and values the cache stores, how big it gets per token, and how GQA, MLA, quantization and offloading shrink it.
Caching for LLM apps
Prompt caching, exact-match and semantic caches: where each fits, and how to avoid serving the wrong answer.