Caching for LLM apps

Caching in LLM apps is not one Redis key. There are at least four layers that people casually call "cache," and mixing them up causes wrong answers or mysterious bills.

Provider prompt caching — reuse of KV state for a prompt prefix
Exact-match response caching — same request → same stored answer
Semantic caching — similar question → reuse answer
Result caches for tools and retrieval

Classic HTTP/Redis caching still applies to your own layers. Prompt caching is a serving-side optimization with different rules.

If you only remember one thing: make the shared prefix byte-stable and version everything you store yourself. Most of the savings in production agents come from that discipline, not from exotic similarity search over past answers.

Concept

During inference, attention computes keys and values for each prompt token. Those tensors are the KV cache. Reusing them for an identical prefix means the GPU skips re-prefilling that prefix. Provider "prompt caching" is mostly cross-request KV reuse with a TTL, priced as cheap cache reads. See prefix caching.

How the layers fit

Request
  → exact-match response cache?  hit → return
  → semantic FAQ cache?          hit → return
  → build prompt (static prefix + dynamic)
  → provider prompt cache        (cheap/fast prefill)
  → tool/retrieval caches        (inside the agent loop)
  → store exact-match / FAQ entry if eligible

Trade-off

Prompt caching is almost always worth structuring for: low risk, large savings on shared prefixes. Exact-match is next for deterministic calls. Semantic caching is last: high leverage only on repetitive FAQ traffic with careful scoping.

Ship layers in that order. Many teams cut costs sharply with prefix discipline alone and never need semantic-cache complexity or risk.

Concepts in this section

Step	Concept	What it answers
1	Prompt caching	How do you reuse the KV prefix across requests?
2	Exact-match LLM cache	When can you skip the model entirely?
3	Semantic LLM cache	When is "similar question" safe to reuse?
4	Cache invalidation	What belongs in the key, and when do you flush?

Anti-patterns

Putting Date.now() or "User timezone: …" at the top of the system prompt.
Building tool lists dynamically per request in random order. Sort and freeze the schema.
Caching personalized completions under a global key.
Semantic-matching refund answers across customers.
Leaving response caches live after a policy prompt bump, with no version in the key.
Assuming provider prompt cache means "the answer is cached." It only helps prefill. You still pay for output tokens.

Common mistakes

A dynamic line too early in the prompt. Hit rate collapses after a deploy.
No model, prompt, tool or knowledge version in the key. A 24-hour TTL on a wrong policy answer is a bad day.
Semantic cache as the first layer. Start with prefix stability.

Key takeaways

Prompt caching reuses KV for an identical prefix. Put static content first and measure hit rate.
Exact-match caches are safe for deterministic, non-personalized calls with versioned keys.
Semantic caches need high thresholds, tenant scoping and false-hit monitoring. Optional, not default.
Cache retrieval and read-only tools. Never cache unsafe mutations.
Version model, prompt, tools and knowledge in every key.