Guide

Production and case studies

Cache invalidation

Every LLM cache key needs model, prompt, tools, knowledge and tenant. Prefer versioned keys over hoping a TTL saves you.

129 of 155 · 2 min read

A 24-hour TTL on a wrong policy answer is a bad day. Every LLM cache entry, response or semantic, should include the versions that produced it. When you bump any of them, change the key prefix or flush.

Concept

Before you start: prompt caching invalidates automatically when prefix bytes change. Your job is to not change them accidentally, and to version the application caches you own.

What belongs in the key

  • Model id (exact snapshot if the provider offers one)
  • Prompt / template version
  • Tool schema version
  • Knowledge / index version for RAG
  • Tenant / user scope when data is private

Prefer versioned keys (prompt@13:…) over hoping TTLs save you.

Invalidation strategies:

  • Deploy-time. CI bumps PROMPT_VERSION. New keys naturally miss until filled.
  • Event-driven. A doc update publishes knowledge_version++ or deletes keys for that doc id.
  • Safety flush. A kill switch clears response caches if a bad prompt ships.

Keep dynamic assembly in one function covered by a unit test that asserts the static prefix is byte-stable across sample users.

Tool and retrieval caches

Tool outputs and retrieval chunks are often more cache-friendly than final LLM answers.

WhatKeyTTL / invalidation
Retrieval chunks for a queryindex_version + query or doc idsInvalidate on index rebuild or doc update
HTTP GET tool resultsURL + auth scopeShort TTL. Respect upstream cache headers
Computed account summariesuser_id + as_ofInvalidate on account write events
Embeddings of documentsdoc_id + embedding_model + text_hashImmutable until the doc changes

Caching retrieval means the generator still runs, but you skip embedding plus vector search latency. Caching tool GETs cuts serial agent time. Do not cache unsafe side effects (POST refund). Cache the read that informed the decision, not the mutation.

Debugging hit rate drops

  1. Diff the rendered static prefix between two requests. Byte-identical?
  2. Check whether tools or system strings changed order or whitespace.
  3. Confirm TTL: idle periods longer than the cache lifetime force rewrites.
  4. Read cache_read vs cache_creation fields. Dashboards that only show "input tokens" hide the story.
  5. Check model id pins. Switching variants usually isolates caches.

Common mistakes

  • TTL-only invalidation after a policy bump. Version the key.
  • Caching a POST refund. Cache the read, never the mutation.
  • A unit test that never asserts prefix stability. The next deploy silently kills prompt-cache hits.

Check yourself

  1. You ship prompt@13. What should happen to prompt@12 response-cache entries? They miss. New keys, or a flush. Do not serve @12 answers under @13.
  2. Safe to cache: GET of a help article, or POST refund? The GET. Never cache the mutation.

Key takeaways

  • Version model, prompt, tools, knowledge and tenant in every key.
  • Prefer version bumps over hoping TTLs save you.
  • Cache retrieval and read-only tools. Never cache unsafe mutations.
  • When prompt-cache hit rate drops, diff the prefix bytes.

Further learning

Back to the layer map in caching for LLM apps. The classic patterns are in caching.

Caching for LLM apps

Prompt caching, exact-match and semantic caches: where each fits, and how to avoid serving the wrong answer.

Prompt caching

Providers reuse KV state for an identical prompt prefix. Put static content first. A one-token change early in the prompt misses everything after it.

Caching

Where to put a cache, how to read and write through it, how to size it from hit-rate math, and how to keep it from serving stale data or melting down.