Guide

Production and case studies

Semantic LLM cache

Embed the query, reuse a prior answer if it is close enough. High leverage on public FAQ. Dangerous on anything personal or policy-sharp.

128 of 155 · 2 min read

A semantic cache embeds the query, finds a prior query whose embedding is close enough, and returns that prior answer. Thresholds are typically cosine similarity in the high 0.9s, tuned on a labeled set of "same intent" vs "different intent" pairs.

Treat this as optional optimization after prompt caching and exact-match. Many teams skip it entirely and still hit their cost targets.

Warning

Never put account-specific answers in a global semantic cache. If you use semantic caching at all, start with public FAQ content and measure false-hit rate with humans before expanding.

The risks

  • False hits. "Cancel my subscription" vs "cancel my trial" may be near in embedding space and disastrously different in policy.
  • Personalization. Caching "What's my balance?" across users is a privacy and correctness bug. Scope keys by user or tenant, or only cache non-personalized FAQs.
  • Stale knowledge. An answer that was right last week may be wrong after a price change. Tie entries to prompt version, model and knowledge version.
key space: tenant_id + prompt_version + knowledge_version + model
lookup:    ANN over query embeddings within tenant
admit:     similarity ≥ τ AND same tool-permission tier

The analogy is a librarian who hands you the answer to a "similar" question. Helpful for "what are your store hours?" Dangerous for "refund order 881" vs "refund order 882." The embedding does not know that one digit is the whole product.

Walking through a near miss

Two queries embed at cosine 0.96:

q1 = "cancel my subscription"
q2 = "cancel my trial"
τ  = 0.95     # looks safe on a generic FAQ set

If q1 was answered with "we'll stop billing at the end of the annual term" and you serve that for q2, you just applied the wrong policy. A labeled pair set would have marked these as different intent. Raise τ, or refuse to cache any cancel/refund/account-mutating intent.

Common mistakes

  • A global cache for "What's my balance?" Privacy bug and wrong answer.
  • Threshold copied from a blog. Tune τ on your own same-intent vs different-intent pairs.
  • No knowledge version. Last week's price lives on.

Check yourself

  1. Should "What's my balance?" go in a global semantic cache? No. Scope by user, or do not cache personalized answers.
  2. Cosine 0.96, threshold 0.95, "cancel subscription" vs "cancel trial." Hit or miss? A hit on the number, and a miss on the policy. Do not cache that intent class.

Key takeaways

  • Semantic cache reuses answers to nearby questions. Optional, not default.
  • High 0.9s thresholds, tuned on labeled pairs.
  • Tenant scope. Never global personal data.
  • Version prompt, model and knowledge. Measure false hits with humans.

Further learning

Next, put versions in every key: cache invalidation. For how closeness is computed, see embeddings.

Caching for LLM apps

Prompt caching, exact-match and semantic caches: where each fits, and how to avoid serving the wrong answer.

Exact-match LLM cache

When the full request fingerprint is identical and sampling is deterministic enough, store the completion and skip the model.

Embeddings and similarity

Turning text into vectors, measuring similarity, and choosing an embedding model.