Exact-match LLM cache

When temperature is 0, or sampling is otherwise deterministic enough for your use case, and the full request fingerprint is identical, you can store the completion in your own cache and skip the model entirely.

This is ordinary cache-aside, not provider KV reuse. Prompt caching still runs a generation. Exact-match can return in milliseconds and cost nothing on a hit.

Walking through a fingerprint

The key must include everything that could change the answer. This example uses an in-memory dict so you can run it as written.

import hashlib, json

store = {}

def cache_key(parts):
    payload = json.dumps(parts, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()

def complete(req, call_model):
    key = cache_key({
        "model": req["model"],
        "promptVersion": "support-answer@12",
        "messages": req["messages"],
        "temperature": 0,
    })
    if key in store:
        return store[key]
    out = call_model(req)
    store[key] = out
    return out

req = {"model": "mid-tier", "messages": ["Where is order 881?"]}
print(complete(req, lambda r: "Warehouse B"))
print(complete(req, lambda r: "SHOULD NOT RUN"))

Output:

Warehouse B
Warehouse B

The second call never hits the model. In production you would set a TTL (24 hours is a common starting point) and include tool-schema and knowledge versions in the key. See invalidation.

Good fits: classification, routing, FAQ answers with fixed prompts, evals, idempotent batch jobs. Bad fits: personalized chat, creative writing, anything where freshness or uniqueness matters.

A 24-hour TTL is a common starting point. That is not a substitute for putting promptVersion in the key. If you bump the policy prompt and keep the old key, you will serve yesterday's answer until the TTL dies.

Common mistakes

• Caching personalized chat under a global key. One customer's balance must not become another's answer.
• Omitting prompt version from the key. Yesterday's policy answer lives on.
• Using exact-match on temperature 1 chat. You will cache one sample and replay it as if it were the only answer.

Check yourself

1. Intent classification at temperature 0, same prompt version, same input. Exact-match or semantic? Exact-match. The fingerprint is identical and the task is deterministic.
2. Why include promptVersion in the key? A policy bump must miss, not serve yesterday's answer for 24 hours.

Key takeaways

• Exact-match skips the model when the full request is identical.
• Put model, prompt version, messages and temperature in the key.
• Great for deterministic subcalls. Poor for open-ended chat.
• Version the key so a prompt bump invalidates naturally.

Further learning

"Similar question" is a different, riskier layer: semantic LLM cache.