Prompt caching

Major APIs (Anthropic, OpenAI, Google and others as of when this was written) can cache a long identical prefix of your request. The first request that establishes the cache often pays a small premium (cache write). Later requests that share that prefix pay a cached read rate, commonly on the order of ~10% of base input price on Anthropic's published shape, and see much lower TTFT for the cached portion.

How matching works

• Matching is prefix-based and exact. Change one token early in the prompt and everything after that breakpoint misses.
• Providers define what is included (tools, system, messages) and how you mark breakpoints (cache_control on Anthropic; automatic prefix caching on OpenAI for eligible prompts).
• TTLs are short by default (often about 5 minutes on Anthropic, with optional longer TTLs at higher write cost). Each hit usually refreshes the TTL.
• Usage fields report cache creation vs cache read tokens. Log them.

Structuring prompts for hits

Put content that is stable across requests first. Put per-request content last.

1. Tool definitions
2. System instructions and policy
3. Large static context (docs pack, few-shot library)
4. Dynamic: user metadata you must include, conversation tail, current question

Avoid injecting timestamps, random IDs or per-user PII into the static prefix. A Current time: … line near the top of the system prompt will nuke the cache every minute.

Measure cache hit rate = cached_read_tokens / (cached_read_tokens + cache_creation_tokens + uncached_input) on the prefix you care about. Alert when hit rate collapses after a deploy. That usually means prompt assembly reordered fields or someone added a dynamic line too early.

Multi-turn agents should keep tools + system + prior turns stable and append only the new message or tool results. That is why long agent sessions get large savings.

Walking through the economics

Cache writes may cost a premium (Anthropic documents writes above base input; reads far below). If every request uses a unique prefix, you only pay write penalties. If many requests share a prefix within the TTL, reads dominate and you win.

Rough planning model for a shared P token prefix and D dynamic tokens, hit rate h, write multiplier w (for example 1.25), read multiplier r (for example 0.1):

effective_input_cost_per_request ∝
  (1 - h) * w * P   +   h * r * P   +   1.0 * D

P=10,000, D=500, h=0.85, w=1.25, r=0.1
prefix weight ≈ 0.15*1.25*10000 + 0.85*0.1*10000
              = 1,875 + 850 = 2,725
vs 10,000 uncached  →  about 3.7× cheaper on the prefix

A second picture: system + tools + help-center pack = 8,000 tokens; user question = 150. With an 80% hit rate on the 8,000-token prefix and ~10% cache-read pricing, most turns pay roughly 0.2×8000 + 150 at full price plus 0.8×8000 at cache-read, often 5 to 10× cheaper on the static portion, plus faster TTFT.

Warm a cache on deploy with one synthetic request that uses the production prefix and no user PII, so the first real users hit reads.

Design session memory so compaction replaces a trailing window without rewriting the static head. See context engineering.

Common mistakes

• A timestamp at the top of the system prompt. The prefix changes every minute.
• Unstable tool order. Sort and freeze the schema.
• Reading only "input tokens" on the dashboard. You need cache_read vs cache_creation.

Check yourself

1. You add Current time: as the first line of the system prompt. What happens to hit rate? It collapses. The prefix is no longer identical across minutes.
2. P=10,000, h=0.85, w=1.25, r=0.1. About how many token-equivalents is the prefix? 2,725, versus 10,000 uncached.

Key takeaways

• Matching is exact and prefix-based. One early token change misses the rest.
• Static first, dynamic last. Measure hit rate.
• Cached reads are much cheaper than full input. Writes may cost a premium. TTLs are short.
• Warm the cache on deploy. Do not rewrite the static head when you compact history.

Further learning

Claude docs: Prompt cachingCache prompt prefixes with `cache_control` to cut costs and latency, using automatic caching or explicit breakpoints with 5-minute or 1-hour TTLs.Claude Platform Docs

Next, skip the model entirely when the full request is identical: exact-match cache.