Prefill, decode and the KV cache

One forward pass over the whole prompt is prefill. After that, producing each new token is decode. Output tokens are typically priced higher than input tokens partly because decode is expensive per token and harder to batch, and partly because markets allow it. Either way, verbose answers cost more and feel slower even when a short sentence would do.

Two phases

Deep dive: prefill vs decode.

When TTFT spikes, check input length and cache hit rate before blaming the model name. When the stream is slow, you are on decode. When finance asks why output is several times input spend, draw this cartoon.

The KV cache and prefix reuse

During prefill, each layer stores keys and values for every prompt token. On decode, those tensors are reused so the model does not re-encode the entire prompt for every new token. That store is the KV cache.

KV bytes ≈ 2 × layers × kv_heads × head_dim × bytes_per_elem × seq_len

Long contexts blow up memory. Grouped-query attention and quantization exist largely to shrink this footprint. For product engineers the important corollary is prefix reuse.

If request N and request N+1 share a long identical prefix, a serving stack can reuse the computed KV for that prefix (prompt caching / prefix caching). Stable system prompts, large tool schemas and frozen policy docs belong at the front of the prompt and should change rarely. Putting a per-request UUID or timestamp at the start destroys the cache hit.

Details in LLM caching and inference KV cache.

Repetition loops are the other KV-shaped bug. Decode that keeps reinforcing recent tokens feeds those tokens back into the cache and amplifies them. Fix with penalties, stop sequences, or a lower max-token cap and a clearer instruction.

Common mistakes

• Putting dynamic content at the start of the prompt. Cache miss storms follow.
• Blaming the model name for a TTFT spike without checking input length and cache hit rate.
• Asking for a long answer when a short one would do. Decode is the expensive phase.

Check yourself

1. Why does a request ID at the start of the system prompt raise the bill? The prefix is no longer identical across calls, so the KV / prompt cache misses and you pay full prefill.
2. TTFT jumped after you added a 6,000-token policy to every request. Which phase? Prefill. The first pass now covers a much longer prompt.

Key takeaways

• Prefill sets time to first token. Decode sets stream speed.
• Output tokens are the expensive phase. Ask for brevity and stream.
• The KV cache reuses keys and values so decode does not redo the prompt.
• Identical prefixes are cheap. Dynamic bytes at the front are not.

Further learning

Prefill vs decode, KV cache, prefix caching. Next, where the model's knowledge lives.