Why the KV cache exists

Autoregressive generation without a KV cache would recompute attention keys and values for every past token on every new step. Cost would grow quadratically with sequence length, and interactive chat would be unusable. The cache stores those tensors once and reuses them, trading GPU memory for compute.

The big idea

A meeting transcript that you retype from scratch every time someone speaks is generation without a cache. A growing notebook you only add a line to is generation with one.

The picture breaks on the attention scores themselves. Projection work for the past is paid once. Each new query still scores against the whole history, so score matmuls still grow with cached length. Very long contexts need IO-aware attention and sometimes sparse patterns. The production surprise is almost always the byte count, not those extra FLOPs.

Walking through it

For token t, without a cache you rebuild K and V for positions 1…t from hidden states every step. With a cache:

1. Prefill writes K and V for every prompt position.
2. Each decode step computes K and V only for the new token and appends them.
3. Attention for the new query reads the full cached K/V.

The compute savings grow with context length. The bill is memory that grows linearly with tokens × layers × heads. After weights are loaded, that memory is often the scarce resource on an inference GPU.

Without the cache, long generations scale like a fresh prefill every step. With the cache, you pay linear memory growth in exchange for linear (not quadratic) compute in the projection part of attention.

Every serious serving stack is, underneath the API, a KV memory manager with a language model attached. Paging and sharing across requests land in PagedAttention and prefix caching.

Common mistakes

• Thinking the cache removes attention FLOPs. It removes repeated K/V projections. Scores still grow with length.
• Calling a replica "fitted" after the weights load. Users need cache room too.
• Recomputing past K/V for "simplicity." That makes chat quadratic and unusable.

Check yourself

1. What work does the cache skip on step t? Answer: Rebuilding K and V for positions 1…t-1. It only projects the new token and reads the history.
2. Does the cache make attention scores O(1) per new token? Answer: No. The new query still attends to all cached positions, so that part grows with length.

Key takeaways

• The cache stores per-layer keys and values so decode does not recompute the past.
• Prefill fills it. Decode appends one column per token.
• You trade linear HBM growth for linear, not quadratic, projection work.
• Byte count, not leftover FLOPs, is the usual production limit.

Further learning

Next, KV bytes per token turns this into a size formula.