The KV cache

After the weights load, the scarce resource on an inference GPU is usually the KV cache. It grows with every token of every live request. Size it wrong and you OOM, preempt, or advertise a context length you cannot batch.

This section is the size formula, the Llama-3-70B numbers, and the levers that shrink the footprint. Paging and sharing across requests live in PagedAttention and prefix caching.

Concepts in this section

Bandwidth and concurrency

From inference arithmetic: after weights and overhead, leftover HBM is the KV budget. Long-context features cut QPS even if FLOPs look idle. Throughput stops scaling with batch once KV is full. The scheduler refuses or preempts.

Even when KV fits, decode must stream it. Extra traffic per step scales with batch × seq_len × bytes_per_token, plus weight traffic. At long context, KV traffic can rival weight traffic and pull tokens/s down. When TPOT worsens as a conversation grows but batch size is flat, suspect KV bandwidth and cache length.

Fragmentation has its own picture. If the engine reserves a contiguous slab sized to max_model_len for every sequence, a user who stops at 200 tokens still owns space for 8k or 128k. Paging fixes allocation. It does not shrink bytes_per_token. You still need GQA, MLA or quant when the working set is genuinely huge.

When sizing a replica, compute bytes per token from the config and multiply by p95 context × target concurrent sequences. Put that next to weight memory in the design doc. In incidents, check KV utilization before blaming the model file.

A feature that "just adds the PDF to the prompt" can multiply KV by 10×. Propose retrieval, summarization, or a smaller-KV model before the cluster pages itself to death.

Common mistakes

• OOM on long prompts. Prefill tries to allocate prompt-length KV at once. Chunking and paging change when bytes appear, not whether.
• "The model fits" from a weight-only check. Production fit includes concurrency × context.
• Cross-request leaks in block free lists. Rare engine defects that look like impossible OOMs after idle periods.

Key takeaways

• The cache stores per-layer K and V so decode does not recompute the past.
• Llama-3-70B-class GQA BF16 is about 320 KB/token.
• KV budget, not FLOPs, often caps concurrency.
• Shrink with architecture first, then quant, then offload. Verify quality and hit rates.

Further learning

Transformer Inference Arithmetic (kipply)This article presents detailed few-principles reasoning about large language model inference performance, with no experiments or difficult math.kipp.ly DeepSeek-V2 (Multi-head Latent Attention)We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and…arXiv.org Fast Transformer Decoding: One Write-Head is All You Need (MQA)Multi-head attention layers, as used in the Transformer neural sequence model, are a powerful alternative to RNNs for moving information across and between sequences. While training these layers is generally fast and …arXiv.org