KV cache quantization

Weights are only half the memory story at long context. The KV cache can dominate. Quantizing KV to FP8, INT8 or INT4 shrinks that footprint, which raises batch size or max context. Attention kernels must read low-precision KV, or dequantize on the fly.

Quality sensitivity is real for long-context retrieval and reasoning. Measure with long-context evals, not only WikiText perplexity. Errors compound over long sequences.

The big idea

Weight quantization is a one-time shrink of the model. KV quantization is a shrink of every live token. A 70B model in FP8 may already fit. The cache for a hundred long chats may not.

The picture breaks if you only smoke-test short chat. INT4 KV can look fine on 512-token replies and fail a needle-in-a-haystack test at 32k.

Walking through the budget

From inference arithmetic, cache bytes grow with batch, sequence, layers, heads and dtype. Halving the KV dtype halves the bytes per token.

Same replica, more room for cache
Weights shrink, or KV dtype drops
  → more token slots
  → higher concurrency, or longer context

Smaller KV also means more cached prefixes fit, which can improve quality-adjusted cost even if per-token logits are slightly noisier. See prefix caching.

Eval the change the same way you eval weight quantization, with extra weight on long context:

1. Perplexity on a held-out corpus, as a cheap smoke test.
2. Task evals that match product: coding, RAG, tool-calling.
3. Long-context needle and multi-hop tests.
4. Reasoning-heavy prompts.
5. Side-by-side with the BF16 teacher on a golden set.

Common mistakes

• Shipping KV INT4 on a short-chat eval. Long-context retrieval is where it breaks.
• Forgetting the kernel must support the KV dtype. A format the attention kernel cannot read is unused memory savings.
• Ignoring the prefix-cache interaction. Smaller KV can raise hit rate because more trunks fit in HBM.

Check yourself

1. Weights already fit in FP8. Why still quantize KV? Answer: at long context or high concurrency the cache, not the weights, is the memory limit.
2. Which eval would you refuse to skip after turning on INT4 KV? Answer: a long-context retrieval or needle test. Short-chat perplexity hides the failure.

Key takeaways

• KV can dominate memory after weights already fit.
• FP8 KV is the usual first step. INT8 and INT4 need long-context evals.
• Smaller KV can raise prefix-cache hit rate as a side effect.
• The attention kernel must read the quantized cache, or you saved nothing.

Further learning

Back to the quantization map. For the cache itself, see the KV cache.