GQA, MQA and MLA

The size formula has n_kv_heads in it for a reason. You can shrink the cache by changing the architecture, not only by shrinking the dtype. MHA → GQA → MQA → MLA is the usual ladder.

The big idea

MHA is a filing cabinet with a drawer per reader. MQA is one shared drawer. GQA is a drawer per team. MLA is a compressed summary of the drawer.

The picture breaks on quality and on tensor parallel. Fewer KV heads usually speeds decode (less traffic) and costs some quality. You must check your evals. MQA also leaves less flexibility to shard heads across GPUs.

Walking through it

Llama 2/3 70B-class models use GQA, for example 64 query heads and 8 KV heads. You get most of MQA's memory win with better quality and friendlier tensor parallel.

MLA absorbs projections so the cached footprint per token is much smaller than even GQA for comparable models. Exact ratios depend on the config. The serving takeaway is that architecture can cut KV pressure by several times.

Sliding-window / local attention layers only need a window of recent tokens, capping growth for those layers. Hybrid stacks mix global and local layers so long context does not mean full dense KV everywhere. See modern architectures and long-context serving.

Prefer the smallest KV design your quality bar allows. Quantizing a bloated MHA cache is a weaker fix than not storing the bytes.

Decode is already memory-bandwidth bound on weights. A large KV adds a second stream of reads that grows with context. GQA and MLA help serving twice: fewer bytes stored, and fewer bytes moved per step.

Common mistakes

• Treating GQA as a free quality win. Measure on your tasks. Coding and long-range reasoning can be more sensitive than short chat.
• Using query-head count in the KV formula on a GQA checkpoint.
• Reaching for KV quant before checking whether the model is still MHA.

Check yourself

1. 64 query heads and 8 KV heads. About how much smaller is KV than MHA? Answer: About 8×, because n_kv_heads dropped from 64 to 8.
2. Why does GQA often speed decode, not only save memory? Answer: Each step streams fewer KV bytes, and decode is already bandwidth-bound.

Key takeaways

• KV size tracks KV heads, not query heads.
• GQA is the common 70B default. MQA is smaller. MLA is smaller still.
• Architecture is the first lever. Quantization is the second.
• Fewer KV bytes also means less traffic per decode step.

Further learning

Fast Transformer Decoding: One Write-Head is All You Need (MQA)Multi-head attention layers, as used in the Transformer neural sequence model, are a powerful alternative to RNNs for moving information across and between sequences. While training these layers is generally fast and …arXiv.org DeepSeek-V2 (Multi-head Latent Attention)We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and…arXiv.org

Next, KV cache quantization halves the remaining bytes.