Guide

Core optimizations

Why reserved KV memory wastes the GPU

Reserving one contiguous block of KV cache for the longest allowed sequence leaves most of it empty, and scatters the free space into pieces too small to use.

48 of 138 · 2 min read

A serving engine must keep the KV cache for every running request in GPU memory. The simple way is to reserve a slab sized for the longest sequence the model allows, as soon as the request arrives. Most requests never get close to that length. The slab sits mostly empty, and the GPU can reject new work even when its memory bar looks half free.

Concept

The KV cache stores the keys and values of every past token, so the model does not recompute them for each new token. It grows by one position per token generated. For how it works, see the KV cache.

The big idea

Think of a hotel that holds the top floor for each guest's maximum possible stay, say 30 nights, even when most guests leave after 2. Most rooms sit empty all the time.

The picture breaks at fragmentation. Hotel rooms do not get scattered. GPU memory does. Free bytes can end up in small pieces, so the total looks large but no single piece is big enough for the next reservation.

Walking through one reservation

KV size per token follows this rule:

bytes = layers × KV heads × head dim × positions × 2 (keys and values) × bytes per value

Here is an illustrative case, with about 0.5 MB of KV per token:

Reserve for max_model_len = 4,096:   4,096 × 0.5 MB = 2 GB per request
Live tokens after a 512-token chat:    512 × 0.5 MB = 256 MB
Empty share of the reservation:        (2,048 − 256) / 2,048 = 87.5%
Eight such requests:  reserved 16 GB, live about 2 GB

The waste comes from the reservation, not from the model. The arithmetic is the same for any model; only the per-token size changes.

Two kinds of fragmentation

  • Internal: unused space inside a single reservation, like the empty 87.5% above.
  • External: free space split into holes. Enough bytes are free in total, but not as one contiguous region for the next large reservation.

Reports on serving systems put the waste under contiguous reservation at roughly 60% to 80% for realistic length spreads. That gap is often the difference between a batch of 4 and a batch of 16 on the same GPU.

Common mistakes

  • Assuming free memory means spare capacity. If the free bytes are in holes, the next request still cannot fit.
  • Thinking GPUs cannot fragment. They can, whenever variable-sized allocations are freed out of order.
  • Blaming the batcher first. If the batch is small while memory looks free, check how the KV cache is allocated.

Check yourself

  1. A server reserves 4,096 tokens per request, and a typical request uses 512. What share of each reservation is empty? Answer: 3,584 of 4,096, which is 87.5%.
  2. Why can a GPU reject a request even when its memory bar shows free space? Answer: the free bytes may be in pieces, and none may be large enough for the next contiguous reservation.

Key takeaways

  • Reserving for the maximum length wastes most of the KV cache for typical requests.
  • Waste comes in two forms: internal (empty space inside a reservation) and external (scattered holes).
  • Free bytes are only useful if they are in one piece large enough for the next allocation.
  • Fixing this needs allocation in small blocks, which is the idea behind PagedAttention.

Further learning

Next, read block tables to see how paged KV blocks solve this. The PagedAttention paper describes the original measurements.

PagedAttention and KV memory management

How reserving contiguous max-length KV wastes most of GPU memory, and how fixed-size KV blocks with block tables eliminate fragmentation and unlock sharing.

The KV cache

What keys and values the cache stores, how big it gets per token, and how GQA, MLA, quantization and offloading shrink it.

Batching and continuous batching

Why batching amortizes weight reads, how static and dynamic batching waste GPU time, and how iteration-level continuous batching plus chunked prefill unlocks 10–20× throughput.