KV bytes per token

Once you decide to cache keys and values, the next question is how many bytes each token costs. That number, not FLOPs, often caps how many chats fit on a GPU.

The big idea

Every layer keeps its own notebook. Every token adds one line to every notebook, twice: a key and a value. More layers, more KV heads, or wider numbers (BF16 vs FP8) all stretch the line.

The picture breaks under tensor parallel. Each GPU holds a shard of the cache, but total bytes across GPUs still follow the same formula.

Walking through it

bytes_per_token = 2 × n_layers × n_kv_heads × head_dim × bytes_per_element
kv_bytes ≈ bytes_per_token × sequence_length

• 2 — one key tensor and one value tensor.
• bytes_per_element — 2 for BF16/FP16, 1 for FP8, 0.5 for INT4-packed.

def kv_bytes_per_token(cfg, bytes_per_elem=2):
    layers = cfg["num_hidden_layers"]
    kv_heads = cfg.get("num_key_value_heads", cfg["num_attention_heads"])
    head_dim = cfg.get("head_dim") or (
        cfg["hidden_size"] // cfg["num_attention_heads"]
    )
    return 2 * layers * kv_heads * head_dim * bytes_per_elem

llama70 = dict(
    num_hidden_layers=80,
    num_key_value_heads=8,
    num_attention_heads=64,
    hidden_size=8192,
)
print(kv_bytes_per_token(llama70))

Output:

327680

Llama-3-70B style: 80 layers, 8 KV heads (GQA), head_dim 128, BF16.

2 × 80 × 8 × 128 × 2 = 327,680 bytes ≈ 320 KB per token.

One long conversation can consume as much memory as a small model. Ten concurrent 32k sessions need about 100 GB of KV, more than a single H100 after weights.

Llama-3-8B class GQA (32 layers, 8 KV heads, head_dim 128, BF16):

2 × 32 × 8 × 128 × 2 = 131,072 bytes ≈ 128 KB/token. A 32k chat is about 4 GB. Several fit beside 16 GB of BF16 weights on one 80 GB GPU.

For a batch, sum over sequences. After weights and overhead, leftover HBM is the KV budget. Max concurrent tokens ≈ kv_budget / bytes_per_token.

Common mistakes

• Reporting "fits on 1× H100" from the weight file alone. Production fit is weights plus target concurrency × context × bytes per token, with slack.
• Using query heads in the formula on a GQA model. Use num_key_value_heads.
• Planning from average context when the SLO is about p99 prompts.

Check yourself

1. Derive the Llama-3-70B BF16 figure from 80 layers, 8 KV heads, dim 128. Answer: 2 × 80 × 8 × 128 × 2 = 327,680 bytes ≈ 320 KB/token.
2. How much KV does one 32k session need at 320 KB/token? Answer: 32,000 × 320 KB ≈ 10 GB.

Key takeaways

• bytes_per_token = 2 × layers × kv_heads × head_dim × dtype_bytes.
• Llama-3-70B GQA BF16 is about 320 KB/token.
• The leftover HBM after weights is the concurrency budget.
• Architecture and dtype change the constant. Length changes the total linearly.

Further learning

Transformer Inference Arithmetic (kipply)This article presents detailed few-principles reasoning about large language model inference performance, with no experiments or difficult math.kipp.ly

Next, the KV cache hub puts size next to bandwidth and shrinking tricks.