Decode latency floor

Decode is the phase where the model writes one new token at a time. At small batch sizes, each token needs to read almost all of the weights from GPU memory once. That read takes time no matter how fast the math is. So the time to read the weights sets a floor under the time per token.

The big idea

Picture a librarian who must fetch the whole encyclopedia for every sentence they write. The fetch takes the same time whether the sentence is short or long. The weights play the encyclopedia.

The picture breaks in two places. Real GPUs cache some data, and the librarian can write several sentences per trip. That second point is the batch ceiling.

Walking through it

t_decode ≈ weight_bytes / memory_bandwidth
tok_per_s ≈ 1 / t_decode

Example 1: 8B model in BF16 on one H100

1. Weights: 16 GB = 16e9 bytes.
2. Bandwidth: 3.35 TB/s = 3.35e12 bytes per second.
3. Time per token: 16e9 / 3.35e12 ≈ 0.0048 s = 4.8 ms.
4. Tokens per second: 1 / 0.0048 ≈ 210.

This assumes perfect bandwidth and ignores KV reads. Real engines land somewhat lower.

Example 2: 70B model, tensor parallel across 2 H100s, FP8

1. Full weights in FP8: about 70 GB. Each GPU holds a 35 GB shard.
2. Each GPU reads its shard: 35e9 / 3.35e12 ≈ 10.4 ms per token.
3. Tokens per second: about 96.

Activation all-reduces add a few percent. Treat about 90 tokens/s as the best single-user speed before batching.

Common mistakes

• Predicting chat speed from peak FLOPS. Twice the FLOPS with the same bandwidth barely changes single-stream decode.
• Ignoring KV reads. At long context, the cache adds bytes to every step.
• Treating the floor as a promise. Kernel inefficiency, sampling and launch overhead push real speed below it.

Check yourself

1. Each GPU holds 35 GB of weights and has 3.35 TB/s of bandwidth. What is the floor? Answer: 35e9 / 3.35e12 ≈ 10.4 ms per token, about 96 tokens/s.
2. You double the GPU's FLOPS but not its bandwidth. What changes for batch-1 decode? Answer: Almost nothing. Each step still waits on the same weight bytes.

Key takeaways

• At small batch, decode reads the weights once per token.
• The floor is weight bytes divided by bandwidth. Check it before trusting any FLOPS figure.
• Faster single-user decode needs fewer bytes (quantization, smaller models) or more bandwidth.
• Batching shares one weight read across many tokens. See the batch ceiling.

Further learning

Next, prefill time covers the phase that is limited by compute instead.