Why batching speeds up decode

Decode for a single sequence is almost always memory-bound. Each step you stream the full weight matrix from HBM, do a tiny matrix–vector multiply, and write one new token. The FLOPs look cheap. The bytes look expensive. Batching is how you fix that. If sixteen sequences share one forward pass, you still read the weights once and do sixteen times the arithmetic on them.

The big idea

A library that fetches one book for one reader is wasteful if twenty people want the same book. Fetch it once, hand it around. The analogy breaks for prefill: a long prompt already has many tokens, so one request can be compute-bound by itself. Batching still helps short prompts. Mixing prefills and decodes is the scheduler's hard problem.

Walking through one linear layer

For a decode step with batch size B, hidden size H, and one linear layer of shape H × H:

Bytes moved ≈ 2 × H × H × sizeof(dtype)     # read weights once (dominant)
FLOPs       ≈ 2 × B × H × H                 # matmul for B tokens
Intensity   ≈ B / sizeof(dtype)             # roughly proportional to B

In FP16, intensity ≈ B / 2. On an H100 (about 3 TB/s HBM, about 990 TFLOP/s dense FP16 Tensor Core), the ridge point is roughly 990e12 / 3e12 ≈ 330 FLOPs/byte. You need a large effective batch before decode stops being memory-bound. You will not get there with B=1. You often will with tens of concurrent sequences.

That is the same story as the batch ceiling: throughput rises with batch until compute or KV memory runs out.

Prefill is different. A long prompt already has many tokens, so intensity is high without a batch. The scheduler then has to mix those fat prefills with skinny decodes without destroying TTFT or TPOT.

Common mistakes

• Thinking batching is only a training habit. Decode needs it to escape pure memory-bound behavior.
• Stopping at B=1 latency. Single-stream tok/s is the floor, not the fleet plan.
• Ignoring prefill. A long prompt can be compute-bound alone; the hard part is mixing it with decode.

Check yourself

1. In FP16, about how does intensity scale with batch size B? Answer: roughly B / 2 FLOPs per byte, because weights are read once.
2. Why does B=1 stay memory-bound on an H100? Answer: intensity is about 0.5 FLOPs/byte, far below the ridge near 330.

Key takeaways

• Decode streams the full weights each step. Extra sequences reuse that read.
• Intensity rises with batch size. Tens of concurrent sequences are how you leave the memory roof.
• Prefill does not need a batch the same way. Mixing the two is the next problem.

Further learning

Next, see what goes wrong when the batch is static.