Decode batch size on the roofline

Transformer layers are dominated by matmuls of activations [B, d] against weights [d, d]. For a weight matrix read once from HBM and reused across a batch of B token positions, intensity scales roughly linearly with B. Doubling batch roughly doubles FLOPs for the same weight bytes.

That is the hardware reason continuous batching exists. Prefill with a long prompt already has a large effective B (many tokens in one forward), so it sits closer to compute-bound even when decode does not.

The big idea

One librarian fetching the same reference book for one reader is slow. Fetching it once for 256 readers is the same walk and much more useful work. The picture breaks when the "readers" are KV-cache slots you do not have: batching walks you toward the roof only until memory fills.

Walking through a d = 4096 layer

BF16 weight matrix d×d with d = 4096:

Weight bytes ≈ 4096² × 2 ≈ 33.6 MB

FLOPs for B tokens ≈ B × 2 × 4096²
                   ≈ B × 33.6e6

I ≈ (B × 33.6e6) / 33.6e6 ≈ B FLOP/byte

Activations add bytes too. Treating weights as the dominant traffic is the usual decode approximation when B is modest and d is large. Compare to I_ridge ≈ 295 on the H100 BF16 sketch:

B = 1    → I ~ 1      → deeply memory-bound
B = 32   → I ~ 32     → still memory-bound
B = 256  → I ~ 256    → near the ridge
B = 512  → I ~ 512    → compute-bound territory

Exact thresholds shift with precision, residual streams and multi-matrix layers. The story is stable: batch-1 decode crawls the bandwidth slope; batching walks you toward the roof.

Quantization is an intensity play. FP8 halves weight bytes, so roughly I ≈ 2B. The batch needed to near a 295 ridge drops from ~300 to ~150. If activations and KV stay wider, the factor is smaller — but the direction holds. That is why quantization helps tokens/s even when "peak FLOPS" at the old dtype looked unused.

Same 7B-class model on H100, rough story:

Prefill 2k tokens, batch 1:
  effective tokens in matmuls ~ 2048 → intensity high → compute-ish

Decode at 8 concurrent users:
  effective B ~ 8 → intensity ~ O(10) FLOP/byte → memory-bound

Decode at 64 concurrent users:
  effective B ~ 64 → intensity climbs → much better tokens/s/GPU

Load shape is part of the algorithm. "Speed up the model" without concurrency is incomplete.

When a teammate says "we're only at 30% of peak FLOPS," answer with intensity. Decode at B=8 has I≈8 FLOP/byte on a 295 ridge, so expected utilization is about 8/295 ≈ 3% of peak FLOPS. 30% would be suspiciously high, not low.

Common mistakes

• Reading 30% of peak FLOPS as a failed kernel when intensity said 3% was the roof.
• Tuning decode kernels at batch 1 and expecting fleet tokens/s to match.
• Ignoring quantization's byte cut. It moves you right on the plot, not just "smaller weights."

Check yourself

1. On the I ≈ B sketch, is decode at B = 32 compute-bound on a 295 ridge? Answer: no; I ~ 32 is still far left of 295.
2. If expected FLOPS util is I / I_ridge, what should B = 8 show? Answer: about 8/295 ≈ 3% of peak FLOPS.

Key takeaways

• Decode intensity scales with batch. Batch-1 is deeply memory-bound.
• Prefill already has a large effective batch, so it sits nearer the compute roof.
• Quantization raises intensity by cutting weight bytes.
• "Low % of peak FLOPS" is often the roofline, not a bug.

Further learning

Collectives add another roof in communication roofline.