Batch ceiling

A decode step reads the weights once. If that step serves 32 sequences instead of one, the same read produces 32 tokens. Throughput for the whole GPU rises with batch size. That gain has two limits: the math runs out, or the KV cache fills memory.

Walking through it

A decode step takes whichever is slower: reading the weights, or doing the math for every sequence in the batch.

t_step(B) ≈ max(weight_bytes / bandwidth, (2 × N × B) / FLOPs)

Example: 70B FP8 on two H100s, using the numbers from weight memory and prefill time.

1. Weight read per step: about 10 ms.
2. Math for a batch of 32: 2 × 70e9 × 32 ≈ 4.5e12 FLOPs, which takes 4.5e12 / 1.2e15 ≈ 3.7 ms.
3. Step time: max(10 ms, 3.7 ms) = 10 ms. Memory still wins.
4. Throughput: 32 tokens per 10 ms, so up to about 3,000 tokens/s.

At batch 32 the step is still memory-bound, so batching still helps. The crossover happens at a larger batch, where the two terms meet.

Past the ridge, more batch adds latency and very little throughput.

The other limit: KV memory

Batch size is also capped by memory for the cache.

kv_budget ≈ GPU_memory − weights − activations − fragmentation_slack

On the same setup, about 80 GB of KV budget and about 1.28 GB per 4,000-token chat give about 60 concurrent chats. That is a rough upper bound. The next section explains why the real number is lower. See the KV cache for the per-token size.

Common mistakes

• Thinking more batch is always more throughput. Past the ridge, it mostly adds latency.
• Sizing concurrency from weights alone. The KV cache usually sets the real limit.
• Planning from average context. Long prompts use the cache much faster.

Check yourself

1. Moving from batch 1 to batch 32 raises throughput. Why? Answer: The same weight read now produces 32 tokens instead of one.
2. Why does a 70 GB weight load leave room for only a few sequences on an 80 GB GPU? Answer: Only a few GB remain for the cache, and each sequence needs hundreds of MB or more.

Key takeaways

• Batching reuses each weight read, so tokens per second rise with batch size.
• Past the ridge point, more batch mostly adds latency.
• Concurrency is roughly KV budget divided by bytes per token times sequence length.
• Check both limits, then measure the operating point in a benchmark.

Further learning

Batching covers how schedulers keep this batch full in practice.