Static and dynamic batching

Amortizing weight reads needs a batch. The naive way is to collect B requests, pad them to one length, and run until every sequence hits EOS or max_tokens. That is static batching. It is simple, and it wastes most of the GPU on real chat traffic.

The big idea

A taxi that will not leave until four passengers arrive, and will not pick anyone new until the last rider is home, sits empty a lot. Sequences finish at different times. Prompts arrive continuously. A static batch cannot use those facts.

Dynamic batching waits up to T milliseconds to accumulate a batch, then runs that batch to completion. It helps when traffic is bursty. It does not fix iteration-level waste. Once the batch starts, finished sequences still leave holes until the longest one ends.

Walking through the waste

Two wastes dominate static batching:

1. Padding. Sequences of length 50 and 500 share a batch shaped for 500. Attention and matmuls still pay for the pads unless kernels skip them.
2. Tail idle. When three of four sequences finish, the fourth keeps the whole batch alive. GPU slots for the finished sequences sit empty.

D never starts until A–C's batch completes. That is head-of-line blocking at admission: a short chat queued behind a long generation waits for the whole static batch, not for a free slot.

Four requests, max decode lengths 40, 80, 120 and 400 tokens, static batch of 4:

Useful decode steps ≈ (40 + 80 + 120 + 400) / 4 = 160 token-steps
Batch runs 400 iterations
Idle share ≈ 1 − 160/400 = 60%

Real traffic is worse because length variance is huge.

Dynamic batching with a timeout does not change the 400-iteration lock. A request that arrives 1 ms after the batch closed waits for the entire generation. For LLMs, where a batch may run for seconds, that timeout policy is the wrong abstraction.

Common mistakes

• Benchmarking a static batch of identical long prompts. Numbers look great. Production chat does not.
• Assuming a 10 ms dynamic-batch timeout fixes LLMs. The waste is inside the generation, not only in forming the batch.
• Blaming the model for high TPOT on short requests. They may be stuck behind a long sequence in the same static batch.

Check yourself

1. Lengths 40, 80, 120, 400 in a static batch of 4. What share of slot-iterations is idle after early EOS? Answer: about 60%, because the batch runs 400 steps and useful work averages 160.
2. Why does a dynamic-batch timeout not fix this? Answer: once the batch starts, finished sequences still leave holes until the longest one ends.

Key takeaways

• Static batching pads to the longest sequence and holds slots until that sequence ends.
• Dynamic batching only changes how the batch forms, not how it runs.
• Short requests wait for the whole batch. That is head-of-line blocking.

Further learning

Continuous batching reschedules every iteration so finished slots refill immediately.