Core optimizations
Batching and continuous batching
Why batching amortizes weight reads, how static and dynamic batching waste GPU time, and how iteration-level continuous batching plus chunked prefill unlocks 10–20× throughput.
Decode for a single sequence streams the full weight matrix from HBM, does a tiny multiply, and writes one token. The FLOPs look cheap. The bytes look expensive. Batching fixes that: many sequences share one weight read. Autoregressive generation then breaks the naive version. Sequences finish at different times, prompts arrive continuously, and a long prefill can stall every decode in the batch.
The jump from "pad a static batch and wait" to continuous (iteration-level) batching is most of the 10–20× throughput gap between a toy server and vLLM, SGLang or TensorRT-LLM.
Concepts in this section
Read them in order. Each one builds on the one before.
| Step | Concept | What it answers |
|---|---|---|
| 1 | Why batching speeds up decode | How does a larger batch raise arithmetic intensity? |
| 2 | Static and dynamic batching | Why do pad-and-wait batches waste most slots? |
| 3 | Continuous batching | How does rescheduling every iteration refill slots? |
| 4 | Chunked prefill | How do you stop a long prompt from stalling decode? |
When memory is tight, the scheduler preempts a sequence by swapping KV or recomputing it. PagedAttention is what makes admission possible in the first place.
How it feels in metrics
When continuous batching and chunking are healthy, a concurrency sweep shows rising throughput with slowly degrading TPOT until a knee where KV fills or the token budget makes each iteration fat. Static batching charts look different: throughput jumps in plateaus, and latency for short requests tracks the long requests in their batch.
Goodput — throughput counting only requests inside SLO — is the metric to put on the dashboard. Raw tokens/s will seduce you into huge batches that miss p95 TPOT. Tie alerts to queue time and preempt rate as well. A server can show high GPU utilization while thrashing under preemption.
At work you rarely implement Orca from scratch. You pick an engine, then the batching policy is what you misconfigure. Interview angle: draw static vs continuous timing, explain why decode wants large batches, and name the prefill interference fix (chunking or disaggregation).
Common mistakes
- Benchmarking a fixed static batch of identical long prompts. Numbers look great. Production chat does not.
- Turning off chunked prefill. TPOT explodes on the first doc-QA prompt.
- Chasing peak tokens/s. Product cares about p95 TTFT and TPOT. Sweep concurrency under an SLO instead. See benchmarking.
Key takeaways
- Batching raises intensity by amortizing weight reads. Decode needs it to leave the memory roof.
- Static and timeout-based dynamic batching leave idle slots and block admission.
- Continuous batching reschedules every iteration. That is the foundation of modern LLM serving.
- Chunked prefill bounds decode stalls from long prompts.
Further learning
How continuous batching enables 23× throughput in LLM inference (Anyscale)In this blog, we discuss continuous batching, a critical systems-level optimization that improves both throughput and latency under load for LLMs.Anyscale
Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022)Orca: A Distributed Serving System for Transformer-Based Generative Models | USENIXusenix.org
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-ServeEach LLM serving request goes through two phases. The first is prefill which processes the entire input prompt and produces the first output token and the second is decode which generates the rest of output tokens, on…arXiv.org
Woosuk Kwon's MLSys seminar talk is the clearest author-level walkthrough of continuous batching and PagedAttention:
Related
Why batching speeds up decode
One weight read from HBM can serve many sequences, so larger batches raise arithmetic intensity.
Prefill vs decode
The two phases of LLM inference. Prefill is compute-bound and sets TTFT. Decode is memory-bound and sets TPOT. They interfere when they share a GPU.
Serving engines: vLLM, SGLang and TensorRT-LLM
How modern engines are built, how they differ, and which knobs move TTFT, TPOT and tokens per GPU-second.