Continuous batching

Continuous batching (also called in-flight batching or iteration-level scheduling) comes from Orca (OSDI 2022). The scheduler does not commit to a batch for the lifetime of its longest sequence. It runs one model forward, then removes finished sequences, admits waiting ones, and repeats.

Finished slots are reused on the next step — typically tens of milliseconds later, not seconds later. That is the jump from a toy server to vLLM, SGLang or TensorRT-LLM.

The big idea

Static batching is a bus that will not board anyone until it returns to the depot. Continuous batching is a bus that opens the door at every stop. The analogy breaks at memory: you can only board a new passenger if a KV seat is free.

Walking through one iteration

Every step, a production scheduler roughly answers:

Orca also introduced selective batching: attention is irregular across requests (different KV lengths), while MLP matmuls can still batch tokens. Modern engines do the same with custom attention over a ragged set of sequences (PagedAttention) and batched GEMMs for the rest.

Anyscale's widely cited measurements put vLLM-style continuous batching, with good memory management, around up to ~23× throughput versus Hugging Face Transformers-style generation on the same hardware. Matmuls did not get 23× faster. The GPU stopped idling and stopped refusing work that would have fit.

Tuning knobs

Names vary; the ideas transfer to SGLang and TensorRT-LLM.

• max_num_seqs: cap on concurrent sequences. Raise it to improve throughput until you hit KV memory or latency SLOs. Lower it to protect TPOT under load.
• max_num_batched_tokens: cap on tokens per iteration. Higher favors throughput and long prefills. Lower bounds step time and helps tail latency.
• Queue policy: FCFS vs priority; whether to admit a huge prefill when many decodes are active.
• Preemption mode: recompute vs swap.

A practical loop: fix model, GPU and quantization; sweep concurrency with a realistic length mix; plot goodput under your TTFT/TPOT limits; raise max_num_seqs until goodput flattens or SLOs break; then adjust the token budget.

Common mistakes

• Setting max_model_len so high that KV admits only a handful of sequences. Paging helps, but the cap still shapes planning.
• Optimizing for peak tokens/s. Product cares about p95 TTFT and TPOT. See benchmarking.
• Forgetting preemption. When KV fills, the scheduler must pause someone. That is a feature, not a crash.

Check yourself

1. When does a finished sequence free its slot under continuous batching? Answer: on the next iteration, typically tens of milliseconds later.
2. Why does continuous batching need paged KV? Answer: a contiguous max-length reservation runs out of memory before the GPU runs out of compute.

Key takeaways

• Reschedule every iteration so finished sequences free capacity immediately.
• Admission is limited by KV blocks, max_num_seqs and max_num_batched_tokens.
• Tune those knobs against latency SLOs, not peak tokens/s.

Further learning

Next, stop long prompts from stalling decode with chunked prefill. For the memory side, read PagedAttention.