Chunked prefill

Naive continuous batching still has a sharp edge. Prefill of a 32k-token prompt is a large, mostly compute-bound chunk of work. If you insert it whole into the next iteration, every decode sequence in that batch waits for that prefill to finish. Interactive users see TPOT spikes. That is prefill–decode interference.

Chunked prefill (Sarathi-Serve and follow-ons) splits a long prompt into chunks of at most C tokens. Decode keeps making progress every step. TTFT for the long prompt gets slightly worse. TPOT for everyone else stays stable.

The big idea

A long speech at the start of a meeting stalls every other agenda item. Split the speech into two-minute slices and take questions between them. The speaker finishes a bit later. Everyone else stays awake.

The analogy breaks when you move prefill and decode onto different GPU pools entirely. That is disaggregated serving. Chunked prefill is the single-pool compromise that most engines ship first.

Walking through a budget

max_num_batched_tokens = 8192
active decodes         = 64 sequences → 64 tokens
remaining budget       = 8192 − 64 = 8128
chunk size C           = 2048
→ this iteration: 64 decode + one 2048-token prefill chunk

Without chunking, a 16k prefill in a batch with 64 decodes might turn one "decode step" into a multi-hundred-millisecond wall-clock step. With C in the 512–2048 range, each step stays closer to normal decode latency while the prompt drains over several iterations.

You are explicitly trading a bit of prefill latency for decode latency SLOs. Raise C (or the token budget) to finish prefills faster. Lower it to protect TPOT.

Common mistakes

• Turning chunked prefill off. TPOT explodes whenever a doc-QA prompt arrives.
• A chunk so large it is the whole prompt. You are back to one fat iteration.
• Chasing TTFT only. The point of chunking is to protect everyone else's stream.

Check yourself

1. Budget 8192 tokens, 64 active decodes, C=2048. How many prefill tokens run this step? Answer: 2048, because 8192 − 64 = 8128 remaining and C caps the chunk.
2. What do you trade when you chunk a long prompt? Answer: a bit more TTFT for the long prompt, in exchange for stable TPOT for the rest.

Key takeaways

• A whole long prefill in one iteration stalls every decode in the batch.
• Chunk the prompt so each step is decode tokens plus at most C prefill tokens.
• Disaggregation is the upgrade when chunking still leaves TPOT spiky.

Further learning

The Sarathi-Serve paper is the source for this design: Taming Throughput-Latency Tradeoff in LLM Inference. When one pool is not enough, read disaggregated serving.