Prefill–decode interference

Every chat completion has two personalities. Prefill chews the prompt: fat parallel matmuls, often compute-bound, judged by TTFT. Decode emits tokens one (or a few) at a time: skinny, memory-bound, judged by TPOT. Colocating them on one GPU makes them fight for SMs, HBM bandwidth and scheduler attention.

The big idea

A blender and a toaster on one circuit: run the blender and the toast waits. Chunked prefill slices the blender into pulses so the toaster keeps clicking. The slices still steal HBM bandwidth. Disaggregation puts them on different circuits.

The picture breaks when transfer of the KV cache across the network costs more than the interference you removed.

Walking through a spike

On one replica:

• Prefill wants large GEMMs, high SM occupancy, sometimes more tensor parallel to cut wall-clock.
• Decode wants a high batch of active sequences, maximum HBM bandwidth for weights and KV, often less TP and more replicas.
• Chunked prefill interleaves chunks with decode steps, but chunks still steal bandwidth and inflate TPOT under load.

A decode-only batch of 64 sequences runs at 40 ms TPOT. Insert one 8k-token prefill without chunking and the next decode step may jump to 80–200 ms while SMs grind the prompt. Chunked prefill (for example 512–2048 token chunks) bounds the spike — say +10–30 ms per chunk — but every chunk still contends for HBM with KV reads for the decode cohort.

Under high QPS the scheduler faces an ugly menu: delay prefills (TTFT suffers), delay decode (TPOT suffers), or over-provision GPUs so both have slack. Disaggregation deletes that menu item. Prefill GPUs run large uninterrupted GEMMs. Decode GPUs keep steady-state batching. The remaining coupling is queueing at the router and KV fabric congestion.

Heterogeneous hardware sharpens the wedge. Decode under-utilizes cutting-edge compute and can sit on slightly older or lower-power GPUs. Prefill prefers the newest FLOPs/$.

Common mistakes

• Reading high tokens/s as healthy. Dual SLOs can still be failing.
• Assuming chunking removes interference. It bounds the spike. It does not delete bandwidth contention.
• One parallelism plan for both phases. Prefill often wants more TP. Decode often wants more replicas.

Check yourself

1. A 64-sequence decode batch runs at 40 ms TPOT. What happens if an 8k prefill joins unchunked? Answer: that decode step can jump to 80–200 ms.
2. What three choices does a colocated scheduler have under high QPS? Answer: delay prefill, delay decode, or over-provision.

Key takeaways

• Prefill and decode want different batching, parallelism and sometimes different GPUs.
• Colocation couples them. Chunking bounds the damage. Disaggregation removes the fight.
• Goodput under both TTFT and TPOT is the metric, not raw tokens/s.

Further learning

Next, cost the KV transfer that disaggregation adds.