Disaggregated prefill and decode

Every chat completion has two personalities. Prefill is fat and often compute-bound. Decode is skinny and memory-bound. Colocating them on one GPU — the default in early vLLM-style servers — makes them fight. A 32k-token prefill landing in a decode-heavy batch spikes everyone's TPOT. Chunking helps, but the hardware plan is still coupled.

Disaggregated serving splits the fleet: prefill workers build the KV cache, decode workers generate tokens, and a fast path ships KV between them. DistServe, Splitwise, Mooncake, NVIDIA Dynamo and llm-d all orbit this idea. It is not free. Under tight dual SLOs it can multiply goodput (requests that meet both TTFT and TPOT).

Concepts in this section

Read them in order. Each one builds on the one before.

The request path

1. The router sends the prompt to a prefill instance (queue depth, prefix-cache locality, or shard).
2. Prefill runs, possibly with TP tuned for TTFT, and materializes KV in paged blocks.
3. Metadata and KV blocks transfer to a decode instance (NIXL/UCX, Mooncake store, NVLink P2P, GPU-direct RDMA).
4. Decode streams tokens. Prefill frees or retains blocks per caching policy.

Treat chunked prefill as the default. Treat prefill/decode split as the upgrade when metrics say so: TPOT still spiky after chunking, different TP/CP meshes, or different GPU SKUs per phase.

Engine support is catching up: experimental or production P/D paths in vLLM V1, SGLang and cloud stacks.

In practice: plot TTFT and TPOT vs QPS on one colocated engine; estimate KV bytes for p50/p95; prototype with the engine's connector; split autoscaling signals; add KV-aware routing; load-test a mix of short decode-heavy traffic and rare huge prefills; keep a rollback flag.

Common mistakes

• Splitting before chunked prefill is exhausted. Most dual-SLO pain dies there.
• Ignoring transfer time. 10 GB of cold KV on IB can spend the whole TTFT budget.
• One ratio and one scaler for both pools.

Key takeaways

• Prefill and decode want different batching, parallelism and sometimes different GPUs.
• Disaggregation removes interference at the cost of KV transfer and ops complexity.
• Size pools from GPU-seconds, not 50/50. Stay colocated until dual SLOs or cost force the split.

Further learning

The DistServe OSDI talk is the clearest short framing:

DistServe paperDistServe improves the performance of large language models (LLMs) serving by disaggregating the prefill and decoding computation. Existing LLM serving systems colocate the two phases and batch the computation of pref…arXiv.org Mooncake: A KVCache-centric Disaggregated Architecture for LLM ServingMooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI. It features a KVCache-centric disaggregated architecture that separates the prefill and decoding clusters. It also leverages th…arXiv.org NVIDIA DynamoA Datacenter Scale Distributed Inference Serving Framework - ai-dynamo/dynamoGitHub