Disaggregation failure modes

Splitting prefill and decode removes interference and adds a new class of bugs. Prefill can succeed while decode dies mid-transfer. A decode worker can be too fragmented to start. The next turn can land on a replica that does not hold the warm KV. The two pools can drift to different quantization.

Stay colocated, with continuous batching, chunked prefill and prefix cache, until dual SLOs or cost force the split.

The big idea

Two kitchens sharing a cart will fail in ways one kitchen cannot: the cart tips, the second kitchen is full, yesterday's recipe changed. Colocation's failure mode is interference. Disaggregation's failure modes are coordination.

Walking through the unique failures

• Orphan KV / leak. Prefill succeeds, decode dies mid-transfer. Blocks pin HBM until TTL GC. Need hard leases on remote blocks.
• Slow-start decode. Transfer completes but the decode worker is KV-fragmented. First tokens miss TPOT. Pre-reserve decode slots before releasing prefill results.
• Cache affinity loss. The user's next turn routes to a different decode replica without the warm KV. You re-prefill or re-fetch. Sticky routing and prefix caching matter more, not less.
• Heterogeneous version skew. Prefill and decode must run bit-compatible model builds (same tokenizer, same RoPE scale, same quantization). Split fleets invite config drift.

Stay colocated when context lengths are modest and TTFT SLOs are loose; when you lack NVLink/RDMA-quality KV transport; when the team cannot own two autoscalers, a KV store and failure UX yet; or when the workload is throughput-batch and dual interactive SLOs do not matter.

Graduate when goodput curves show decode TPOT blowing up whenever prefills arrive, or when prefill wants H100/B200 FLOPs and decode is happier on denser, cheaper memory-bandwidth capacity.

A rollback flag that colocates again is part of the design, not an afterthought.

Common mistakes

• Splitting before chunked prefill is on. Exhaust the single-pool fix first.
• No lease on remote KV. Prefill success plus decode death leaks HBM.
• Different quant or RoPE on the two pools. The first token looks like a sampler bug.

Check yourself

1. Prefill succeeds and decode dies mid-transfer. What leaks? Answer: KV blocks on the prefill GPU, until a lease or TTL frees them.
2. When is colocated plus chunked prefill still the right default? Answer: modest context, loose TTFT, no fast KV fabric, or a small team.

Key takeaways

• Disaggregation adds orphan KV, slow-start decode, affinity loss and version skew.
• Prefill and decode builds must be bit-compatible.
• Keep a flag to colocate if the KV path flakes.

Further learning

Back to disaggregated serving.