Sizing prefill and decode pools

Once prefill and decode live on separate pools, the next question is how many GPUs each pool gets. A 50/50 split is almost never right. Interactive chat with short outputs still needs a large decode pool, because tokens stream for seconds while prefill was a one-shot. RAG with 20k-token prompts and 100-token answers is prefill-heavy.

The big idea

Staff a kitchen by how long each station is occupied, not by giving grill and pastry the same headcount. Prefill is a burst of GPU-seconds per request. Decode is GPU-seconds that stretch with output length and batching.

The analogy breaks if KV transfer does not fit the TTFT budget. Then the math says "win" and users still wait on the fabric.

Walking through the counts

Let λ be request rate, t_p mean prefill GPU-seconds per request, t_d mean decode GPU-seconds per request (tokens × TPOT / effective batching).

N_p ≈ λ × t_p / target_util_p
N_d ≈ λ × t_d / target_util_d

Tune from measured t_p, t_d and SLO burn rate — not from GPU-count symmetry. Autoscale the pools on different signals: queue time for prefill vs active decode slots / KV memory. See autoscaling.

Parallelism often diverges too. DistServe notes intra-op (TP) helps TTFT at lower load, while more replicas help capacity. Search TP/PP/DP separately for each pool.

Worked case. Target: TTFT p90 ≤ 2 s, TPOT p90 ≤ 50 ms, λ = 20 req/s. Mean prompt 2k tokens cold, mean output 500 tokens.

Colocated: each request occupies a GPU for prefill (~0.4 s) plus decode (~500 × 0.04 s = 20 s of wall time, multiplexed). Interference forces low utilization to protect TPOT — say you need 32 GPUs empirically.

Disaggregated:

Prefill: 20 × 0.4 = 8 GPU-seconds/s → ~10 GPUs at 80% util
Decode:  20 req/s × 20 s residency = 400 active sequences
         at 64 sequences/GPU → ~7 GPUs, padded to ~12 for KV and spikes
Total ≈ 22 GPUs before transfer overhead

That is a win only if KV transfer fits the TTFT budget and ops cost is acceptable. If p95 prompt is 32k tokens, redo the math; transfer and prefill both explode.

Agent workloads that prefill tools repeatedly stress prefix cache and prefill differently than single-shot chat. Do not reuse a chat ratio for agents.

Common mistakes

• 50/50 GPU splits. Chat is usually decode-heavy. Long RAG is prefill-heavy.
• One autoscaler for both pools. The signals are different: prefill queue time vs decode KV / slots.
• Reusing a chat ratio for agents. Tool loops change t_p more than t_d.

Check yourself

1. λ = 20 req/s, t_p = 0.4 s, target util 80%. About how many prefill GPUs? Answer: 20 × 0.4 / 0.8 = 10.
2. Same λ, 20 s average decode residency, 64 sequences per GPU. About how many decode GPUs before padding? Answer: 400 / 64 ≈ 7.

Key takeaways

• N_p and N_d come from GPU-seconds per request, not from symmetry.
• Autoscale the two pools on different signals.
• Recalculate when the length mix changes. A 32k p95 prompt changes everything.

Further learning

Next, the failure modes that only appear after you split.