KV transfer

After a prefill worker builds the KV cache, that cache has to reach a decode worker. The transfer is the tax you pay to stop prefill and decode fighting on one GPU. It is worth it only when interference costs more latency than the move, and when the payload is small or the link is fast.

The big idea

You cooked a meal in one kitchen and need it on another table before it goes cold. A cart across the hallway is fine for a plate. A banquet on a bicycle is not.

The analogy breaks for prefix cache hits and MLA-style compressed KV: the leftover payload can be tens of megabytes, which is milliseconds on NVLink or good RDMA.

Walking through the bytes

Let S be prompt tokens after cache hits, L layers, H_kv the KV hidden size per token per layer (accounting for GQA/MLA), B bytes per element.

bytes ≈ 2 × S × L × H_kv × B    (K and V; MLA compresses this a lot)

Dense GQA model, S = 4096 new tokens, L = 80, kv_heads × head_dim = 8 × 128 = 1024, FP16 KV (B = 2):

2 × 4096 × 80 × 1024 × 2 ≈ 10.7 × 10^9 bytes ≈ 10.7 GB

On 400 Gb/s IB (~50 GB/s line rate, less in practice): order 200+ ms just to move KV, before protocol overhead — painful vs a 200 ms TTFT budget.

The same prompt with MLA-style compressed KV or FP8 KV might be several× smaller. A 512-token prompt with a prefix-cache hit on the heavy system prompt might transfer only tens of MB — milliseconds on NVLink or good RDMA.

Rule of thumb: disaggregation pays when (a) prefill/decode interference costs you more than the transfer, and (b) you keep transfers on high-bandwidth links or small residual KV (short prompts, cache hits, compressed KV). DistServe's placement explicitly accounts for cluster bandwidth.

Besides raw K and V (or compressed MLA latents), the path carries block tables, request IDs, sampling state, and sometimes speculative draft state. Under-estimating metadata leads to "KV arrived but decode cannot start." Prefer engines that migrate an opaque execution-state blob.

Bandwidth planning should use payload × overhead. RDMA verbs, GPU-direct registration and encryption proxies can cut effective throughput in half versus brochure line rate. Measure with a synthetic KV blob of your p95 size between real node pairs before promising TTFT.

If prefill and decode share an NVLink domain (same HGX node or NVL72 rack), consider in-node disaggregation: some GPUs prefill-only, some decode-only, KV via NVLink P2P. You get isolation without crossing IB. Multi-node is for when one node cannot hold both pools at the ratio you need, or when decode should sit on a different SKU.

Engines differ on whether decode pulls KV or prefill pushes. Pull fits "decode owns the request." Push can hide latency if decode is pre-reserved.

Common mistakes

• Promising a 200 ms TTFT with 10 GB of cold KV on IB. The move alone can take 200+ ms.
• Planning on brochure line rate. Protocol and registration overhead can cut it in half.
• Forgetting block tables and sampler state. KV tensors arriving is not enough to start decode.

Check yourself

1. 4096 new tokens, 80 layers, H_kv=1024, FP16. About how large is KV? Answer: about 10.7 GB.
2. When is the transfer cheap? Answer: short residual KV (cache hits, small prompts, compressed KV) on NVLink or good RDMA.

Key takeaways

• Transfer bytes ≈ 2 × S × L × H_kv × B.
• Multi-GB cold KV on slow links kills TTFT. Small residual KV on fast links is milliseconds.
• Measure p95 payload on real node pairs. Include metadata, not only K and V.

Further learning

Next, size the two pools in prefill/decode ratio.