Preemption, swap and recomputation

Even with paging, the KV cache can outgrow GPU memory under heavy load. The scheduler then preempts some running sequences to free blocks. It has two options. It can swap the blocks to CPU memory and copy them back later. Or it can recompute: drop the blocks, and re-run the prefill when the sequence resumes.

The big idea

Picture a restaurant at full capacity. A diner is asked to wait. Their dishes can go in the fridge and come back when the table is free (swap). Or the kitchen can throw them out and cook the order again (recompute). Either way the diner waits longer, and the kitchen spends time on the same dish twice.

The picture breaks in two places. The fridge has limited space and a slow door, which is the PCIe link to host memory. Recooking uses kitchen time that could serve other tables.

Walking through a preemption

GPU KV pool: fully used
Request C needs 200 blocks, none free
Preempt request A, which holds 300 blocks and shares none:
  swap:       copy 300 blocks to CPU memory; copy back when A resumes
  recompute:  drop 300 blocks; re-run A's prefill when A resumes
Request C now runs

Paging makes both choices finer-grained. The engine can move or drop single blocks, not whole per-request slabs.

Choosing between them

Common mistakes

• Treating preemption as free. A preempted request pays in time to first token or in stutters.
• Preempting shared blocks first. A shared block hurts every sequence that uses it. Prefer unique tails.
• Ignoring preemption counts. Constant preemption means the server is oversubscribed. Add capacity or lower concurrency.

Check yourself

1. What does paging change about preemption? Answer: the engine can move or drop individual blocks, not whole per-request slabs.
2. When is recomputation the better choice? Answer: when re-running the prefill is cheap compared with moving the blocks over the host link, which is typical for moderate prompts.

Key takeaways

• Under memory pressure, the scheduler pauses sequences to free KV blocks.
• Swap moves blocks to host memory. Recomputation rebuilds them from the prompt.
• Moderate prompts usually favor recomputation. Very long contexts often favor swapping or tiered caching.
• Count preemptions. Constant preemption is a capacity signal.

Further learning

Next, read copy-on-write sharing if you have not already. For the long-context version of this trade-off, see serving long contexts.