Replica drain and GPU failures

GPUs and NVLink links fail more often than "cattle" CPU nodes in folklore. Scale-down is just a planned version of the same event. If you cut a streaming response mid-token, the Deployment can still look healthy while the user sees an outage.

The big idea

A replica holds live generations and a warm KV cache. Ending the process is not like bouncing a stateless web worker. You must stop new traffic, let in-flight work finish or hand off, and only then terminate.

The restaurant-closing picture helps: lock the door, let seated tables finish, then turn out the lights. It breaks when tensor-parallel groups share fate. Losing one GPU can take down the whole shard, not one table.

Walking through a safe scale-down

1. Mark not-ready. Readiness should mean the engine accepts traffic and KV headroom and queue are within bounds. A "ready" pod at 98% KV will violate SLOs the moment the load balancer dumps more sessions on it.
2. Stop new requests. The gateway or Service drops the replica from the pool.
3. Drain. Let in-flight generations finish, or hand them to another replica if your stack supports it.
4. Terminate. Only then send SIGTERM.

Liveness stays simple: the process is up. Do not conflate it with readiness. A live process with a full KV cache is not ready.

Replica loss with TP>1. Losing one GPU can take down the whole shard group. Prefer N independent replicas, with tensor parallel as small as quality allows, over one giant TP=8 snowflake when availability matters.

Node replacement. Cordoning a bad GPU node should not delete the only copy of a warm weight cache without a plan to rebuild it elsewhere.

Common mistakes

• Killing a replica that is still streaming. The user sees a truncated answer. The dashboard still says healthy.
• Readiness = process up. A full KV cache should fail readiness so the balancer stops sending work.
• One TP=8 replica for a 70B. One GPU failure takes the whole endpoint down.

Check yourself

1. What should happen before a replica receives SIGTERM? Answer: stop new requests and let in-flight generations finish or hand off.
2. Why is a "ready" pod at 98% KV dangerous? Answer: the load balancer will send it more sessions and those will miss the SLO or OOM.

Key takeaways

• Drain streams on scale-down. Cutting a token stream is a user-visible outage.
• Readiness must include KV headroom and queue bounds, not only process liveness.
• Prefer more independent replicas over one large tensor-parallel group when availability matters.

Further learning

This is the last piece of autoscaling and fleet management.