Platform failure drills

A serving platform is a multi-tenant compute service. The three failures you should rehearse are a dead host, a spike faster than cold start, and a bad model or LoRA rollout. Write the policy down. Product should know that 429s under overload are a safety valve, not flaky GPUs.

Walking through three incidents

GPU node loss. A host with 8 GPUs dies. You lose 4 TP=2 replicas. The load balancer drains as TCP dies. The gateway marks replicas unhealthy. The autoscaler replaces them in minutes if warm weights exist; otherwise TTFT degrades until cold start completes. Mitigation: N+ capacity, PodDisruptionBudgets, zone spreading, local weight caches on every node class. See replica drain.

Traffic spike. A marketing event doubles RPM. Waiting queue climbs → scale-up. If the spike rises faster than boot time, shed: reject batch, tighten per-tenant rates, optionally degrade to a smaller model for non-premium tenants. Practice this with load tests.

Bad model rollout. A new 70B checkpoint or LoRA fails evals in production (toxicity, tool schema breaks). Canary 5% traffic. Compare the eval harness, user thumbs and latent toxicity classifiers. Automatic rollback on eval delta or 5xx spike. Pin tenants to adapter versions. Never "latest" floating tags in prod.

When demand exceeds capacity, use the admission ladder: queue with a deadline, shed batch, degrade the model, disable expensive features.

Observability you need before the drill

Trace IDs from gateway → engine → tool calls. Alert on SLO burn rate and on “admission rejects > 0 for 5 minutes.”

Tenancy. Isolate admin APIs. Never let tenant A's prompt pull tenant B's LoRA. Scrub logs for secrets; token prompts often contain API keys pasted by users. Rate-limit by API key and by GPU-seconds, not only RPM. Sign model artifacts. Pin digests in the Deployment so a registry tag move cannot silently swap weights.

A realistic rollout is weeks, not a weekend: benchmark and FP8 eval first (1–2 weeks), then single-region chat plus LoRA, then gateway quotas and prefix routing, then autoscaling and a warm pool, then the batch tier and cost dashboards, then multi-AZ drills. Hardware follows measured goodput.

The big idea

Fire drills only work if the extinguisher is already on the wall. Warm weights, a canary switch and a published 429 policy are the extinguisher. The picture breaks if you treat "the model got slower" as the first hypothesis: check reject rate and queue time before you retune kernels.

Common mistakes

• No warm weight cache. Every replace is a multi-minute TTFT hole.
• Canary without evals. You only notice toxicity from users.
• RPM-only rate limits during a spike of huge RAG prompts.

Check yourself

1. An 8-GPU node dies and replicas are TP=2. How many replicas vanish? Answer: four.
2. What do you shed first when boot cannot keep up with a spike? Answer: batch and offline traffic, then tighten tenant rates, then degrade the model.

Key takeaways

• Rehearse node loss, spikes and bad rollouts.
• N+ capacity, warm weights and PDBs turn a dead node into a blip.
• Canary plus eval gates plus pinned digests contain a bad checkpoint.
• Overload policy belongs in the runbook, not in someone's head.

Further learning

Back to the serving platform design checklist.