Production and economics
Autoscaling and GPU fleet management
Scale GPU replicas on queue depth, KV pressure and SLOs — not GPU util — then plan fleets around cold starts, warm pools and heterogeneous hardware.
A chat endpoint that looks healthy at noon can miss its TTFT SLO by 4 pm because three replicas are stuck with full KV caches while CPU and GPU utilization still read "busy but fine." Autoscaling for LLM inference is not web-app autoscaling. The scarce resources are HBM and tokens in flight. Cold starts take minutes. The wrong scale-down kills live streams.
The control loop is the same as any scaler: measure demand, compare it to a target, add or remove capacity. The measurement is what changes. Pair this with cost per token when you turn capacity into dollars, and with GPU infrastructure when you need the node and driver layer.
Concepts in this section
Read them in order. Each one builds on the one before.
| Step | Concept | What it answers |
|---|---|---|
| 1 | Scaling signals | Why do CPU and GPU util lie, and what should you scale on? |
| 2 | Cold starts | How long until a new replica can serve, and how do you hide that? |
| 3 | Scale-to-zero | When is it safe to let the last replica exit? |
| 4 | GPU fleet planning | Reserved vs burst, and which GPU class for which model? |
| 5 | Replica drain | How do you scale down or lose a GPU without cutting streams? |
Why this is a production problem
A replica at the real limit — waiting queue growing, KV at 90%+ — often shows the same GPU util as an underloaded one, because decode is memory-bound. Scaling on that gauge adds replicas too late, or not at all.
Even a correct signal is late if boot takes six minutes. Image pull, ~70 GB of 70B FP8 weights, compile caches and CUDA graphs all sit on the critical path. Startup probes copied from a Node service will kill the pod while it is still loading.
The weekly job looks like this: read concurrency and TTFT against the benchmark envelope; set KEDA thresholds on queue and KV; keep a warm pool sized to cold-start × ramp rate; review reserved vs on-demand mix against last month's GPU bill.
Common mistakes
- Scaling on GPU util. It does not tell you if the replica can admit more work.
- Reactive HPA with no warm pool. Traffic ramps in minutes; 70B boots take minutes more.
- Scale-to-zero on interactive paths. The first request after idle pays the full cold start.
- SIGTERM while a stream is live. The Deployment looks healthy. The user sees a truncated answer.
Key takeaways
- Scale on queue depth, KV utilization, in-flight tokens and TTFT, not busy percentage.
- Cold starts are measured in minutes. Size warm pools to that horizon.
- Interactive paths keep minReplicas or a shared multi-LoRA host. Scale-to-zero is for batch and long-tail.
- Fleet shape is reserved baseline plus burst, with models pinned to the right GPU class.
- Drain streams on scale-down. Treat GPU failure as a first-class reliability case.
Further learning
KubeCon EU 2025 talk on autoscaling LLM workloads with OpenTelemetry + KEDA on KV-cache and queue metrics:
KEDA maintainers on event-driven scaling beyond CPU HPA:
KEDAApplication autoscaling made simpleKEDA
Related
Scaling signals for LLM servers
Scale on waiting queue, KV cache and TTFT, not on CPU or GPU utilization.
The economics of inference
Turn $/GPU-hour into $/million tokens with utilization and throughput, then decide API vs self-host and price products with real margins.
GPU infrastructure
Provisioning and scheduling GPUs for training and inference, covering hardware and fabrics, Kubernetes device plugins, gang scheduling, failures and checkpoints. Start here for the map of GPU pages.