Production and economics
Scaling signals for LLM servers
Scale on waiting queue, KV cache and TTFT, not on CPU or GPU utilization.
A chat replica can look "busy but fine" on GPU utilization while users wait. Decode is memory-bandwidth bound, so the SMs stay occupied even when the replica still has free KV slots. The opposite happens too: the waiting queue grows and KV sits at 90%, while the util gauge looks the same as an underloaded box.
Autoscaling is a control loop: measure demand, compare it to a target, add or remove capacity. For GPUs the hard part is the measurement. The metric must track the thing that limits goodput, not a proxy borrowed from CPU web services.
Concept
Goodput is throughput that still meets the SLO, such as TTFT p95 under 2 s. Raw tokens/s that miss the SLO is not capacity you can sell. See inference metrics.
The big idea
CPU autoscalers assume "hot" means "needs more replicas." On an LLM server that assumption fails both ways. A long decode pins SMs while KV is half empty. A replica at the real limit — queue growing, KV almost full — often shows the same util percentage, because it is waiting on HBM, not starving for compute.
Memory gauges mix weights, KV and fragmentation. High VRAM can mean a healthy full batch, or an OOM on the next long prompt.
Walking through the signals
Scale on signals that track user pain and admission control.
| Signal | Scale up when | Why it works |
|---|---|---|
| Waiting queue depth | Sustained above a small N per replica | Direct backlog; drives TTFT |
| Queue time / TTFT percentiles | p95 queue time eats the TTFT SLO | Ties capacity to the SLO you sell |
| KV cache utilization | Sustained above about 0.85–0.90 | Predicts OOM and forced preemptions |
| In-flight tokens | Near max_num_batched_tokens | Captures batch capacity, not SM busy% |
| 503 / admission rejects | Non-zero sustained rate | You are already shedding load |
Treat GPU util as a debug metric, not a scaler input. Pair pod autoscaling (KEDA into HPA, or a custom controller) with node provisioning. Pending GPU pods help nobody if the cluster autoscaler still needs five minutes for an H100 node.
Common mistakes
- Scaling on GPU util. Decode looks busy while the replica could still take work, or looks the same when KV is full.
- Treating high VRAM as healthy. It can also mean the next long prompt will OOM.
- Autoscaling pods without nodes. The scaler adds replicas that sit Pending until a GPU node appears.
Check yourself
- A replica shows 90% GPU util and a growing waiting queue. What do you scale on? Answer: the queue (and KV / TTFT), not the util percentage.
- Why can two replicas show the same GPU util while only one is saturated? Answer: decode is memory-bound, so SMs stay busy whether KV still has room or not.
Key takeaways
- Scale on queue depth, KV utilization, in-flight tokens and TTFT, not CPU or GPU busy%.
- GPU util is a debug signal. It does not tell you if the replica can admit more work.
- Pod autoscaling only helps if GPU nodes appear in time.
Further learning
Next, see how long a new replica takes in cold starts.
Related
Autoscaling and GPU fleet management
Scale GPU replicas on queue depth, KV pressure and SLOs — not GPU util — then plan fleets around cold starts, warm pools and heterogeneous hardware.
Prefill vs decode
The two phases of LLM inference. Prefill is compute-bound and sets TTFT. Decode is memory-bound and sets TPOT. They interfere when they share a GPU.
Inference metrics
TTFT, TPOT/ITL, end-to-end latency, throughput and goodput — definitions, percentiles and how they trade off as concurrency rises.