Production and economics
Scale-to-zero for inference
Scale-to-zero saves GPU-hours on sparse models, and turns the first request into a multi-minute stall.
Scale-to-zero looks free until the first request after idle pays the full cold start. For a long-tail fine-tune that sees three requests an hour, that may be fine if callers accept multi-minute TTFT or a "warming up" screen. For an interactive product SLO, scale-to-zero is usually the wrong default.
Concept
Scale-to-zero means the last replica is allowed to exit when traffic is gone. The next request must boot a node, load weights and capture graphs before it can answer.
The big idea
Always-on burns dollars every hour, even at 3 a.m. Scale-to-zero burns the user's first request instead. You are choosing who pays the cold-start tax: the GPU bill, or the person who hits the endpoint after it went idle.
The analogy of "turn the lights off when you leave the room" breaks when turning them back on takes six minutes and the user is waiting on the first token.
Walking through the choice
| Path | What you pay | What the user sees |
|---|---|---|
| minReplicas ≥ 1 | GPU-hours 24/7 | TTFT stays in the SLO |
| Scale-to-zero | Almost nothing while idle | First request waits for the full boot |
| Shared multi-LoRA host | One warm base, many adapters | First hit for a cold adapter is a page-in, not a full reload |
A useful rule: keep minReplicas at least 1, or a shared warm multi-LoRA host, for anything on the critical path. Scale-to-zero only for batch, internal, or explicitly "best effort" models.
Multi-LoRA serving often beats N zero-scaled full copies. One warm base model can hold many adapters. You pay for one GPU and page adapters, instead of booting a 70B replica per tenant.
Trade-off
Always-on keeps TTFT honest and wastes money on sparse models. Scale-to-zero saves money and makes the first request a stall. Shared multi-LoRA is the usual middle path when many fine-tunes share one base.
Common mistakes
- Scale-to-zero on an interactive chat path. The first user after lunch gets a multi-minute TTFT and files a ticket.
- One full replica per long-tail adapter. Prefer a shared base with paging, not N cold 70B copies.
- Hiding the warm-up. If you do scale to zero, show a warming state. A silent hang looks like an outage.
Check yourself
- A support copilot is on the critical path and sees traffic all day. What should minReplicas be? Answer: at least 1, or a shared warm multi-LoRA host.
- When is scale-to-zero a reasonable default? Answer: batch, internal, or best-effort models whose callers accept a cold start.
Key takeaways
- The first request after scale-to-zero pays the full cold start.
- Interactive paths want minReplicas or a shared warm host.
- Many fine-tunes on one base beat many zero-scaled full copies.
Further learning
Next, plan reserved and burst capacity in GPU fleet planning.
Related
Autoscaling and GPU fleet management
Scale GPU replicas on queue depth, KV pressure and SLOs — not GPU util — then plan fleets around cold starts, warm pools and heterogeneous hardware.
Inference cold starts
A new replica is not ready when the pod starts: image, weights and CUDA graphs take minutes.
Multi-LoRA and multi-model serving
Serving many fine-tuned adapters and models on shared GPUs efficiently.