Inference cold starts

When a new replica is scheduled, the clock that matters is time until it can serve within the SLO, not time until the container PID exists. For a 70B-class checkpoint that gap is minutes. Reactive scale-up from "the queue just went red" is always late by roughly that entire boot budget.

The big idea

A web app boots in seconds. An LLM replica must pull a huge image, move tens of gigabytes of weights into HBM, warm the allocator, restore compile caches, and capture CUDA graphs. Until that finishes, the replica cannot take traffic. You have to add capacity before the ramp, not after.

The picture of "spin up another box when load rises" breaks here: the box cannot serve until the weights are on the GPU.

Walking through a 70B boot

Orders of magnitude as of when this was written; your numbers will vary.

1. Node + driver — 1–5 minutes if the GPU node is not already warm.
2. Image pull — 30 s to several minutes unless the node already has the layer cache.
3. Weight load — tens of GB from object storage into host RAM, then into HBM. A 70B FP8 model is on the order of ~70 GB of weights alone. 100+ GB including tokenizer, compile caches and extras is common.
4. CUDA / engine init — allocator warmup, torch.compile or kernel cache restore, CUDA graph capture across batch sizes.
5. Readiness — first real request to prove the path. This is often when you discover a missing secret or a wrong tensor-parallel size.

Suppose each warm replica goodputs 40 req/s within SLO, current load is 200 req/s (5 replicas at target), and load will grow 15%/min for 10 minutes while a new replica takes 6 minutes to become ready.

Demand in 6 minutes = 200 × (1.15^6) ≈ 463 req/s
Replicas needed     ≈ 463 / 40 ≈ 12

You must already be running enough extras, or have a warm pool, to cover the gap. Otherwise TTFT collapses while pods are still downloading weights.

Mitigations that move the needle

• Local NVMe weight caches. Pin checkpoints on the node. Network-to-GPU is the usual killer; disk-to-GPU with fast safetensors loaders is much kinder.
• Persistent compile / graph caches. Point VLLM_CACHE_ROOT (or the engine's equivalent) at a volume that survives pod death, so you do not recapture CUDA graphs on every scale event.
• Weight streaming and sleep/wake. Some stacks park weights in host RAM and wake into GPU. Useful for multi-model hosts and paused replicas.
• Process / GPU snapshotting. Some vendors snapshot a fully initialized engine and restore it 3–5× faster than a cold load. Treat this as an advanced fleet feature.
• Warm pools. Keep N idle-but-loaded replicas, or nodes with image and weights pre-staged, sized to absorb the cold-start horizon. You pay for idle GPUs; you buy SLA headroom.

Common mistakes

• Treating "pod Running" as "can serve." Weights and graphs may still be loading.
• Reactive HPA only. If traffic ramps 20% per minute and boot is seven minutes, you needed spare capacity before the ramp.
• 30-second startup probes. They kill healthy pods that are still loading a 70B checkpoint.

Check yourself

1. Load is 200 req/s, each replica holds 40 req/s, and a replica takes 6 minutes to boot while load grows 15%/min. About how many replicas do you need in six minutes? Answer: about 12, from 200 × 1.15^6 ≈ 463 req/s.
2. Why is a local NVMe weight cache worth more than a faster scaler? Answer: the usual killer is moving tens of GB onto the GPU, not the time to schedule the pod.

Key takeaways

• Ready means serving within SLO, not that the container started.
• 70B-class boots are measured in minutes: node, image, ~70–100 GB of weights, then graphs.
• Size warm pools and reserved capacity to the cold-start horizon times the ramp rate.

Further learning

Next, decide whether idle models should scale to zero.