GPU fleet planning

Traffic is rarely flat. Chat products show diurnal curves. Enterprise APIs spike on business hours and month-end jobs. A sane fleet has three layers of capacity, and it does not put every model on an H100.

The big idea

Buy the capacity you are almost sure to use at a discount. Rent the rest when the day peaks. Keep spot or preemptible GPUs off the interactive path unless you can fail over instantly.

Think of a power company: baseload plants run all day, peaker plants fire at 6 p.m. The analogy breaks because GPU peakers take minutes to become ready. The reserved line has to cover the cold-start lag, not only the trough.

Walking through the three layers

1. Reserved / committed baseline — capacity you are almost sure to use, from the weekday trough to the peak shoulder. Discounted $/GPU-hour, predictable placement.
2. On-demand burst — the same SKU or compatible GPUs for the daily peak and for incidents. Higher price, elastic.
3. Spot / preemptible (optional) — only for batch, offline eval, or preemptible decode pools with drain logic. Never the sole home of interactive SLO traffic unless you have instant failover.

If your peak is 2.5× the trough and boot is slow, a baseline at trough + 30% still fails the evening spike unless a warm pool or predictive scale-up covers the lag.

Multi-cloud sourcing. H100, H200 and B200 availability and price vary across hyperscalers and GPU clouds. A common pattern: primary region on reserved capacity, secondary provider for burst, with the same container image and model digest pinned so failover is a routing change. Watch driver versions, NVLink topology and NIC bandwidth for tensor parallel.

Match the model to the silicon

Not every endpoint deserves a B200.

Put the right model on the right GPU with separate deployments or node selectors, not one mega-pool that schedules a 70B onto an L4 and OOMs. Routing then sends traffic to the pool that can meet the SLO at the lowest cost.

Common mistakes

• One GPU class for every model. A 70B on an L4 OOMs; an 8B on a B200 wastes money.
• Baseline sized to the trough. Evening traffic plus cold-start lag will miss the SLO.
• Spot as the only home for chat. A preemption is a user-visible outage unless failover is instant.

Check yourself

1. Peak is 2.5× trough and a replica takes several minutes to boot. Why is baseline = trough a bad plan? Answer: the evening spike arrives before new replicas are ready, so TTFT collapses.
2. Where should interactive chat live relative to spot GPUs? Answer: on reserved or on-demand capacity, unless you have instant failover.

Key takeaways

• Fleet shape is reserved baseline plus on-demand burst, with spot only for batch.
• Size the baseline for the cold-start horizon, not only average load.
• Pin each model to a GPU class that can meet the SLO at the lowest cost.

Further learning

Next, plan for node loss in replica drain. Turn capacity into dollars in cost per token.