Capacity from goodput

Capacity planning starts from traffic, not from a GPU SKU. Convert requests per minute → tokens per second → replicas, using the tokens/s a replica can sustain while still meeting SLOs. That number is goodput, not the peak tokens/s on a vendor slide.

Numbers below are worked assumptions for teaching. Re-benchmark on your GPUs before you buy anything.

Walking through a 50k RPM peak

Peak load: 50,000 requests/min ≈ 833 req/s aggregate. Assume interactive peak is 70% of that ≈ 580 req/s, average 400 output tokens and 2,000 input tokens (mix of chat + RAG):

Output tok/s ≈ 580 × 400 = 232,000 tok/s
Input tok/s  ≈ 580 × 2,000 = 1,160,000 tok/s  (prefill pressure)

Prefill tokens dominate FLOPs. Output tokens dominate decode memory traffic and user-visible speed. You need enough decode capacity for 232k tok/s goodput.

Suppose a concurrency sweep on vLLM, Llama-3-class 70B FP8, TP=2 on 2×H100, prefix caching on, shows ~4,500 output tok/s goodput per replica at the concurrency that still meets p90 TTFT 800 ms / TPOT 40 ms. Higher concurrency raises tok/s but breaks TTFT — that region is not usable for interactive.

output_tok_s = 580 * 400
goodput = 4500
print(round(output_tok_s / goodput, 1))

Output:

51.6

Interactive replicas needed ≈ 52. GPUs ≈ 52 × 2 = 104 H100s for the colocated interactive path. Add ~20–30% headroom for failure, deploys and diurnal mis-prediction → ~65 replicas / ~130 GPUs before the batch tier.

If measured goodput is only 3,000 tok/s/replica under the real RAG length mix, need 232k/3k ≈ 77 replicas before headroom. Always recompute from your benchmark.

At $2.50 / H100-hour and 130 GPUs:

$325 / hour ≈ $237k / month if 24/7 reserved
At 232k output tok/s sustained: tok/hour ≈ 835e9
$/M ≈ 325 / 835 ≈ $0.39 / M output  (idealized full util)

Real utilization 50–70% plus idle headroom pushes ~$0.55–0.80 / M before eng overhead — the number you take to pricing. See cost per token.

Do not buy 130 GPUs before the concurrency sweep. Hardware follows measured goodput.

Common mistakes

• Dividing by single-stream tok/s from a blog. That is the floor, not fleet capacity.
• Using the concurrency that maxes tok/s even though TTFT already missed the SLO.
• Skipping headroom. Node loss and deploys happen on the day of the launch.

Check yourself

1. 580 req/s × 400 output tokens, 4,500 tok/s goodput. About how many replicas? Answer: about 52, before headroom.
2. Why is 8,000 tok/s at broken TTFT not the denominator? Answer: those tokens missed the SLO, so they are not interactive capacity.

Key takeaways

• RPM → tok/s → replicas, using SLO-constrained goodput.
• Prefill tok/s and decode tok/s are different pressures.
• Add 20–30% headroom after the replica math.
• Re-benchmark on the real length mix before procurement.

Further learning

Next, split that fleet into interactive and batch pools.