The latency-throughput-cost triangle

Every serving decision sits on a three-way trade-off. You can make users wait less, you can push more tokens through each GPU, or you can spend fewer dollars per million tokens. You cannot max all three at once.

• Latency per user — how long until the first token, and how fast tokens stream after that.
• Throughput per GPU — how many tokens or requests the hardware produces per second.
• Cost — dollars per million tokens, from GPU-hours, utilization and software efficiency.

The big idea

A bakery can serve one customer at a time, or bake a full tray. The tray is cheaper per loaf and slower for the person who arrived first. The picture breaks when loaves have very different sizes: a 32k-token prompt is not a dinner roll.

Interactive chat wants small effective batches and low queueing. Offline eval and batch APIs want large batches and max tokens/s. Pick the point from the SLO, not from a peak benchmark.

Walking through a cost napkin

An H100-class GPU costs on the order of $2–4/hour retail cloud (as of when this was written; hyperscaler internal cost is lower). Suppose a tuned stack sustains ~3,000 output tokens/s on a mid-size model at high concurrency, and a naive stack sustains ~300 tokens/s at batch 1. Use $3/hour:

Cost per 1M tokens (tuned)  ≈ $3 / (3000 tok/s × 3600 s/hr) × 1e6
                            ≈ $3 / 10.8e6 × 1e6
                            ≈ $0.28 / M tokens

Cost per 1M tokens (naive)  ≈ $3 / (300 × 3600) × 1e6
                            ≈ $2.8 / M tokens

Same GPU, same model, about 10× in cost — before idle time, retries or over-provisioning. Raising batch size usually raises throughput and lowers $/token, but adds queueing delay and can hurt time-to-first-token. Shrinking or quantizing a model cuts cost and can cut latency, but may hurt quality. Buying more GPUs fixes latency under load until money or power runs out.

Common mistakes

• Chasing peak tokens/s. That number lives at huge batches your chat SLO will not allow.
• Treating hardware price as the only cost. Idle GPUs and retries move $/M more than the list price.
• Fixing latency only by buying GPUs. Software efficiency (batching, quantization, caching) is the cheaper first lever.

Check yourself

1. Why does raising batch size usually lower $/token but raise user wait? Answer: more sequences share one weight read, so throughput rises, but newcomers queue longer.
2. Using $3/hour and 300 tok/s, about what is $/M tokens? Answer: about $2.8 per million tokens.

Key takeaways

• Latency, throughput and cost pull against each other.
• Batch size is the primary dial that moves you along the curve.
• A 10× tokens/s gap on the same GPU is a 10× cost gap.
• Choose the point from the SLO, not from a vendor slide.

Further learning

Next, see how those choices sit in the serving stack. For the dollar formula, see cost per token.