Goodput

Raw throughput rises as you raise concurrency. Past a point you are still completing tokens, but users miss their latency targets. Goodput counts only the work that satisfied the contract.

The big idea

A highway can move more cars by packing them bumper to bumper. Throughput looks great. Nobody is happy. Goodput is the cars that still arrived on time.

The analogy breaks on tokens. A dual-H100 replica at 2,000 tok/s is not "faster" than a single H100 at 1,100 tok/s in efficiency terms. Divide by GPU, and by dollar-hour, before you compare hardware.

Walking through it

Useful throughput units:

• Output tokens/s per GPU or replica (decode-heavy chat).
• Input tokens/s (prefill-heavy RAG).
• Requests/s (misleading alone when lengths vary).

Normalize when comparing hardware. Tokens/s per GPU beat raw replica totals.

SLO: p90 TTFT < 500 ms AND p90 TPOT < 50 ms
goodput = rate of requests meeting both

A config that maximizes unrestricted tokens/s can have near-zero goodput if every stream is janky. DistServe and related papers popularized optimizing goodput explicitly. Production teams rediscover it the first time a sales demo stutters under load.

Suppose at concurrency 8 you meet SLOs with 1,200 goodput tok/s per GPU, and at concurrency 32 you get 2,000 unrestricted tok/s but p99 TPOT doubles past the SLO. Publish and autoscale using 1,200, not 2,000. The extra 800 tok/s are not product capacity.

Peak blog numbers usually come from huge batches, short outputs, warm caches and no TTFT constraint. They are not a fleet plan.

Requests/s with 5-token outputs is a vanity metric. So is "peak tok/s on ShareGPT at batch 256, ignore TTFT."

Common mistakes

• Capacity-planning from unrestricted tokens/s. The SLO knee is the real ceiling.
• Comparing replicas without dividing by GPU or dollar-hour.
• Celebrating a higher batch size after goodput fell.

Check yourself

1. Concurrency 8 holds the SLO at 1,200 tok/s/GPU. Concurrency 32 hits 2,000 tok/s but misses TPOT. What is capacity? Answer: 1,200 tok/s per GPU. The rest is not goodput.
2. Why is requests/s a weak headline number? Answer: Lengths vary. Five-token replies inflate the rate without proving the product works.

Key takeaways

• Throughput is completed work. Goodput is completed work that met the SLO.
• Plan and autoscale on goodput, not on peak tokens/s.
• Normalize by GPU and by cost before comparing hardware.
• Dual thresholds (TTFT and TPOT) are the usual contract.

Further learning

DistServe: goodput-optimized disaggregated servingDistServe improves the performance of large language models (LLMs) serving by disaggregating the prefill and decoding computation. Existing LLM serving systems colocate the two phases and batch the computation of pref…arXiv.org

Next, the latency–throughput curve is how you find the knee.