Guide

Serving systems

Goodput and realistic workloads

Capacity is throughput that still meets the SLO, measured on your length mix, not on 512/128 prompts.

99 of 138 · 2 min read

A blog claims "2× faster" because they measured throughput at concurrency 1 with a warm prefix cache on 128-token prompts, while your production p99 is a 6k-token RAG prefill. The first job of benchmarking is to write down the workload and the SLO, so the number you publish is capacity you can sell.

Concept

Goodput is throughput that still meets your service-level objectives — for example tokens/s or requests/s where TTFT p95 ≤ 2 s and TPOT p95 ≤ 50 ms. Raw throughput that violates the SLO is a stress-test curiosity. See inference metrics for TTFT and TPOT.

The big idea

Tokens per second without a length mix or an SLO line is a trophy, not a plan. Two configs can print the same peak tok/s while only one stays under the latency you promised users.

A bathroom scale that you stand on after emptying your pockets is accurate and useless. You need the number with the pockets full.

Walking through a workload

Before you open a load generator, write down:

  • Input length distribution — tokens in the prompt, including system, tools and retrieved context.
  • Output length distribution — tokens generated before stop or EOS.
  • Arrival pattern — steady QPS, diurnal, spiky agent fan-out.
  • Prefix structure — shared system prompts vs unique per request. This drives cache hit rate.
  • Extras — multimodal images, JSON schema constraints, LoRA adapter IDs, streaming vs not.

Synthetic fixed lengths (every request 512 in / 256 out) are fine for relative comparisons while tuning one knob. They are terrible for absolute capacity planning. Real traffic has a long tail: one 32k-context request can stall TTFT for everyone else if chunked prefill and admission control are off.

ShareGPT-style public chats are better than fixed lengths for general chat, but they under-represent RAG (long in, short out), code agents (repeated prefixes, medium out) and batch summarization (long in, long out). Best: sample anonymized length histograms from production for a week and replay those distributions.

Suppose API logs show prompt tokens at p50=800, p95=6,000, p99=24,000 and completion tokens at p50=180, p95=900. A benchmark that only uses 512/128 will overstate concurrency headroom by a large factor, because KV memory and prefill time scale with the tail. Draw (input_len, output_len) from the empirical CDF, or bucket traffic into three shapes (short chat, RAG, long doc) and weight them 50/35/15 to match revenue-critical paths.

Common mistakes

  • Publishing peak tok/s with no SLO. That number must not appear on an exec slide without the latency next to it.
  • Using only 512/128 prompts. The tail is what fills KV and stalls TTFT.
  • Ignoring prefix structure. A shared system prompt with cache on is a different workload from unique prompts.

Check yourself

  1. What is goodput? Answer: throughput at which all latency SLOs still pass.
  2. Why does a 512/128 benchmark overstate capacity for a service whose p95 prompt is 6,000 tokens? Answer: KV and prefill scale with the tail, so real concurrency headroom is much smaller.

Key takeaways

  • Write the length mix, arrival pattern and prefix structure before generating load.
  • Optimize for max goodput under the SLO, not peak tokens/s.
  • Replay production length histograms, or at least a weighted mix of chat, RAG and long doc.

Further learning

Next, pick how you drive load in closed vs open loop.

Benchmarking inference

Measuring honestly: workloads, load generators, concurrency sweeps and the mistakes that make numbers meaningless.

Inference metrics

TTFT, TPOT/ITL, end-to-end latency, throughput and goodput — definitions, percentiles and how they trade off as concurrency rises.

Closed-loop vs open-loop load

Fixed concurrency finds the latency–throughput curve. An arrival process shows what happens when QPS exceeds goodput.