Concurrency sweeps

A useful benchmark is a concurrency sweep (or a QPS ladder). You do not pick one load and declare a winner. You walk the curve until you find the cliff, then mark the last point that still meets the SLO. That number — max goodput — feeds capacity planning and cost models.

The big idea

Low concurrency shows the physics of the model and GPU. Medium concurrency shows whether continuous batching is earning its keep. High concurrency shows when KV or compute saturates. Operate just below that knee, under the SLO line, with headroom for the p99 prompt, not only the p50.

A car dyno that only reports top speed hides whether you can drive it. The sweep is the whole tachometer.

Walking through a sweep

1. Warm up until KV pools, CUDA graphs and prefix caches are in steady state. Discard the first 30–120 seconds, or the first N requests. Compile, disk weight load and cache fill skew early samples.
2. Step concurrency through {1, 2, 4, 8, 16, 32, 64, ...} or QPS through a ladder around expected load.
3. At each step, record TTFT (p50/p95/p99), TPOT/ITL, end-to-end latency, input/output tok/s, error rate, and if available KV utilization and queue time.
4. Run each step long enough for p99 to stabilize — often 1–5 minutes at high concurrency. Short runs lie about tails.
5. Stop when error rate rises or latency is many× the SLO. Plot throughput on x, latency percentile on y, and mark the SLO.

Three regions appear on almost every honest sweep:

1. Latency floor — low concurrency. TTFT ≈ prefill plus a tiny queue. TPOT ≈ single-batch decode.
2. Efficient packing — medium concurrency. Throughput rises fast. Latency rises slowly.
3. Saturation — KV full or compute pegged. Queues grow. Percentiles bend up sharply.

Example. At concurrency 32 you measure 20k output tok/s and TTFT p95 = 1.1 s (SLO 1.5 s). At 48 you measure 23k tok/s but TTFT p95 = 2.8 s. Max goodput is closer to 20k than 23k. Buying 15% more peak tokens by violating the SLO is how on-call gets pages.

Three configs on the same 8×H100 node, same model, same workload file:

Without the SLO column, FP16 at absurd concurrency might still print a huge tok/s number while p95 TTFT is 8 s.

Pair the curve with GPU SM utilization and HBM bandwidth. If latency is already bad while the GPU is half idle, you have a host, networking or client problem — not an engine problem.

Common mistakes

• Skipping warmup. First-minute samples include compile and graph capture.
• One-minute steps at high concurrency. p99 has not settled.
• Taking the peak tok/s row. Max goodput is the last row under the SLO.

Check yourself

1. Concurrency 32: 20k tok/s, TTFT p95 1.1 s. Concurrency 48: 23k tok/s, TTFT p95 2.8 s. SLO is 1.5 s. What is max goodput? Answer: about 20k tok/s, the last point under the SLO.
2. Latency is bad and the GPU is half idle. Where do you look first? Answer: host, network or the load-generator client, not the engine.

Key takeaways

• Warm up, step load, plot latency vs throughput, mark the SLO.
• Max goodput is the highest point that still passes every latency SLO.
• Operate just below the knee, with headroom for the p99 prompt.

Further learning

Next, avoid the pitfalls that make the curve meaningless.