Anatomy of LLM inference
The latency-throughput curve
As concurrency rises, tokens per second climb then flatten, per-user speed falls, and tail TTFT spikes. The SLO knee is the operating point.
Every replica has a curve, not a single speed. Raise concurrency and aggregate throughput climbs, then flattens. Per-user tokens/s fall. Tail TTFT climbs sharply near saturation. You pick an operating point on purpose.
Concept
Concurrency is how many in-flight requests share the replica. The SLO knee is the last concurrency where goodput still meets the latency contract. Past that point extra tokens are not product capacity.
The big idea
A single cashier speeds up as the line grows, then the line stops helping and people wait longer. GPUs do the same. Continuous batching moves the whole curve up and to the right. It does not remove the knee.
The analogy breaks on prefill waves. A steady 20 QPS of 500-token prompts is not the same as 5 QPS of quiet traffic plus bursts of 50 RAG jobs. The same average utilization can hide synchronized prefills that pin tensor cores and pause decode.
Walking through it
Sweep concurrency (1, 2, 4, …) with a realistic mix of input and output lengths. Plot output tok/s, p50/p99 TTFT, and p50/p99 TPOT. Mark the concurrency where SLOs break. That is max useful load per replica.
| Concurrency | Throughput | Per-user tok/s | p99 TTFT |
|---|---|---|---|
| 1 | low | highest | flat |
| 16 | rising | falling | rising |
| 64 | flat | low | sharp climb |
Shapes are schematic. Measure your model.
Means lie on a shared GPU. Prefill bursts delay decode for everyone in the batch. Queueing before prefill grows nonlinearly as utilization approaches 1. One 100k-token prompt can steal the step budget if chunking is off. So p99 TTFT often blows up first as QPS rises, then p99 TPOT. Watching only p50 looks fine until tickets arrive.
| Goal | Tendency | Cost |
|---|---|---|
| Max tokens/s | High batch and concurrency | Bad TTFT/TPOT tails |
| Max snappiness | Low concurrency | Idle GPUs, high $/token |
| Max goodput | Medium concurrency at the SLO knee | Requires a real sweep |
Disaggregation and chunked prefill reshape the TTFT vs TPOT trade-off. They do not invent a free lunch.
At concurrency 8 you meet SLOs with 1,200 goodput tok/s per GPU. At concurrency 32 you get 2,000 unrestricted tok/s but p99 TPOT doubles past the SLO. Operate near 8. Capacity is offered goodput divided by goodput per replica, with headroom.
When testing, replay production traces or at least mix short and long inputs. A single synthetic length will lie about tails. Benchmarking covers the load generator.
Common mistakes
- Publishing the top of the throughput curve as capacity. That point is usually past the SLO.
- Sweeping one prompt length. Tails live in the mix.
- Ignoring KV utilization. A full cache is a leading indicator of preemption and TTFT collapse.
Check yourself
- Throughput is still rising at concurrency 64 but p99 TTFT has spiked. Are you at capacity? Answer: No. Capacity is the last point that still meets the SLO, not the throughput peak.
- Why does p99 TTFT usually break before the mean looks bad? Answer: Queueing and prefill interference hit the tail first. The average still includes the lucky short jobs.
Key takeaways
- Throughput climbs then flattens. Per-user speed falls. Tail latency spikes near saturation.
- Find the SLO knee with a concurrency sweep and a realistic mix.
- Continuous batching moves the curve. It does not remove the knee.
- Operate with headroom below the knee.
Further learning
Next, inference SLOs turns the knee into a contract you can page on.
Related
Inference metrics
TTFT, TPOT/ITL, end-to-end latency, throughput and goodput — definitions, percentiles and how they trade off as concurrency rises.
Goodput
Throughput is tokens you completed. Goodput is tokens that also met the latency contract. Capacity planning uses the second number.
Batching and continuous batching
Why batching amortizes weight reads, how static and dynamic batching waste GPU time, and how iteration-level continuous batching plus chunked prefill unlocks 10–20× throughput.