Anatomy of LLM inference
Inference metrics
TTFT, TPOT/ITL, end-to-end latency, throughput and goodput — definitions, percentiles and how they trade off as concurrency rises.
Dashboards that only show "tokens per second" hide the product. Users feel time to first token and smoothness of the stream. Finance feels tokens that met the contract. The right metrics separate those, expose queueing tails, and tell you whether to add GPUs or change the batcher.
Prefill vs decode is what shows up in the numbers. Deeper load-generator craft lives in benchmarking. Objective-setting in the SRE sense lives in SLOs.
Humans read on the order of 5–10 tokens per second. Chat therefore cares about a snappy first token and a steady 20–50 ms TPOT. Agents wait on end-to-end completion and will trade first-token polish for more goodput. One number cannot serve both products.
Concepts in this section
| Step | Concept | What it answers |
|---|---|---|
| 1 | TTFT and TPOT | Which wait is first-token, and which is the stream? |
| 2 | Goodput | Which tokens count as real capacity? |
| 3 | The latency-throughput curve | Where does a bigger batch stop helping users? |
| 4 | Inference SLOs | How do you write a contract you can page on? |
What to put on the dashboard
- TTFT histogram (ms), p50/p90/p99.
- TPOT / ITL histogram (ms), p50/p90/p99.
- Output and input tokens/s per replica.
- Goodput under your dual SLO.
- Queue time vs prefill time vs decode time.
- KV cache utilization — leading indicator of preemption and TTFT collapse.
- Batch size / num sequences.
- Cache hit rate if prefix caching is on.
Reject vanity metrics: peak tok/s with batch 256 and no TTFT bound, or requests/s with 5-token outputs.
Relate each swing back to hardware. When TTFT rises with input length on an idle box, you are seeing prefill FLOPs. When TPOT is flat across output length at batch 1, you are seeing the memory-bound decode floor. When TPOT rises with concurrency, you are seeing batching and interference. Keep that mapping in the runbook so the next on-call engineer inherits it.
Product will ask for one latency number. Give e2e at a stated output length plus the two streaming metrics, or every quarter's optimization will chase the wrong knob.
Common mistakes
- Timing only the server GPU and ignoring client or gateway queue.
- Using mean latency across mixed short and long jobs.
- Reporting tok/s while including idle gaps between requests.
- Warm-cache numbers presented as cold-start performance.
- Comparing two engines with different output lengths or stop criteria.
- Celebrating higher batch size when goodput fell.
Key takeaways
- TTFT ≈ queue + prefill. TPOT ≈ decode smoothness. e2e ≈ TTFT + TPOT × (N − 1).
- Throughput without SLOs is not capacity. Goodput is.
- p99 TTFT and TPOT expose interference and queueing that averages hide.
- Humans need about 5–10 tok/s to read. Agents need deadline-aware bulk tokens/s.
- Sweep concurrency, find the SLO knee, and operate there.
Further learning
LLM Inference Benchmarking: Fundamental Concepts (NVIDIA)This is the first post in the large language model latency-throughput benchmarking series, which aims to instruct developers on common metrics used for LLM benchmarking, fundamental concepts…NVIDIA Technical Blog
DistServe: goodput-optimized disaggregated servingDistServe improves the performance of large language models (LLMs) serving by disaggregating the prefill and decoding computation. Existing LLM serving systems colocate the two phases and batch the computation of pref…arXiv.org
Prefill vs decode and why TTFT and TPOT move differently:
Related
TTFT and TPOT
Time to first token is queue plus prefill. Time per output token is decode smoothness. End-to-end latency is those two put together.
Goodput
Throughput is tokens you completed. Goodput is tokens that also met the latency contract. Capacity planning uses the second number.
Benchmarking inference
Measuring honestly: workloads, load generators, concurrency sweeps and the mistakes that make numbers meaningless.