Inference SLOs

A single "latency" number on a scorecard will send the team after the wrong knob every quarter. LLM serving needs a dual SLO: a first-token bound and a per-token bound, each at a stated percentile, next to a stated workload.

The big idea

Chat wants bread on the table fast and a steady stream after that. Agents want the whole meal by a deadline and will tolerate a slower first plate. One GPU pool with one SLO rarely serves both. Use separate deployments or priority classes. See routing and scheduling.

Humans read English prose on the order of 5–10 tokens per second. Streaming faster than that barely improves comprehension. Many consumer chat apps target TPOT in the 20–50 ms range (about 20–50 tok/s) with TTFT under a few hundred milliseconds when idle.

Agents consume tokens programmatically. A reasoning model at 30 tok/s generating 3,000 tokens needs about 100 s of decode alone. For that product, tokens/s and goodput under a deadline matter more than "feels typable."

Walking through it

1. Pick percentiles that match product risk (often p90 or p95 for chat; p99 for payments-adjacent tooling).
2. Set TTFT from UX research or competitive parity, for example first token before about 500 ms idle.
3. Set TPOT from reading comfort or agent deadlines, for example ≤ 50 ms ≈ ≥ 20 tok/s.
4. Define goodput as the request rate meeting both.
5. Autoscale and page on goodput headroom and p99 burn rate, not on GPU utilization alone.

Utilization can be high while goodput is fine (healthy decode) or low while users suffer (stuck on a few huge prefills).

SLO: p90 TTFT ≤ 500 ms, p90 TPOT ≤ 40 ms

In a concurrency sweep, concurrency 12 is the last point both hold, delivering 900 goodput tok/s/GPU. Run at concurrency about 8–10 in steady state. The autoscaler adds replicas when goodput demand exceeds about 70% of that budget. A slide that claims 1,800 tok/s at concurrency 48 is irrelevant to capacity.

Document the workload next to the SLO: input/output length percentiles, cache-hit assumptions, model and precision. An SLO without a workload is a slogan.

Give product managers e2e at a stated output length plus the two streaming metrics. When a page fires on "high latency," ask which metric. TTFT-only points to prompts, queues, cold starts or prefill capacity. TPOT-only points to batch pressure, KV thrash or prefill interference. Both rising usually means overload.

Common mistakes

• One latency number for the scorecard. Optimizations then chase the wrong knob.
• Autoscaling on GPU utilization. It does not tell you whether goodput is healthy.
• Copying a chat SLO onto an agent farm. The product is different.

Check yourself

1. Why write TTFT and TPOT as two clauses instead of one e2e number? Answer: They fail for different reasons and need different fixes. E2e hides which one moved.
2. GPU util is 90% and users are fine. Should you add replicas? Answer: Not from util alone. Add them when goodput headroom is gone.

Key takeaways

• Dual thresholds: TTFT and TPOT, at a percentile, with a named workload.
• Chat and agents want different points on the curve. Split the pools if you must.
• Autoscale on goodput, not on utilization.
• Map TTFT to prefill/queue and TPOT to decode/interference in the runbook.

Further learning

LLM Inference Benchmarking: Fundamental Concepts (NVIDIA)This is the first post in the large language model latency-throughput benchmarking series, which aims to instruct developers on common metrics used for LLM benchmarking, fundamental concepts…NVIDIA Technical Blog

Back to the inference metrics hub, then benchmarking for the load-generator craft.