Guide

Anatomy of LLM inference

TTFT and TPOT

Time to first token is queue plus prefill. Time per output token is decode smoothness. End-to-end latency is those two put together.

38 of 138 · 2 min read

Users do not feel "tokens per second" on a dashboard. They feel how long they wait for the first word, and whether the rest of the stream arrives smoothly. Those two waits have names: TTFT and TPOT.

Concept

Prefill reads the prompt and builds the KV cache. Decode writes one new token per step. A percentile such as p99 is the value below which that fraction of requests fall. The tail is what angry users and multi-step agents hit.

The big idea

A restaurant gives you bread, then the rest of the meal one plate at a time. TTFT is the wait for bread. TPOT is the gap between plates. End-to-end is when you leave.

The analogy breaks under load. The kitchen can be fine while the queue at the door dominates TTFT. That wait is not "the model got slower." The fleet is full.

Walking through it

Time to first token (TTFT). Wall time from request accept until the client receives the first output token. Includes queue time, prefill, sampling and network. Dominated by prefill plus queueing under load.

Time per output token (TPOT) / inter-token latency (ITL). Time between consecutive output tokens once streaming has started. Reflects decode step time and interference from other work. For one stream, tokens/s ≈ 1 / TPOT.

End-to-end latency. Time until the last token (or until the client stops). For N output tokens:

t_e2e ≈ TTFT + TPOT × (N − 1)

Exact indexing varies if TTFT already includes the first token. Be consistent.

TTFT = 400 ms, TPOT = 25 ms, N = 200 output tokens:

t_e2e ≈ 0.4 + 0.025 × 199 ≈ 5.4 s

Halving TPOT saves about 2.5 s. Halving TTFT saves 0.2 s. For long answers, optimize decode. For a snappy typing indicator, optimize TTFT and the queue.

TTFT ≈ t_queue + t_tokenize + t_prefill + t_sample + t_first_byte_network
TPOT ≈ t_decode_step + t_sample + t_stream_flush + t_interference

Under light load, t_prefill dominates for long prompts. Under heavy load, t_queue often dominates. Cold starts (new replica, CUDA graphs not captured, weights loading) inject multi-second tails.

t_interference is the stall from colocated prefills or scheduler preemptions. It is invisible if you only time a microbenchmark with one request. Flushing every 10 ms instead of every token also changes client-perceived ITL without changing GPU math.

Common mistakes

  • Timing only the GPU and ignoring the gateway queue. Users feel the sum.
  • Using one "latency" number for short and long jobs. The mix hides both knobs.
  • Chasing TTFT on a product that spends five seconds in decode. Do the arithmetic first.

Check yourself

  1. TTFT is 400 ms, TPOT is 25 ms, and the reply has 200 tokens. About how long is e2e? Answer: 0.4 + 0.025 × 199 ≈ 5.4 s.
  2. p50 TTFT looks fine but p99 explodes. Where do you look first? Answer: Queue time, cold starts, and one huge prefill stealing the step budget.

Key takeaways

  • TTFT ≈ queue + prefill. TPOT ≈ decode smoothness.
  • e2e ≈ TTFT + TPOT × (N − 1). Long answers are a decode problem.
  • Split queue vs prefill vs decode in traces before you tune.
  • Averages hide the tail. Histogram TTFT and TPOT separately.

Further learning

Next, goodput counts only the streams that met both waits.

Inference metrics

TTFT, TPOT/ITL, end-to-end latency, throughput and goodput — definitions, percentiles and how they trade off as concurrency rises.

Prefill vs decode

The two phases of LLM inference. Prefill is compute-bound and sets TTFT. Decode is memory-bound and sets TPOT. They interfere when they share a GPU.

Goodput

Throughput is tokens you completed. Goodput is tokens that also met the latency contract. Capacity planning uses the second number.