Anatomy of LLM inference
The decode phase
Decode emits one token per step, re-reads the weights and the growing KV cache, and is usually memory-bandwidth-bound. It sets time per output token.
After the first token, generation is autoregressive: each new token depends on all previous ones. The engine runs a forward pass that reads essentially all model weights from HBM, reads the growing KV cache, computes queries for the new position, and produces one token.
Only one new position is computed (or a few, with speculative decoding), so math per weight byte is tiny. Decode is typically memory-bandwidth-bound. That sets time per output token (TPOT).
Concept
Memory-bound means the chip is waiting on HBM. Fewer bytes, or more bandwidth, helps. Extra TFLOPS sit idle. See the roofline model.
The big idea
Decode is like writing a reply one word at a time, and re-reading the whole dictionary before every word. The dictionary does not get smaller. The page of notes (the KV cache) gets longer.
The picture breaks once you batch. At batch B, one weight read serves B sequences. Aggregate tokens per second climb. Each sequence still advances only one token per step. A busy cluster can show high bandwidth utilization while a single idle user still tops out near the memory-bound floor.
Walking through the time
t_decode ~ weight_bytes / HBM_bandwidth (small batch)A user who streams a long answer pays this cost on every token. Agents that generate thousands of reasoning tokens feel it as wall-clock time and as billable output tokens.
Example
Same 70B FP8 replica, from inference arithmetic:
Decode 4,000 tokens at about 10 ms/token single-stream is on the order of 40 s.
Equal token counts to a 4,000-token prefill (~0.5 s), wildly different wall time.
| Writes | Reads heavily | Downstream effect |
|---|---|---|
| One new KV column per step; next token | Weights every step; full KV so far | Extends the answer; grows memory |
Speculative decoding attacks the same wall from another angle: draft several tokens, verify them in one pass, and keep accepted tokens. That is more progress per weight read when the draft is accurate.
Common mistakes
- Using peak FLOPs to predict chat speed. Single-stream decode is limited by memory bandwidth.
- Expecting batching to make one user faster. Batching raises fleet tokens/s. One stream still waits on one step per token.
- Forgetting KV grows. Each step re-reads more cache, so late tokens in a long answer can be slower than early ones.
Check yourself
- Why can 4,000 output tokens take ~40 s when 4,000 input tokens took ~0.5 s on the same 70B replica? Answer: prefill reuses each weight load across the prompt. Decode re-reads the weights for every output token.
- You raise the batch from 1 to 8 and fleet tokens/s climb, but one user's stream does not speed up. Is that a bug? Answer: no. Each sequence still advances one token per step. Batching amortizes the weight read across users.
Key takeaways
- Decode emits one token per step and re-reads weights and KV every time.
- It is usually memory-bandwidth-bound and sets TPOT.
- Batching raises aggregate tokens/s without changing the one-token-per-step rule.
- Output tokens dominate interactive latency once the first token has arrived.
Further learning
Next, how the two phases share a GPU: prefill vs decode. For the single-stream floor, see decode latency floor.
Related
Prefill vs decode
The two phases of LLM inference. Prefill is compute-bound and sets TTFT. Decode is memory-bound and sets TPOT. They interfere when they share a GPU.
The prefill phase
Prefill reads the whole prompt in parallel, writes the KV cache, and emits the first token. It is usually compute-bound and sets time to first token.
Inference arithmetic
Napkin math for any model on any GPU: weight memory, decode and prefill time, batch limits and KV budget.