Anatomy of LLM inference
Prefill vs decode
The two phases of LLM inference. Prefill is compute-bound and sets TTFT. Decode is memory-bound and sets TPOT. They interfere when they share a GPU.
Every chat completion is two different jobs glued into one API call. Prefill digests the prompt. Decode emits the reply one token at a time. They stress the GPU in opposite ways, they dominate different latency metrics, and when they share a batch they can ruin each other's SLOs.
The inference arithmetic page gave the napkin formulas. This section is the why: why input tokens and output tokens are not the same commodity, why a long RAG prompt can stall someone else's stream, and why engines invent chunked prefill and disaggregated pools.
Concepts in this section
| Step | Concept | What it answers |
|---|---|---|
| 1 | The prefill phase | Why is the prompt compute-bound? |
| 2 | The decode phase | Why is each output token memory-bound? |
| 3 | Prefill–decode interference | Why does a new prompt hitch other streams? |
| 4 | Workload shapes | Which products stress which phase? |
Why the asymmetry matters
| Prefill | Decode | |
|---|---|---|
| Parallelism over sequence | Yes, the prompt is known | No, one new token |
| Typical bottleneck | Compute | Memory bandwidth |
| Primary metric | TTFT | TPOT / tok/s |
| Scales with | Input length | Output length × steps |
| Batching effect | Larger GEMMs, still heavy | Amortizes weight reads |
This is why APIs often price input and output differently. Prefill turns one weight read into work for thousands of prompt tokens. Decode re-reads weights for each output token. A 10k-token system prompt plus a 20-token "yes" is cheap on decode and expensive on prefill. A 50-token question plus a 2k-token chain of thought is the opposite.
Example
Same 70B FP8 replica: prefill 4,000 tokens is about 0.5 s. Decode 4,000 tokens is about 40 s. Equal counts, different wall time.
Preview of the fixes
You do not need the full mechanisms yet, but you should know the map.
- Continuous batching — admit and finish sequences every step (batching).
- Chunked prefill — bound how much prefill work lands in one step.
- Paged KV — pack variable-length caches (PagedAttention).
- Disaggregation — specialize hardware pools (disaggregated serving).
- Speculative decoding — verify several draft tokens per weight read (speculative decoding).
All of them exist because prefill and decode are different programs that happen to share a model.
Common mistakes
- One latency SLO for both phases. "p50 latency under 2 s" hides a RAG spinner and a hitching agent stream.
- Treating input and output tokens as the same cost. They are not, in time or in money.
- Buying more copies of the same replica to fix a mixed workload. Prefill-heavy and decode-heavy traffic want different pools or at least different queues.
Key takeaways
- Prefill is usually compute-bound and drives TTFT. Decode is usually memory-bound and drives TPOT.
- Input versus output pricing follows this split.
- Long prefills on a shared GPU stall other users' decode streams.
- Chat, RAG, reasoning and agents stress different sides. Measure both.
Further learning
LLM Inference Performance Engineering (Databricks)databricks.comSplitwise and DistServe on splitting the two phases.
Related
The KV cache
What keys and values the cache stores, how big it gets per token, and how GQA, MLA, quantization and offloading shrink it.
Batching and continuous batching
Why batching amortizes weight reads, how static and dynamic batching waste GPU time, and how iteration-level continuous batching plus chunked prefill unlocks 10–20× throughput.
Disaggregated prefill and decode
Running prefill and decode on separate GPU pools so each phase gets its own parallelism, batching, and hardware — and when colocated chunked prefill is enough.