Anatomy of LLM inference
The prefill phase
Prefill reads the whole prompt in parallel, writes the KV cache, and emits the first token. It is usually compute-bound and sets time to first token.
When a request arrives, the engine must turn the prompt into two things: the first output token, and a filled KV cache for every prompt position. That work is prefill. Because the prompt is known up front, the model can process all prompt tokens in parallel, the same large matrix multiplies you see in a training forward pass.
Arithmetic intensity is high: each weight load is reused across many tokens. On modern GPUs, prefill is typically compute-bound. Prefill time is the main hardware contribution to time to first token (TTFT).
Concept
Compute-bound means the chip's math units are the bottleneck. Adding FLOPs, or shortening the work, helps. Adding memory bandwidth barely does. See the roofline model.
The big idea
Prefill is like reading a whole brief before you write the first sentence of a reply. You can scan every page at once. You cannot start the reply until that scan is done.
The picture breaks for tiny prompts. A 16-token prompt may not feed the GPU enough parallel work to stay compute-bound, especially at batch 1. Engines still call it prefill, but the roofline can look more like a small decode. Production traffic with system prompts, chat history and retrieved chunks is usually long enough that the compute-bound story holds.
Walking through the time
t_prefill ~ (2 × N_params × n_input) / achieved_FLOPsQueueing, tokenization and network add more, but if TTFT is 2 seconds on an idle GPU, you are almost always looking at a long prompt or an undersized replica.
Example
Same 70B FP8 replica, using the floors from inference arithmetic:
Prefill 4,000 tokens at about 1.2 PFLOPS effective is on the order of 0.5 s of pure compute.
What prefill writes and reads:
| Writes | Reads heavily | Downstream effect |
|---|---|---|
| Full prompt KV; first token | Weights (amortized), activations | Unlocks streaming; fills memory |
After prefill, the request is resident: its KV must stay addressable until the request finishes or is preempted. Admitting a flood of long-prompt jobs can exhaust HBM even if FLOPs look fine. You ran out of places to put caches, not of math.
Common mistakes
- Treating input tokens like output tokens. Prefill amortizes one weight read across the whole prompt. Decode cannot.
- Ignoring residency. A finished prefill still occupies KV until the request ends.
- Reading a 2 s TTFT as a decode problem. On an idle GPU, that is almost always prompt length or replica size.
Check yourself
- Why is prefill usually compute-bound when decode is not? Answer: the whole prompt is known, so each weight load is reused across many tokens. Math per byte is high.
- A request prefills 4,000 tokens in 0.5 s, then sits in the batch. Where did the memory go? Answer: into the prompt KV cache, which stays resident until the request finishes or is preempted.
Key takeaways
- Prefill processes the prompt in parallel, fills the KV cache, and emits the first token.
- It is usually compute-bound and is the hardware core of TTFT.
- Time scales with input length and parameter count, divided by achieved FLOPs.
- After prefill, KV residency can exhaust memory even when the math units are free.
Further learning
Next, the opposite phase: decode. For the napkin formula, see prefill time.
Related
Prefill vs decode
The two phases of LLM inference. Prefill is compute-bound and sets TTFT. Decode is memory-bound and sets TPOT. They interfere when they share a GPU.
The decode phase
Decode emits one token per step, re-reads the weights and the growing KV cache, and is usually memory-bandwidth-bound. It sets time per output token.
The KV cache
What keys and values the cache stores, how big it gets per token, and how GQA, MLA, quantization and offloading shrink it.