Guide

Anatomy of LLM inference

Prefill time

How long the prompt takes to process before the first token, estimated from parameter count, prompt length and achieved FLOPs.

27 of 138 · 2 min read

Prefill reads the whole prompt at once and builds the KV cache for it. All prompt tokens pass through the model together. That makes large matrix multiplies, which keep the math units busy. Prefill is therefore mostly limited by compute, not memory. You can estimate its time from the prompt length.

Concept

A FLOP is one floating-point operation. FLOP/s is how many a chip does per second. Achieved FLOPs is what your kernels really reach, which is usually well below the datasheet peak.

Walking through it

A forward pass costs about two FLOPs per parameter per token. Each weight does one multiply and one add for each token.

FLOPs_per_token ≈ 2 × N_params
t_prefill ≈ (2 × N_params × prompt_tokens) / achieved_FLOPs

Example: 70B model in FP8, a 2,048-token prompt, and a pair of H100s that sustain about 1,200 TFLOPS.

  1. FLOPs: 2 × 70e9 × 2,048 ≈ 2.9e14.
  2. Time: 2.9e14 / 1.2e15 ≈ 0.24 s.

So the first token takes about 240 ms before any queueing. For a tuned stack on H100, plan for 40 to 70% of the dense peak. Naive PyTorch can be far worse.

A 1,000-token prompt on the same setup takes about 0.12 s. The estimate grows roughly with prompt length. The attention term, which grows faster, is left out here.

Prefill also spreads one weight read over many tokens. That is one reason input tokens are often cheaper than output tokens on hosted APIs.

Common mistakes

  • Using the datasheet peak. Achieved FLOPs are much lower, so the estimate comes out too optimistic.
  • Forgetting queue time. Time to first token is queue time plus prefill. Under load the queue often dominates.
  • Assuming linear growth forever. Attention cost grows faster with length. See long-context serving.

Check yourself

  1. Use the same setup with a 1,000-token prompt. How long is prefill? Answer: 2 × 70e9 × 1,000 / 1.2e15 ≈ 0.12 s.
  2. Why is prefill compute-bound while decode is memory-bound? Answer: Prefill pushes many tokens through the same weights, so each byte read does a lot of math. Decode at small batch does one token's math per weight read.

Key takeaways

  • Prefill time ≈ 2 × parameters × prompt tokens / achieved FLOPs.
  • Use achieved FLOPs, not peak.
  • Prefill is compute-bound because it reuses each weight read across many tokens.
  • Long prompts add attention cost on top of this estimate.

Further learning

Next, the batch ceiling shows how decode gains from batching until it hits a limit. For the two phases side by side, see prefill vs decode.

Inference arithmetic

Napkin math for any model on any GPU: weight memory, decode and prefill time, batch limits and KV budget.

Prefill vs decode

The two phases of LLM inference. Prefill is compute-bound and sets TTFT. Decode is memory-bound and sets TPOT. They interfere when they share a GPU.

GPU architecture

SMs, warps, tensor cores and the memory hierarchy from registers to HBM, with H100/B200 numbers.