Anatomy of LLM inference
Inference workload shapes
Chat, RAG, reasoning and agents sit in different corners of the input-versus-output plane. Each corner wants a different capacity plan.
Different products push different corners of the prefill/decode plane. A chat app is balanced. A RAG search is prefill-heavy. A reasoning model is decode-heavy. An agent grows context and repeats prefixes. One concurrency knob cannot fix all four.
A practical habit: draw traffic as a scatter of (input tokens, output tokens) from production logs, then mark which quadrant dominates.
Concept
A workload shape is the typical pair of input length and output length, plus how those lengths change over a session. Shape, not QPS alone, tells you which phase to buy capacity for.
The big idea
Think of a restaurant that serves espresso and banquets from one kitchen. The espresso line (short in, short out) and the banquet line (long in or long out) need different stations. Mixing them in one queue makes both worse.
The picture breaks when a product is mixed on purpose. Then you split pools, or at least priority queues, instead of picking one "average" shape.
Walking through the plane
| Workload | Input | Output | Pain point |
|---|---|---|---|
| Chat | Short–medium | Medium | Balanced; TTFT and TPOT both visible |
| RAG / doc QA | Long, retrieved context | Short–medium | Prefill, TTFT, KV size |
| Reasoning / "think" models | Short | Very long | Decode tokens/s, cost per answer |
| Agents / tool loops | Growing context | Bursts of medium | KV growth, repeated prefills of prefixes |
| Code autocomplete | Medium, file prefix | Short | Ultra-low TTFT |
Agents are especially nasty: each tool round can append observations, so the KV cache grows and later steps re-read more memory. Shared system prompts and tool schemas are why prefix caching pays off.
Tip
When someone says "we need more GPUs," ask whether users are waiting on first token or tokens after that. Prefill-heavy and decode-heavy problems want different hardware and different scheduler settings.
Example
Two products share one 70B replica. Product A is RAG support search: p50 input 6k tokens, p50 output 150 tokens. Product B is an internal coding agent: p50 input 800 tokens, p50 output 2.5k tokens. A's incidents are "spinner for two seconds." B's are "stream hitching then slow finish." Split pools or at least priority queues: A prefers prefill-friendly scheduling and prefix cache; B prefers a stable decode batch and higher TPOT headroom.
If most mass sits in "long in, short out," buy prefill capacity, aggressive caching and TTFT SLOs. If most mass sits in "short in, long out," buy decode bandwidth (quantization, speculative decoding, replicas tuned for batch) and TPOT SLOs. Mixed mass is when colocated scheduling gets ugly and disaggregation starts to pencil out.
As a builder: shrink output with tighter prompts and stopping criteria when cost matters. Shrink input with better retrieval, summarization and prompt caching when TTFT and prefill cost matter. For reasoning models, budget output tokens as the dominant line item.
Common mistakes
- Planning from QPS alone. Ten RAG queries and ten reasoning traces are not the same load.
- One pool for two shapes. A's spinner and B's hitch will fight over the same batch.
- Forgetting agents grow KV. Later tool rounds are slower and more expensive than the first.
Check yourself
- p50 is 6k input and 150 output. Which metric and which lever come first? Answer: TTFT and prefill capacity. Prefix cache and retrieval trimming beat another decode kernel.
- p50 is 800 input and 2.5k output. What do you buy? Answer: decode bandwidth and a stable decode batch. Watch TPOT, not only first token.
Key takeaways
- Shape is the (input, output) pair, plus how it grows over a session.
- RAG is prefill-heavy. Reasoning is decode-heavy. Agents do both and grow KV.
- Split pools or queues when two products sit in different corners.
- Shrink the side of the plane that dominates your bill: input for TTFT, output for long answers.
Further learning
Back to prefill vs decode. For the agent and RAG prefix problem, see prefix caching.
Related
Prefill vs decode
The two phases of LLM inference. Prefill is compute-bound and sets TTFT. Decode is memory-bound and sets TPOT. They interfere when they share a GPU.
Prefix caching
Reusing KV across requests that share prompts. Block hashing, RadixAttention, cache-aware routing, and why cached input tokens are billed cheaply.
Disaggregated prefill and decode
Running prefill and decode on separate GPU pools so each phase gets its own parallelism, batching, and hardware — and when colocated chunked prefill is enough.