The roofline model and arithmetic intensity

Peak TFLOPS on a datasheet are a ceiling, not a promise. The roofline model tells you which ceiling you hit: the compute roof or the memory-bandwidth slope. Once you know which, you stop applying the wrong optimizations. More FLOPS fusion will not fix a kernel that is already waiting on HBM. More cache tiling will not fix a pure compute-bound GEMM that already hits tensor cores.

The model comes from Williams, Waterman and Patterson (2008). Inference engineers use it constantly, often without naming it: "decode is memory-bound," "prefill is compute-bound," "batch harder."

Think of it as a budget identity. Every second the machine can either move a fixed number of bytes or complete a fixed number of FLOPs. Whichever budget you exhaust first is your speed limit. Once that clicks, most inference folklore becomes a special case.

On a log-log plot, X is arithmetic intensity (FLOP/byte), Y is attainable performance (FLOP/s). A diagonal is y = x * bandwidth. A horizontal is y = peak_FLOP/s. The shape looks like a roof — hence the name.

Before changing engine flags, ask: is this request's hot phase compute-, memory-, or overhead-bound? A Nsight timeline that shows HBM bandwidth saturated and tensor cores idle is a memory-bound diagnosis. Gaps between tiny kernels are overhead. Tensor cores pegged with spare bandwidth is compute-bound.

Carry a one-line checklist into design reviews: (1) estimate intensity, (2) compare to ridge, (3) name the bound, (4) pick the matching lever. If you cannot do step 1, you are not ready to argue about flags. Inference math turns this checklist into token-level floors and ceilings for a chosen model and GPU.

Concepts in this section

Step	Concept	What it answers
1	Arithmetic intensity and the ridge	What is FLOPs per byte, and where do the roofs meet?
2	Roofline examples	Where do elementwise add and a big GEMM sit?
3	Decode batch size on the roofline	Why does batching move decode toward the compute roof?
4	Communication rooflines	When do collectives beat local compute?

A single user request occupies two different points over its lifetime. Prefill processes many prompt tokens in parallel and often leaves the memory slope. Decode at low concurrency sits on the slope. The engine's job is to keep enough sequences in flight to climb toward the ridge without blowing TTFT for newcomers. See prefill vs decode.

Launch overhead is the third roof: the kernel never runs long enough to hit FLOPs or bytes. Then fuse, compile or graph before buying silicon.

Common mistakes

Treating 30% of peak FLOPS as shame when intensity said 3% was the roof.
Applying compute-bound tools (more SMs, fancier GEMM) to a bandwidth-bound decode job.
Trusting a blog that claims batch-1 decode near peak TFLOPS on a 70B. Intensity math says no.
Ignoring collectives. Extra TP ranks can slow tokens/s when the link is the roof.

Key takeaways

Intensity is FLOPs / bytes. The ridge is peak FLOP/s ÷ peak bandwidth.
Below the ridge you are memory-bound. Above it, compute-bound.
Elementwise sits far left. Large matmuls sit right. Decode intensity scales with batch.
Quantization and fusion move intensity right by cutting bytes.
Collectives and launch overhead are extra roofs. Bound the claim before you trust the slide.