GPU fundamentals
Roofline examples
Elementwise add sits far left of the ridge. A large square matmul sits to the right. Tiny GEMMs fall back onto the bandwidth slope.
The ridge is only useful if you can place a kernel on it. Two kernels that look like "just math" sit at opposite ends of the plot: an elementwise add and a large square matmul.
Concept
Memory-bound means you empty the bandwidth budget first. Compute-bound means you empty the FLOP budget first. Sitting under the roof means the implementation is leaving performance on the table (occupancy, coalescing, communication, launch tax).
The big idea
Adding two grocery lists is a lot of walking and almost no arithmetic. Multiplying two big spreadsheets is a lot of arithmetic on data you can keep nearby. The picture breaks for a tiny spreadsheet: there is not enough reuse, so even "matmul" walks the memory slope.
Walking through elementwise add
c[i] = a[i] + b[i] for n floats (4 bytes each). Per element: 1 add ≈ 1 FLOP. Bytes: read a, read b, write c → 12 bytes (write-allocate can make this worse; 12 is a lower bound).
I_add ≈ 1 FLOP / 12 bytes ≈ 0.083 FLOP/byteFar below the H100 BF16 ridge of ~295. Bandwidth ceiling ≈ 0.083 × 3.35 TB/s ≈ 0.28 TFLOPS — nowhere near 989. Elementwise is memory-bound. Fusing five elementwise ops into one kernel cuts trips to HBM and multiplies effective intensity. That is why fusion matters.
Walking through a large square matmul
For C = A @ B with A and B sized N×N in BF16 (2 bytes/elem), FLOPs are 2 N^3 (mul+add). If you stream each matrix once: 2 N^2 inputs + N^2 output → 3 N^2 elements × 2 bytes = 6 N^2 bytes.
I_matmul ≈ (2 N^3) / (6 N^2) = N/3 FLOP/byteFor N = 4096: I ≈ 1365 FLOP/byte — above the H100 BF16 ridge. A well-written GEMM can be compute-bound. For tiny N, intensity drops and matmul becomes bandwidth-bound too. Microbenchmarks of small GEMMs mislead.
| Region | Bound | What helps |
|---|---|---|
| Left of ridge | Memory-bound | Fewer bytes (fusion, quantization, layouts), more reuse (tiling, batching), higher bandwidth |
| Right of ridge | Compute-bound | Better algorithms, tensor cores, more SMs, fewer algorithmic FLOPs |
| Under the roof | Implementation gap | Occupancy, coalescing, instruction mix, communication, launch overhead |
Common mistakes
- Fusing a GEMM that already hits tensor cores and expecting a big win. Fusion is a bandwidth-bound tool.
- Trusting a small-N matmul bench as "the GPU's GEMM speed." Intensity was too low.
- Counting only reads. The write of
Cis part of the byte model above.
Check yourself
- Why is
c = a + bstuck near 0.28 TFLOPS on this H100 sketch? Answer: intensity is about 0.083 FLOP/byte, so the bandwidth slope caps you. - At what N is a square BF16 matmul at the 295 ridge, using I ≈ N/3? Answer: N ≈ 885; N = 4096 is well above it.
Key takeaways
- Elementwise ops sit far left. Large matmuls sit right.
- Fusion raises intensity by cutting HBM trips.
- Tiny GEMMs are memory-bound even though they are "matmul."
- Match the lever to the region: bytes and reuse on the left, FLOPs and tensor cores on the right.
Further learning
Next, see how decode batch size walks an LLM along the same plot.
Related
Arithmetic intensity and the ridge point
Intensity is FLOPs per byte moved from HBM. The ridge is where the bandwidth slope meets the compute roof.
The roofline model and arithmetic intensity
Predicting performance from FLOPs and bytes: compute-bound vs memory-bound. Start here for the map, then work the examples.
Kernels, fusion and compilation
Kernel fusion, torch.compile, CUDA graphs and the libraries that make models fast.