Guide

GPU fundamentals

Launch overhead

Decode runs many short kernels. Python dispatch and kernel launches can cost a millisecond per token before any matmul starts.

19 of 138 · 2 min read

Training steps run large batches for long kernels. The GPU stays busy and PyTorch's asynchronous launches stay ahead of the device. Decode steps are short, repeat millions of times per day, and are paced by user-visible latency. The same model code that felt fine in a training loop can be death by a thousand launches in a token loop.

Concept

Horace He's Making Deep Learning Go Brrrr splits poor performance into three buckets: not enough compute, not enough memory bandwidth, and overhead. Overhead is Python dispatch, kernel launch latency, synchronization, allocator thrash and graph breaks. The GPU is fine. You are not feeding it.

The big idea

A restaurant that plates each pea as its own order will never be limited by the stove. Decode is that restaurant when every add and mul is its own kernel.

The analogy breaks for huge prefills. Those kernels are long enough that launch tax disappears into the math. The tax comes back the moment you generate tokens one by one.

Walking through it

Suppose each kernel launch costs about 8 µs and a decode step runs 120 unfused pointwise kernels that each do almost no math:

Overhead alone ≈ 120 × 8 µs = 960 µs ≈ 1 ms/token

That is already 1,000 tokens/s as a ceiling from launches alone, before matmuls, attention or memory traffic. Fusion and CUDA graphs exist to collapse that tax.

If Nsight Systems shows 0.4 ms of real GEMM and attention and 0.6 ms of gaps per token at batch 4:

Useful fraction ≈ 0.4 / 1.0 = 40%

Closing the gaps can approach a ~2.5× TPOT win without changing the model. That is often cheaper than next-gen hardware.

A barcode of tiny kernels with white gaps means reach for fusion, graphs or compile. One long attention or GEMM with low achieved bandwidth means reach for algorithms or batching, not more compile hope.

Common mistakes

  • Buying a faster GPU to fix a CPU-bound launch loop. The old GPU was waiting on the host.
  • Reading high GPU-Util as proof the math is the limit. Launch-heavy decode can still look busy.
  • Optimizing a training-sized benchmark and shipping it as decode. The overhead bucket was hidden.

Check yourself

  1. 120 launches at 8 µs each. What is the overhead floor? Answer: 960 µs, about 1 ms/token, or at most about 1,000 tokens/s from launches alone.
  2. Nsight shows 0.4 ms of kernels and 0.6 ms of gaps. Which bucket are you in? Answer: Overhead. Fusion and graphs, not more FLOPs.

Key takeaways

  • Slowdowns are compute, bandwidth or overhead. Decode is often the last two.
  • Launch tax is tiny per kernel and huge across a token loop.
  • Gaps in a timeline are a fixable millisecond, not a hardware problem.
  • Measure which bucket you are in before you change code.

Further learning

Making Deep Learning Go Brrrr From First Principles (Horace He)Making Deep Learning go Brrrr From First Principleshorace.io

Next, kernel fusion is the first way to cut both launches and HBM trips.

Kernels, fusion and compilation

Kernel fusion, torch.compile, CUDA graphs and the libraries that make models fast.

Kernel fusion

Combine several ops into one GPU kernel so intermediates stay in registers or shared memory instead of round-tripping through HBM.

CUDA graphs

Record a fixed sequence of GPU launches once and replay it every token, so the CPU is no longer the metronome for decode.