Kernels, fusion and compilation

Knowing GPU architecture and the roofline is not enough if the runtime launches 200 tiny kernels per token and spends half its life in Python. Modern inference performance is as much compilation and fusion as it is raw CUDA skill.

You will meet these ideas in every serving engine README ("CUDA graphs enabled", "torch.compile", "fused MoE kernel"). Those phrases are levers: overhead, bytes and libraries.

Concepts in this section

What a healthy decode step looks like

1. The scheduler picks a batch of sequences that fit in KV pages.
2. One or a few attention kernels — FlashAttention or FlashInfer style, not a naive matmul plus softmax chain.
3. GEMM library calls for QKV projections and the MLP, ideally with fused epilogues.
4. Light sampling kernels.
5. Many of those launches replayed from a CUDA graph, so the CPU is not the metronome.

An unhealthy timeline is hundreds of eager ops, aten::add / aten::mul barcodes, device-host sync for token checks, and allocator spikes when KV grows.

Measure top-down:

1. Token-level metrics — TTFT, TPOT, tokens/s at a fixed concurrency.
2. PyTorch profiler — unexpected copies and syncs.
3. Nsight Systems — gaps between kernels vs long saturated kernels.
4. Nsight Compute — when one kernel is hot, is it HBM-bound, compute-bound, or stalled?

If Nsight Systems shows a barcode of tiny kernels, reach for fusion, graphs or compile. If it shows one long GEMM with low achieved bandwidth, reach for algorithms or batching.

Engines such as vLLM and SGLang already package these techniques. Prefer their defaults. Use torch.compile for custom PyTorch paths. Treat new dtypes as a library problem: no kernel, no speedup.

Fusion does not fix a communication-bound tensor-parallel setup. Graphs do not fix an OOM. A new GEMM library does not fix Python .item() syncs in the token loop. Measure which of the three buckets you are in before changing code.

Common mistakes

• Flipping flags without a profile, then blaming the model.
• Shipping compile caches as a surprise after autoscaling.
• Padding never, then wondering why graphs recapture under traffic.

Key takeaways

• Decode is often overhead plus bandwidth, not missing FLOPs.
• Fusion raises intensity and cuts launches.
• CUDA graphs slash per-token launch tax when the step is static enough.
• torch.compile automates much fusion. Watch shapes and compile time.
• Production speed comes from libraries plus engine-level scheduling.

Further learning

Making Deep Learning Go Brrrr From First Principles (Horace He)Making Deep Learning go Brrrr From First Principleshorace.io torch.compile documentationRedirecting…pytorch.org Getting Started with CUDA Graphs (NVIDIA)The performance of GPU architectures continue to increase with every new generation. Modern GPUs are so fast that, in many cases of interest, the time taken by each GPU operation (e.g.NVIDIA Technical Blog

The Jane Street talk expands fusion, graphs and why eager mode hits a wall.