GPU fundamentals
Kernel fusion
Combine several ops into one GPU kernel so intermediates stay in registers or shared memory instead of round-tripping through HBM.
A naive PyTorch step writes every intermediate to HBM and launches a kernel for each op. Fusion combines several logical ops into one kernel so those intermediates stay in registers or shared memory. You save bytes, which raises intensity, and you save launches, which cuts overhead.
Concept
An epilogue is the pointwise work that follows a matmul while the tile is still in registers: bias, activation, scaling, a residual add. Epilogue fusion attaches that work to the GEMM so you do not start a second memory round trip.
The big idea
Cooking a sauce in one pan is fusion. Pouring it into a new pot after every pinch of salt is eager mode. The stove (tensor cores) is fine either way. The walking is the tax.
The picture breaks at library boundaries and at data-dependent control flow. Those edges are hard to fuse. Serving engines and torch.compile spend a lot of effort widening the fusible set.
Walking through it
What commonly gets fused in LLM stacks:
- Pointwise chains (add, mul, activation)
- LayerNorm / RMSNorm with adjacent ops when the compiler allows
- Attention kernels that fuse softmax scaling and masking (FlashAttention)
- Quantized GEMM with dequant on the fly
When a profiler shows a huge gemm followed by a tiny relu, ask whether an epilogue API (cuBLASLt, CUTLASS, Inductor) could absorb it.
Almost nobody hand-rolls the hottest GEMMs. The compiler's job is often: fuse the glue, call the library for the meat.
| Library | Role |
|---|---|
| cuBLAS / cuBLASLt | Vendor GEMM; workhorse for dense matmul |
| CUTLASS | Template GEMM toolkit for custom shapes and precisions |
| cuDNN | Convolutions, some norm and attention paths historically |
| FlashAttention / FlashInfer | IO-aware attention and decode primitives |
| Custom quantized GEMMs | INT4 / FP8 / FP4 paths in engines |
When a precision is new (FP4 on Blackwell), the availability of a good GEMM decides whether the datasheet peak is usable. A new dtype flag without a kernel is not a speedup.
Eager PyTorch is debuggable. Fully fused graphs are faster and more brittle. For research, prefer eager. For production decode, prefer compiled and fused paths, then measure TTFT and TPOT.
Common mistakes
- Fusing for its own sake on a communication-bound tensor-parallel setup. Fusion does not fix the link.
- Leaving epilogues as separate kernels after a fat GEMM. The bytes and the launches are both free wins.
- Writing a custom kernel that a library already has. Maintainership cost is real.
Check yourself
- A profile shows
gemmthenreluthenadd. What should you look for? Answer: An epilogue fusion that keeps the tile in registers through bias, activation and residual add. - Does fusion change the FLOP count of the matmul? Answer: No. It cuts HBM trips and launches around the matmul.
Key takeaways
- Fusion keeps intermediates on-chip and cuts launches.
- Epilogue fusion attaches pointwise work to GEMMs.
- Libraries do the meat. Compilers fuse the glue.
- New precisions only help when a fused GEMM path exists.
Further learning
Next, CUDA graphs record a fused decode step and replay it every token.
Related
Kernels, fusion and compilation
Kernel fusion, torch.compile, CUDA graphs and the libraries that make models fast.
Launch overhead
Decode runs many short kernels. Python dispatch and kernel launches can cost a millisecond per token before any matmul starts.
FlashAttention
Why standard attention is HBM-bound, how tiling plus online softmax computes exact attention in on-chip SRAM, and what FlashAttention-2/3 and decode variants change.