GPU fundamentals
The CUDA programming model
Kernels, grids, blocks and threads, memory coalescing and occupancy, plus Triton as the higher-level path.
CUDA is how you turn the GPU's SMs and warps into a program. You will not rewrite vLLM in raw CUDA tomorrow, but every serving mystery — why a tiny elementwise kernel kills decode, why a matmul needs tiling, why Triton exists — is a programming-model story. This section gives you enough to read a kernel, sketch one, and know when to stop and call a library.
The host is the CPU. The device is the GPU. They have separate memories, and a kernel is a device function the host launches as a grid of blocks of threads. Most slow kernels are too many trips to HBM, or too many launches per token.
You rarely need a CUDA interview to ship inference. You do need to read a profile and say "we're launch-bound" versus "we're HBM-bound," know why block size 256 versus 128 might matter, decide Triton versus waiting for a library, and avoid inventing a matmul when cuBLAS exists.
Concepts in this section
Read them in order. Each one builds on the one before.
| Step | Concept | What it answers |
|---|---|---|
| 1 | Kernels, grids and threads | How does a launch map work onto SMs? |
| 2 | Coalescing and tiling | Why is naive matmul slow, and how do you reuse bytes? |
| 3 | Occupancy and launch overhead | How do you hide HBM latency, and why do tiny kernels kill decode? |
| 4 | Triton and when to write a kernel | Library, Triton, or raw CUDA? |
Fusion, torch.compile and CUDA graphs — the serving-engine levers that sit on top of this model — live in kernels and compilation.
At work
When a custom op in a serving engine is slow, confirm the grid covers the problem, loads coalesce, and you are not synchronizing every token. Only then dive into bank conflicts and instruction-level tricks. Print blockDim, gridDim and the problem size. Off-by-one grids that leave most of the GPU idle are common when porting a tutorial sized for n=1024.
Global memory is the scarce bandwidth resource. Registers and shared memory are how you stop visiting it. Launch overhead of a few microseconds is how decode dies when ops stay tiny.
Common mistakes
- Writing a GEMM. Call cuBLAS, CUTLASS or a serving-engine kernel.
- Chasing occupancy or bank conflicts before a profile. Most decode pain is launch tax or uncoalesced HBM.
- Host-device sync per token. It destroys the async pipeline that was hiding Python.
Key takeaways
- Kernels run on device. Launches specify a grid of blocks of threads mapped onto SMs.
- Coalesced HBM access and shared-memory tiling are the first two performance levers after correctness.
- Occupancy hides latency. Launch overhead wrecks decode. Fuse, compile and graph.
- Prefer libraries. Use Triton for fusions. Raw CUDA is last-mile control.
Further learning
CUDA C++ Programming GuideThe programming guide to the CUDA model and interface.docs.nvidia.com How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance (Simon Boehm)In this post, I’ll iteratively optimize an implementation of matrix multiplication written in CUDA.My goal is not to build a cuBLAS replacement, but to deepl...siboehm.com Triton tutorialsTutorials — Triton documentationtriton-lang.orgThis ML systems lecture builds the programming model with a live vector-add walkthrough:
NVIDIA's modern CUDA C++ class starts from execution spaces and memory spaces:
Related
Kernels, grids and threads
A kernel is a device function launched as a grid of blocks of threads, each computing a different index.
GPU architecture
SMs, warps, tensor cores and the memory hierarchy from registers to HBM, with H100/B200 numbers.
Kernels, fusion and compilation
Kernel fusion, torch.compile, CUDA graphs and the libraries that make models fast.