Warps and SIMT

A GPU is not a fast CPU. It is a machine for running the same math on many elements at once. The hardware does not pick threads one by one. It picks a warp: 32 threads that share one instruction and apply it to different data.

The big idea

Picture 32 clerks who must turn the same page at the same time. When they all add numbers, the room is busy. When half of them take an if and half take an else, the room runs both paths and masks the idle clerks. That is divergence.

The picture breaks for real programs: different warps on the same SM are independent. One warp can take the if while another takes the else with no penalty.

Walking through it

You launch a grid of blocks. Each block is scheduled onto one SM. Several blocks can sit on one SM if registers and shared memory allow. That packing is occupancy.

HBM reads take on the order of hundreds of cycles. The SM hides that wait by switching to another ready warp. You need enough warps in flight, or the math units sit idle even though "the GPU is waiting on memory."

A data-dependent branch in a hot loop can cut a warp's useful work roughly in half. A grid that only fills a few SMs, or a kernel so register-heavy that only one warp fits, shows up as low bandwidth and low FLOPs.

H100 SXM: 132 SMs
Warp size: 32 threads
Shared memory / L1: up to about 228 KB per SM

You almost never size work as "one thread per matrix." You size the grid so thousands of warps are resident, then let the hardware cover stalls.

Common mistakes

• Writing a kernel that looks parallel but only fills a few SMs. The chip is wide. A tiny grid leaves most of it idle.
• Ignoring divergence in a hot loop. One if on the data can serialize the warp.
• Chasing occupancy as a score. Enough warps to hide latency is the goal. Past that, extra occupancy can even hurt if it starves registers.

Check yourself

1. Why does an if/else inside a warp cost more than the same branch across two different warps? Answer: Lanes in one warp share an instruction, so both paths run with some lanes masked. Separate warps do not.
2. A kernel uses so many registers that only one warp fits on each SM. What breaks? Answer: Latency hiding. When that warp waits on HBM, the SM has nobody else to run.

Key takeaways

• The GPU schedules warps of 32 threads that execute in lockstep.
• Divergence happens inside a warp, not between warps.
• Occupancy is how many warps fit. You need enough of them to hide HBM latency.
• Size grids for the whole chip, not for one matrix.

Further learning

Next, see where those threads keep their data in GPU memory hierarchy.