GPU memory hierarchy

Every GPU calculation sits on a ladder of memories. The top rungs are tiny and fast. The bottom rung is huge and slow. Inference lives or dies on that bottom rung: HBM, the main memory on the card, is where weights, activations and the KV cache all sit.

Concept

HBM (high-bandwidth memory) is the DRAM stacked next to the GPU die. It is what nvidia-smi reports as used vs free. Shared memory is a programmable scratchpad on each SM. Registers are the fastest storage, private to one thread.

The big idea

Think of a kitchen. Registers are the cutting board. Shared memory is the counter. HBM is the walk-in fridge. You can store a whole model in the fridge. You cannot cook from it one ingredient at a time without wasting the stove.

The analogy breaks on bandwidth. A fridge is slow to walk to. HBM is slow and has a published peak rate. An H100 SXM moves about 3.35 TB/s. That number, not the FLOPs on the box, often sets decode speed.

Walking through it

Same idea as the CPU memory hierarchy, different sizes.

Approximate H100 SXM numbers:

Level	Ballpark
Registers	Huge in total; the per-thread cap limits occupancy
Shared memory / L1	About 228 KB configurable per SM
L2	About 50 MB
HBM3	80 GB at ~3.35 TB/s

Related cards you will see in production, as of when this was written:

GPU	HBM capacity	Peak HBM bandwidth
H100 SXM	80 GB	~3.35 TB/s
H200 SXM	141 GB	~4.8 TB/s
B200 SXM	~180 GB	up to ~8 TB/s

Reading 80 GB of weights once at 3.35 TB/s takes:

80e9 bytes / 3.35e12 bytes/s ≈ 0.024 s ≈ 24 ms

A decode step that must touch most weights for a single token cannot beat a floor near "weight bytes / bandwidth." That is why batch-1 decode feels bandwidth-bound. The roofline page turns this into a plot.

Inference runs out of memory more often than it runs out of FLOPs. A "fast" GPU with too little HBM forces a smaller batch, more GPUs, or compression.

Common mistakes

Reading GPU utilization as FLOPs. A memory-bound decode loop can show high util while it mostly copies.
Stopping at "the model fits." Weights plus the KV cache must fit, with slack.
Treating HBM bandwidth as infinite. Streaming the weights once already costs tens of milliseconds on an H100.

Check yourself

How long does it take to stream 80 GB once at 3.35 TB/s? Answer: 80e9 / 3.35e12 ≈ 24 ms.
You have spare tensor-core FLOPs but only 10 GB of free HBM. What fails first? Answer: Capacity. You cannot raise batch or context if the cache has nowhere to live.

Key takeaways

Registers and shared memory are fast and small. HBM is large and is the usual limit.
H100-class HBM is about 80 GB at about 3.35 TB/s. Later cards add capacity and bandwidth, not a new hierarchy.
Time to stream the weights once is a real floor for single-stream decode.
Pick GPUs for memory as much as for FLOPs.

Further learning

Next, the GPU architecture hub puts this ladder next to warps and interconnects.