GPU architecture

A CPU is built to finish one tricky instruction stream quickly. A GPU is built to finish millions of nearly identical arithmetic ops. Inference sits on that throughput machine. If you treat a GPU like a fast CPU, you will write kernels that look parallel and still crawl.

A modern server CPU might have tens of cores and hundreds of GB/s of DRAM bandwidth. A datacenter GPU has on the order of 100+ streaming multiprocessors, multiple TB/s of HBM, and tensor cores for matrix math. Clock rates are similar. The shape of parallelism is not.

Decode often looks small: one token, a batch of a few dozen sequences. The art is packing enough parallel work that the GPU's width is used.

Concepts in this section

Read them in order. Each one builds on the one before.

Step	Concept	What it answers
1	Warps and SIMT	How does the chip actually run thousands of threads?
2	GPU memory hierarchy	Where do weights and the KV cache live, and how fast is that path?
3	Tensor cores and peak FLOPs	Which number on the datasheet should you believe?
4	GPU interconnects	When is tensor parallel viable, and when does the link become the model?

Latency hiding in one picture

HBM latency is large compared to one arithmetic instruction. The SM keeps many warps resident and switches among them when one stalls. That is why occupancy and memory-level parallelism matter for bandwidth-bound kernels: without them the math units sit idle even though the chip is "waiting on memory."

Time to stream 80 GB on H100 ≈ 80e9 / 3.35e12 ≈ 24 ms
H100 BF16 ridge ≈ 989e12 / 3.35e12 ≈ 295 FLOP/byte

Those two lines are the hardware map for later pages. Single-stream decode lives near the 24 ms-per-weight-read floor. Prefill and large-batch work climb toward the ridge.

nvidia-smi reports HBM used vs free, GPU utilization, and sometimes power. Utilization near 100% does not mean you are compute-bound. A memory-bound decode loop can keep the chip busy copying. Power draw and tensor-core activity from Nsight are better hints.

HBM is the DRAM attached to the GPU. "GPU-Util" in many tools is an activity sampler, not FLOPs. A process can show high util while achieving a few percent of peak tensor TFLOPS.

Multi-Instance GPU (MIG) partitions one physical GPU into isolated slices with dedicated memory. Useful for many small models. Painful for a 70B-class model that needs the full HBM pool. See GPU sharing.

Not every fleet is NVIDIA. The concepts transfer: compute vs bandwidth, on-chip vs off-chip memory, collectives. The APIs and peak tables do not. When you evaluate other accelerators, ask whether the model runs, what tokens/s you get at your SLO, and what the software tax is.

Common mistakes

Explaining an OOM with leftover FLOPs. Memory is the scarce resource after weights load.
Reading utilization as proof you are compute-bound. Copy-heavy decode looks busy too.
Picking tensor-parallel size without reading NVLink topology.
Using sparse datasheet peaks for a dense model.

Key takeaways

GPUs buy throughput with massive multithreading and hide HBM latency by switching warps.
Work is grids of blocks on SMs. Warps of 32 threads run in SIMT lockstep.
HBM capacity and bandwidth bound model size, batch and decode speed. FLOPs bound prefill.
Tensor cores dominate useful LLM math. Believe dense peaks.
NVLink makes in-node tensor parallel viable. PCIe and multi-node links are much thinner.