GPU fundamentals
GPU interconnects
NVLink, PCIe and InfiniBand set how fast GPUs can share work. Keep latency-sensitive collectives inside an NVLink domain.
One GPU is often not enough. How the cards talk decides whether tensor parallelism is viable, or whether every layer waits on a thin pipe.
Concept
A collective is a group operation such as all-reduce: every GPU contributes a shard and everyone gets the sum. Tensor-parallel attention and MLP layers do this on every layer. The link that carries it is as much part of the model as the math.
The big idea
Think of an office. NVLink is the hallway inside one floor. PCIe is the elevator to the host. InfiniBand is the street between buildings. You can move a filing cabinet down the hallway all day. You cannot do that through the elevator on every sentence.
The analogy breaks for coarse work. Pipeline parallel stages and some MoE expert routing can survive a thinner link, because they move less often.
Walking through it
| Link | Ballpark useful rate | Role |
|---|---|---|
| NVLink / NVSwitch (H100 node) | ~900 GB/s GPU-to-GPU class fabric inside an 8-GPU HGX domain | Tensor and expert parallel collectives inside the node |
| PCIe Gen5 x16 | ~64 GB/s per direction (~128 GB/s duplex in marketing) | Host↔GPU, or GPU↔GPU without NVLink |
| InfiniBand / RoCE | 200–800 Gb/s NICs common | Pipeline, data or expert parallel across servers |
Keep latency-sensitive, high-bandwidth collectives inside an NVLink domain when you can. Crossing nodes for those collectives is often a large regression unless the shards are coarse.
PCIe also caps how fast you can stage weights from CPU or NVMe. That path is useful for offload and painful as the primary path for hot weights.
On a multi-GPU box, read the topology before you pick a tensor-parallel size. The size should usually match a fully connected NVLink island, commonly 2, 4 or 8 on HGX-class machines. Crossing PCIe for all-reduces on every layer is a classic self-inflicted wound.
Not every fleet is NVIDIA. AMD MI300X-class cards, Google TPUs, AWS Trainium and Inferentia, and others share the same questions: compute vs bandwidth, on-chip vs off-chip memory, and collectives. The APIs and peak tables do not transfer. Ask whether your model and precision run, what tokens/s you get at your SLO, and what the software tax is.
Common mistakes
- Picking tensor-parallel size from model size alone. Topology comes first. A TP=8 split across PCIe-only peers will stall.
- Using peak duplex PCIe as if it were NVLink. It is an order of magnitude thinner.
- Buying FLOPs and ignoring the fabric. A fast GPU behind a slow link is a prefill chip that cannot decode as a group.
Check yourself
- Why keep tensor-parallel all-reduces inside NVLink when you can? Answer: They run on every layer and need hundreds of GB/s. PCIe and multi-node links are much thinner.
- A box has eight GPUs but only pairs are NVLink-connected. What TP sizes are safe? Answer: 2, matching each pair. TP=8 would cross the slow links.
Key takeaways
- NVLink-class fabrics make in-node tensor parallel viable.
- PCIe is the host path and a poor GPU-to-GPU path for hot collectives.
- Multi-node links fit coarser parallelism, not per-layer all-reduces.
- Match parallelism to topology, not the other way around.
Further learning
The parallelism page turns these links into TP, PP and EP choices. NVIDIA's Hopper architecture writeup has the official node diagrams.
Related
GPU architecture
SMs, warps, tensor cores and the memory hierarchy from registers to HBM, with H100/B200 numbers.
Parallelism for inference
Tensor, pipeline, data and context parallelism: splitting a model across GPUs so weights and KV fit, and single-request latency stays under SLO.
GPU infrastructure
Provisioning and scheduling GPUs for training and inference, covering hardware and fabrics, Kubernetes device plugins, gang scheduling, failures and checkpoints. Start here for the map of GPU pages.