GPU hardware and interconnects

A GPU's memory, the link to other GPUs, and the network between nodes decide what a job can do. Peak FLOPS on a slide matters much less. This page covers those three checks, and a quick memory calculation you can do before you order any hardware.

The big idea

There are two networks. Inside a node, GPUs share a fast link, such as NVLink through NVSwitch. Between nodes, they talk over the cluster network, InfiniBand or RoCE.

Think of a kitchen with one shared counter, and a hallway to other kitchens. Work on the counter is fast. Work that goes down the hallway is slower and needs a bigger door. The analogy hides how much slower the hallway can be.

Choosing a GPU

SXM modules sit on a baseboard with NVLink and NVSwitch. PCIe cards plug into a standard server and get less GPU-to-GPU bandwidth. For multi-GPU training on one node, pick SXM. For single-GPU inference, a PCIe L4 or L40S is often enough.

Walking through a memory check

A 70-billion-parameter model in FP16 stores each parameter in 2 bytes.

70 billion parameters x 2 bytes = 140 GB for weights alone

That is before the KV cache and activations. One H100 with 80 GB cannot hold it in FP16 without splitting it across GPUs. An H200 with about 141 GB might hold the weights, but leaves little room for long contexts. Production serving usually splits the model across 2 to 8 GPUs with tensor parallelism, or uses lower-precision weights.

The infrastructure job is to refuse a "one GPU, please" request that breaks this arithmetic, and to propose a parallel layout the fabric can support.

Cross-node bandwidth

Across nodes, training uses NCCL collectives such as all-reduce. Large training designs target roughly 400 to 800 Gb/s of network bandwidth per GPU, so collectives are not starved while the GPUs wait. GPUDirect RDMA lets the network card move data straight to GPU memory, skipping the CPU and host memory.

Common mistakes

• Choosing by FLOPS. Memory and interconnect usually decide the job.
• Buying PCIe GPUs for multi-GPU training. Without NVSwitch, tensor parallelism runs slower.
• Forgetting the KV cache. It takes memory on top of the weights, and it grows with context length.
• Designing your own fabric. Start from a reference design.

Check yourself

1. Roughly how much memory do a 70B model's FP16 weights need? About 140 GB, from 70 billion parameters times 2 bytes, before cache and activations.
2. Why does NVSwitch matter for tensor parallelism? It lets every GPU in the node talk at NVLink speed. Without it, traffic falls back toward PCIe speeds.

Key takeaways

• Check memory size, the intra-node link and the inter-node network before you pick hardware.
• Multi-node training needs RDMA-capable fabric and enough bandwidth per GPU.
• Do the memory arithmetic before ordering. It rules out many shapes early.

Further learning

Next, see how Kubernetes hands these GPUs to pods in GPUs on Kubernetes.