Parallelism for inference

One GPU is the unit of deployment until it is not. A 70B model in FP8 is roughly 70 GB of weights. Add KV cache for a busy batch and you blow past an 80 GB H100. A 405B dense model does not fit on one node at all. Even when weights do fit, a latency target may need more memory bandwidth than one GPU can deliver.

Parallelism for inference is how you cut the model, or the request stream, across devices. Training uses the same names (distributed training). Inference cares about different costs: there is no backward pass, the critical path is often one token, and KV residency dominates memory.

Concept

Model parallelism shards one copy so serving one request uses several GPUs. Data parallelism keeps replicas and partitions requests. You almost always need both: tensor or pipeline parallelism to fit and speed up one replica, then replicas to scale throughput.

When one GPU is not enough

Leave a single-GPU layout for two independent reasons.

Capacity. Weights plus activations plus KV must fit in HBM. Weight footprint is params × bytes_per_param. KV grows with batch, sequence, layers, heads and dtype. See the KV cache.
Latency. Sharding weights with tensor parallelism multiplies effective HBM bandwidth for each matmul, at the price of collectives. That can cut time per output token when you are memory-bound.

If the model fits and your time-to-first-token and time-per-output-token targets are already met, do not add tensor parallelism. Extra ranks buy latency only when communication is cheaper than the work you remove.

Concepts in this section

Step	Concept	What it answers
1	Tensor parallelism	How do you split each layer across GPUs?
2	Pipeline parallelism	How do you split layers across nodes?
3	Data parallelism	How do replicas raise throughput?
4	Context and sequence parallelism	How do you shard a huge prompt?

Choosing a layout

Assume H100 80 GB, NVLink within a node, and FP8 weights. Numbers are napkin math for planning.

70B dense, FP8. Weights ≈ 70 GB. One H100 holds weights but leaves little room for KV. Common choice: TP=2 (or TP=4 for lower latency or more KV headroom). Scale QPS with replicas. Pipeline parallelism is usually unnecessary on one node.

405B dense, FP8. Weights ≈ 405 GB. One 8×H100 node has 640 GB, so TP=8 (~50 GB/GPU) fits with some KV left. For larger KV or BF16, use TP=8 × PP=2 across two nodes.

For MoE models, expert parallelism often replaces or shrinks TP on expert MLPs. See serving mixture-of-experts.

Common mistakes

Adding TP by default. If one GPU meets the SLO, stop.
Stretching TP across InfiniBand. Collectives then dominate. Add pipeline stages across nodes instead.
Using more TP to raise QPS. Replicas raise QPS. TP is for fit and single-request latency.
Ignoring topology. NCCL timeouts after raising TP are usually NVLink or NUMA, not the model.

Key takeaways

Leave one GPU when weights plus KV do not fit, or when one GPU misses the latency target.
Tensor parallelism shards every layer. Keep it inside the NVLink domain.
Pipeline parallelism shards layers across devices. Use it across nodes, and watch bubbles on decode.
Replicas scale throughput, not per-request latency.
Context parallelism shards the sequence for long prefill.