Scaling out
Pipeline parallelism
Assign blocks of layers to different GPUs or nodes. Inter-stage traffic is small, but decode leaves pipeline bubbles that waste GPU time.
Pipeline parallelism (PP) assigns contiguous blocks of layers to different devices. Rank 0 runs the first block, rank 1 the next, and so on. Activations flow forward at each boundary. There is no weight all-reduce on every layer.
You reach for PP when the model cannot fit in one node's memory even with tensor parallelism maxed out. The traffic between stages is one activation tensor per boundary per token, about H × dtype. That is far less chatter than tensor parallelism's per-layer all-reduces, so PP prefers weaker cross-node links.
Concept
A pipeline stage is a contiguous block of layers on one device. A pipeline bubble is idle time while a stage waits for input from the stage before it.
The big idea
Think of an assembly line. Station 1 does layers 0–19, station 2 does 20–39, and the part moves down the line. Training hides idle stations by feeding many microbatches at once.
The picture breaks on decode. Continuous batching changes the batch every step, and you often want low latency more than a full line. Stages then sit idle waiting for the next token.
Walking through when PP wins
A 405B model in FP8 is about 405 GB of weights. One 8×H100 node has 640 GB of HBM, so TP=8 can fit (~50 GB per GPU) with some KV left. For larger KV or BF16 weights, you need TP=8 × PP=2 across two nodes.
| Goal | Prefer | Avoid |
|---|---|---|
| Fit weights on one node | Raise TP, up to 8 | Premature PP |
| Fit beyond one node | TP=8 plus PP=2 or more | TP across InfiniBand |
| Low decode latency | Max NVLink TP first | PP=2 on one 8-GPU node if TP=8 fits |
Engines hide some bubbles with asynchronous point-to-point sends, microbatching during prefill, and overlap. Expect worse GPU utilization than a pure-TP single-node replica unless the stack is tuned for PP.
Trade-off
Use TP within a node and PP across nodes when the model is huge. Avoid PP=2 "just because" on a single 8-GPU node if TP=8 fits. You pay bubbles without leaving the NVLink domain.
Common mistakes
- Using PP on one node to "be safe." If TP fits, PP only adds bubbles.
- Assuming training-style microbatching will hide decode bubbles. Continuous batching changes the batch every step, so the old trick does not apply cleanly.
- Ignoring engine support. Empty stages waiting for microbatches can deadlock. Prefer engines with a tested PP inference path.
Check yourself
- Why is PP a better cross-node tool than wide TP?
Answer: each boundary sends one activation of size about
H × dtype, not an all-reduce on every layer. - A 70B model in FP8 fits on two GPUs with TP=2. Should you add PP? Answer: no. PP is for when the model cannot fit in one NVLink node even with TP.
Key takeaways
- PP shards layers across devices. Activations, not weights, cross each boundary.
- Decode is hostile to pipelines: stages idle between tokens.
- Keep TP inside a node. Add PP only to fit across nodes.
- Prefill can microbatch to hide some bubbles. Decode usually cannot.
Further learning
Next, data parallelism scales throughput with replicas. For the sharding picture, see How to Scale Your Model: Sharded Matrices.
Related
Parallelism for inference
Tensor, pipeline, data and context parallelism: splitting a model across GPUs so weights and KV fit, and single-request latency stays under SLO.
Tensor parallelism
Split each layer across GPUs so every token uses every rank. Column-parallel and row-parallel matmuls plus all-reduces do the work.
Data parallelism for inference
Independent replicas of an already-sharded model. Each replica has its own KV pool. Throughput scales; single-request latency does not.