Guide

Scale and post-training

Data parallelism and all-reduce

Data parallelism gives each GPU a full copy of the model and a different slice of the batch. The GPUs then sum their gradients with an all-reduce, so every copy applies the same update.

144 of 187 · 2 min read

Data parallelism is the simplest way to use many GPUs. Each GPU holds the whole model and processes its own microbatch. After the backward pass, the GPUs combine their gradients so that every copy of the model applies the same update. The model copies stay identical, and the throughput grows with the number of GPUs.

Concept

Before you start: an all-reduce is a collective operation. Every GPU contributes a tensor, and every GPU ends with the sum (or average) of all of them. It is the communication step that keeps the replicas in sync.

The flow

Each GPU does the same four things:

  1. Run the forward pass on its own microbatch.
  2. Run the backward pass to get local gradients.
  3. Join an all-reduce with the other GPUs, so every GPU holds the summed gradient.
  4. Apply the same optimizer step, so the weights stay identical.

PyTorch's DistributedDataParallel overlaps the all-reduce with the backward pass when it can, so communication runs while the GPUs are still computing.

How much data moves

The efficient method is a ring all-reduce. With N GPUs and M bytes of gradients, each GPU sends and receives about 2 × (N − 1) / N × M bytes. As N grows, that approaches 2M.

Example

A 7B model in bf16 has 7e9 × 2 = 14 GB of gradients. Across 8 GPUs, each GPU moves:

2 × (8 − 1) / 8 × 14 GB = 24.5 GB per step

On a fast 400 GB/s link, that is about 24.5 / 400 ≈ 0.06 seconds, or roughly 60 milliseconds, before overlap. On a slow network, the same traffic can dominate the step. That is why the topology of the cluster matters as much as the GPU count.

What data parallelism does not do

Data parallelism does not reduce the memory each GPU needs. Every GPU holds the full weights, gradients and optimizer state. It scales throughput and the effective batch size. When the replica does not fit, you need sharding, which is the next page's topic: ZeRO and FSDP.

Common mistakes

  • Expecting a bigger model to fit. Adding data-parallel GPUs does not shrink what one GPU must hold.
  • Ignoring communication when adding nodes. Throughput flattens when the all-reduce takes as long as the compute.
  • Forgetting the effective batch. The global batch is microbatch × DP, plus any gradient accumulation. Learning-rate choices depend on it.

Check yourself

  1. With 8 GPUs and 14 GB of gradients, roughly how much does each GPU move per step in a ring all-reduce? About 2 × 7/8 × 14 = 24.5 GB.
  2. Does adding data-parallel GPUs reduce memory per GPU? No. Each GPU still holds the full model and optimizer state. It adds throughput.

Key takeaways

  • Data parallelism keeps identical model copies and averages gradients with an all-reduce.
  • A ring all-reduce moves about 2M bytes per GPU as the GPU count grows.
  • Overlapping communication with the backward pass hides some of that cost.
  • Data parallelism adds throughput, not memory headroom.

Further learning

When the replica does not fit, sharding splits the state: ZeRO and FSDP.

Distributed training

How to train models too big for one GPU, using data, sharding, tensor, pipeline and context parallelism. Covers the memory math that forces the choice and how to judge the result.

ZeRO and FSDP

ZeRO and FSDP remove the copies that data parallelism makes of optimizer state, gradients and weights. Each GPU keeps only a shard and gathers the rest when a layer needs it.

Parallelism for inference

Tensor, pipeline, data and context parallelism: splitting a model across GPUs so weights and KV fit, and single-request latency stays under SLO.