Distributed training

A 7B-parameter model is modest by frontier standards, yet training it on one GPU is not possible. The weights fit, but the optimizer state, gradients and activations do not. Distributed training turns a cluster of GPUs into one logical accelerator. It shards state, moves only what it must, and keeps the GPUs busy enough to justify the cost.

This topic is the training-side map. Inference parallelism reuses some of the same splits, but training stores Adam's moments and backpropagates through activations, so the memory budget is different.

Concepts in this topic

Read in this order. Each page builds on the one before it.

Common mistakes across the topic

• Using data parallelism to fit a model. Data parallelism adds throughput. Each GPU still holds the full model and optimizer state.
• Judging a layout by whether it runs. A layout can run at 20% efficiency. Measure MFU.
• Changing several axes at once. Change one of tensor, pipeline or data parallelism at a time, so you know which change helped.

Fault tolerance at scale

At thousands of GPUs, hardware failure is routine. Treat the training job like a distributed system.

• Checkpoint on a cadence. Save every N minutes or N steps. Too rare, and a crash wastes days. Too frequent, and I/O dominates. Asynchronous, sharded checkpoints are standard.
• Watch for stragglers. One slow rank stalls every synchronous step. Causes include thermal throttling, network congestion and a bad network card. Track per-rank step time as closely as the loss.
• Rehearse the restart. Restart from a checkpoint before you scale out, and set alerts on collective timeouts.

Key takeaways

• Mixed-precision Adam needs about 16 bytes per parameter before activations.
• Data parallelism scales throughput. Sharding and model parallelism reduce memory per GPU.
• Use bf16 with fp32 master weights by default.
• Match the parallel layout to the cluster's topology: fast links inside a node, slower links between nodes.
• Track MFU, and expect 35 to 55% for well-tuned pretraining. Much lower usually means a layout or data problem.

Further learning

Zachary Mueller's walkthrough covers DDP, ZeRO, tensor and pipeline parallelism with clear visuals:

For FSDP internals from a PyTorch-oriented talk:

The Ultra-Scale Playbook (Hugging Face)The ultimate guide to training LLM on large GPU Clustershuggingface.co ZeRO: Memory Optimizations Toward Training Trillion Parameter ModelsLarge deep learning models offer significant accuracy gains, but training billions to trillions of parameters is challenging. Existing solutions such as data and model parallelisms exhibit fundamental limitations to f…arXiv.org Megatron-LM: Training Multi-Billion Parameter Language Models Using Model ParallelismRecent work in language modeling demonstrates that training large transformer models advances the state of the art in Natural Language Processing applications. However, very large models can be quite difficult to trai…arXiv.org