Scale and post-training
Memory math for training
Training memory is much more than the weights. Mixed-precision Adam needs about 16 bytes per parameter before any activations, which is why a 7B model needs over 100 GB.
Most people estimate memory as "parameters times two bytes". That is only the weights. During training each parameter also needs a gradient, a full-precision master copy and two optimizer moments. Counting all of them gives the real budget, and it is what forces every parallelism strategy that follows.
Concept
Before you start: Adam keeps two running averages per parameter. The first moment is the mean of recent gradients. The second moment tracks their mean square, which is a measure of their spread. Both are usually kept in fp32 for numerical stability.
The bytes per parameter
With mixed-precision Adam, each parameter typically stores:
| Tensor | Precision | Bytes per parameter |
|---|---|---|
| Weights used in compute | bf16 or fp16 | 2 |
| Gradients | bf16 or fp16 | 2 |
| Master weights | fp32 | 4 |
Adam first moment m | fp32 | 4 |
Adam second moment v | fp32 | 4 |
| Total | 16 |
The optimizer state, the master weights and the moments make up 12 of the 16 bytes. That is why optimizer state dominates training memory.
Walking through a 7B model
Multiply each row by 7 billion parameters:
weights (bf16) 7e9 × 2 = 14 GB
gradients (bf16) 7e9 × 2 = 14 GB
master + m + v (fp32) 7e9 × 12 = 84 GB
-------
total before activations 112 GBAn 80 GB GPU cannot hold a single replica of this state. Activations for a long context and a large batch then add tens of gigabytes more. The naive floor is well above one GPU.
Example
Activations scale with batch size × sequence length × hidden size × number of layers. Attention can add more. Long-context training therefore needs memory tricks even when the weights are small.
What the number tells you
- Sharding the 12 bytes of optimizer state is the largest saving available, which is why ZeRO and FSDP exist.
- Lower-precision compute saves memory on the weights and gradients, but the fp32 master copy stays. See mixed precision.
- Activations are a separate budget. Checkpointing and gradient accumulation trade compute for them.
Common mistakes
- Counting only the weights. A 7B model at 2 bytes is 14 GB, but the training state is 112 GB.
- Forgetting the fp32 master copy. Dropping it to save memory changes the optimizer's numerical behavior.
- Ignoring activations. They can outgrow the weights for long sequences and large batches.
Check yourself
- How much memory do the two Adam moments take for a 7B model? Two fp32 moments take
2 × 4 × 7e9 = 56 GB. - Why is 112 GB only a floor? Activations come on top of the 16 bytes per parameter, and they grow with batch size and sequence length.
Key takeaways
- Mixed-precision Adam needs about 16 bytes per parameter, before activations.
- For a 7B model, that is 112 GB of training state.
- Most of the cost is optimizer state and master weights, not the weights you run the model with.
- Every parallelism choice on the next pages is an answer to this arithmetic.
Further learning
The precision choices behind these bytes are on the mixed precision page.
Related
Distributed training
How to train models too big for one GPU, using data, sharding, tensor, pipeline and context parallelism. Covers the memory math that forces the choice and how to judge the result.
Mixed precision
Mixed precision runs most matrix multiplies in 16-bit floats and keeps a 32-bit master copy of the weights. fp16 needs loss scaling to avoid underflow. bf16 usually does not.
ZeRO and FSDP
ZeRO and FSDP remove the copies that data parallelism makes of optimizer state, gradients and weights. Each GPU keeps only a shard and gathers the rest when a layer needs it.