Mixed precision

Mixed precision runs most of the math in 16-bit floating point, which takes half the memory and runs faster on modern GPUs. It keeps a 32-bit copy of the weights for the optimizer update, so small changes are not lost. The two common 16-bit formats, fp16 and bf16, make different trade-offs.

fp16: small range, needs loss scaling

fp16 has a narrow exponent range. Very large values overflow to infinity, and very small ones underflow to zero. A gradient of 1e-8 rounds to zero in fp16, because it is smaller than the smallest positive value fp16 can represent, about 6 × 10⁻⁸.

The fix is loss scaling. Multiply the loss by a large factor, such as 1024, before the backward pass. Every gradient is scaled up by the same factor, so small ones stay representable. Then divide the gradients by the factor before the optimizer step.

gradient 1e-8       → rounds to 0 in fp16       (lost)
gradient 1e-8 × 1024 = 1.024e-5                  (representable)
unscale: 1.024e-5 / 1024 = 1e-8                  (back to the true value)

Dynamic loss scaling adjusts the factor automatically. It lowers the factor when it sees infinity or NaN, and raises it again after a stable run.

bf16: the fp32 range, fewer digits

bf16 keeps the same exponent range as fp32, so tiny gradients rarely underflow. It has fewer mantissa bits, so each number is less precise, but the optimizer's fp32 master copy absorbs that. Most LLM training on current NVIDIA and Google hardware uses bf16, and it rarely needs loss scaling.

fp8: more gain, more work

As of 2025 and 2026, fp8 training runs on Hopper and Blackwell-class GPUs. It needs careful scaling of each tensor, and often delayed or hybrid recipes. Expect more engineering effort than with bf16.

Why keep the fp32 master copy

A weight stored in 16 bits has few significant digits. Adding a tiny update to it can round the update away completely. The fp32 master copy keeps every small update, and the 16-bit copy is refreshed from it after each step.

Common mistakes

• Using fp16 without loss scaling. Gradients underflow, and the model stops learning in some layers without an error.
• Dropping the master copy to save memory. Small updates then disappear. See the memory math for what that copy costs.
• Adopting fp8 for speed alone. Without a measured gain and a stable recipe, it adds risk and little benefit.

Check yourself

1. Why does a 16-bit weight lose a tiny update? It has few significant digits, so adding a very small number can round back to the original value.
2. Why does bf16 usually skip loss scaling? Its exponent range matches fp32, so small gradients rarely underflow to zero.

Key takeaways

• Mixed precision computes in 16 bits and keeps fp32 master weights.
• fp16 has a narrow range and needs loss scaling. bf16 has the fp32 range and usually does not.
• bf16 is the usual default for LLM training on current hardware.
• fp8 can help, but only once you have measured the gain and a stable recipe.

Further learning

Mixed precision cuts the weight and gradient bytes in the memory budget. Parallelism is the next step: data parallelism and all-reduce.