Deep learning
BatchNorm, LayerNorm and RMSNorm
Normalization rescales activations so each layer sees well-scaled inputs. BatchNorm uses the batch, LayerNorm uses one example's features, and RMSNorm drops the mean.
A correct model definition is not enough. If activations explode or die, gradients follow. Normalization rescales those activations so layers see well-scaled inputs. The three names you will meet are BatchNorm, LayerNorm and RMSNorm. They differ in what they average over.
Concept
Internal covariate shift was the original BatchNorm story. The practical win is a smoother loss landscape and more forgiving initialization and learning rate.
The big idea
| Method | Normalizes over | Typical use |
|---|---|---|
| BatchNorm | batch × spatial, per channel | CNNs (train carefully with small batches) |
| LayerNorm | features per example | Transformers, RNNs |
| RMSNorm | like LayerNorm without mean centering | Many modern LLMs |
BatchNorm uses batch statistics at train time and running averages at eval. That is one reason model.train() versus eval() matters. Small batches make BN noisy. LayerNorm does not depend on batch size.
RMSNorm scales by the root-mean-square of the vector, with no mean centering:
RMSNorm(x) = x / RMS(x) * γ
RMS(x) = sqrt(mean(x²) + ε)Fewer ops than LayerNorm, and slightly better empirically in many LLM recipes. γ is a learned per-channel scale (d parameters per norm).
Walking through residual plus norm order
Two common patterns: post-norm and pre-norm. Modern transformers lean pre-norm (h = h + F(Norm(h))) because training is stabler at depth. Exact placement interacts with init and warmup. When copying an architecture, copy the norm order too. "ResNet + LayerNorm" is underspecified.
If your deep net diverges only past layer 20, suspect norm order or missing residuals before you invent a new optimizer. Depth without residual plus norm is usually untrainable noise. See residual connections.
Prefer LayerNorm or RMSNorm inside transformers. BatchNorm remains common in classic ResNets.
Dropout placement similarly matters: drop on the residual branch output is common. Dropping inside attention scores needs care. When a paper reports a recipe, the diagram's arrows are as important as the optimizer table.
A picture: BatchNorm is a class curve, graded on the whole exam group. LayerNorm is grading each student against their own answers. The picture breaks for BatchNorm at batch size 2, because the "class" is then noise.
Gradient accumulation simulates a larger batch when memory is tight, but BatchNorm statistics still see only the micro-batch. That is another reason transformers prefer LayerNorm.
Common mistakes
- Leaving BatchNorm in train mode at eval. Running averages are ignored, and metrics jitter.
- Using BatchNorm with batch size 2 and calling the model unstable. The noise is the batch statistic.
- Copying a transformer block but flipping pre-norm to post-norm. Depth then diverges.
Check yourself
- Why does LayerNorm suit transformers better than BatchNorm? Answer: It normalizes per example, so it does not depend on batch size, and LLM batches are often small per device.
- What does RMSNorm drop compared with LayerNorm? Answer: Mean centering. It only scales by RMS and a learned
γ.
Key takeaways
- Normalization keeps activations on a scale that gradients can travel through.
- BatchNorm for classic CNNs. LayerNorm or RMSNorm for transformers.
- Pre-norm plus residuals is why deep stacks train.
- BatchNorm stats see the micro-batch, even if you accumulate gradients.
Further learning
Next: debugging loss curves.
Related
Training neural networks well
Optimizers, schedules, initialization, normalization, regularization and a practical recipe for debugging loss curves.
The transformer architecture
Every component of a modern decoder-only LLM, from embeddings to the LM head, with parameter and FLOP counts.
Convolutional networks
How CNNs use small shared filters, stacked layers and residual connections to learn images, and why transfer learning from pretrained models is the default.