Regularization for neural nets

Once a net can overfit one batch, the next job is to generalize. The regularizers that actually get used are boring: weight decay, a little dropout, data augmentation and early stopping. Exotic penalties are rarely the lever.

The big idea

• Weight decay: L2 penalty on weights, via AdamW. Primary regularizer for large models.
• Dropout: zero random activations at train time, scale at eval. Less fashionable in modern LLMs (weight decay plus data mix suffice) but still used in many smaller nets.
• Data augmentation: the best regularizer when you can define label-preserving transforms (vision crops/flips, SpecAugment, NLP token noise carefully).
• Early stopping: watch validation loss. Stop when it worsens patiently, not on one noisy step.
• Label smoothing: soft targets instead of one-hot. A mild regularizer for classification.

For linear models the same L1/L2 story is in regularization and penalties. Neural nets mostly use decoupled weight decay plus data.

Walking through why AdamW exists

In vanilla SGD, adding λ/2 ‖θ‖² to the loss is the same as decaying weights each step. In Adam, the adaptive scaling is applied to the whole gradient including the L2 term, so decay is warped per-parameter. AdamW applies decay directly to θ outside the adaptive path:

θ ← θ − η ( m̂ / (√v̂ + ε) + λ θ )

When a paper says "AdamW, weight decay 0.1", they mean the decoupled form. Do not substitute Adam + L2 and expect the same result. Use AdamW for almost all modern LLM and transformer training. Typical defaults: β1=0.9, β2=0.95 or 0.999, ε=1e-8, weight decay 0.01–0.1.

Optimizer state memory:

A 7B model in bf16 weights (~14 GB) with AdamW in fp32 moments needs tens of GB more for optimizer state alone, before activations. That is a primary reason for ZeRO/FSDP in distributed training. Details of Adam itself live in momentum and Adam.

Reasonable starting points, not laws:

SGD plus momentum plus a long cosine schedule can match or beat Adam on large vision training at the cost of tuning. If unsure, AdamW.

Add regularization gradually. Do not jump from nothing to heavy dropout plus decay plus aug. See debugging loss curves.

Common mistakes

• Using Adam plus L2 and calling it AdamW. The decay is not the same.
• Heavy dropout on a large LM. Weight decay and data mix usually suffice.
• Early stopping on one noisy val step. Wait for a patient plateau.

Check yourself

1. Why does Adam plus an L2 term in the loss not match AdamW? Answer: Adam's adaptive scaling warps the L2 gradient. AdamW decays θ outside that path.
2. A 7B model in bf16 is ~14 GB of weights. What extra memory do AdamW fp32 moments need, as a multiple of the weights? Answer: About 2× in fp32, tens of GB, before activations.

Key takeaways

• Weight decay, data and early stopping beat clever tricks.
• AdamW is the default modern optimizer. Remember 2× moment memory.
• Dropout is optional in large LMs. It still helps smaller nets.
• Add regularizers one at a time after you can overfit a batch.

Further learning

Decoupled Weight Decay Regularization (AdamW)L$_2$ regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is \emph{not} the case for adaptive gradien…arXiv.org