Training neural networks well

A correct model definition is maybe 20% of getting a neural net to work. The rest is optimization: learning rates, initialization, normalization, regularization and a disciplined debugging loop. This page is the craft layer on top of gradient descent and the PyTorch training loop.

Ship a boring stack: AdamW plus warmup and cosine, LayerNorm or RMSNorm residuals, and weight decay. Spend budget on data quality and eval, not exotic optimizers. When pretraining at scale, the same knobs appear with bigger numbers. See pretraining and scaling laws.

Concepts in this section

Read them in order. Each one builds on the one before.

Related pages already cover pieces of this craft: momentum and Adam, warmup and schedules, initialization and residual connections.

The learning rate is usually the single most important hyperparameter. Warmup stops early updates from destroying randomly initialized features. Cosine decay is standard for long runs. If you invent a deep architecture, default to residual plus norm.

PyTorch nn.Linear already uses reasonable Kaiming-style defaults. Do not randomly re-init unless you know why. For a balanced K-way classifier, loss at step 0 should be about ln K (2.3 for 10 classes, about 10.8 for a 50k vocab). If you see 15 or 0.01, something is wrong before you blame Adam.

Keep a training diary: git commit hash, config YAML, dataset revision, best checkpoint metric. Reproducibility is a process, not a seed.

Interview angle: explain AdamW versus SGD, why warmup exists, what ln K checks, diagnose a curve that diverges versus overfits, and how grad accumulation interacts with BatchNorm.

Common mistakes

• Blaming the optimizer before you overfit one batch. If one batch will not go to ~0, the bug is in code, loss or shapes.
• Skipping model.train() / eval(). BatchNorm and dropout then silently misbehave.
• One learning rate on a frozen backbone plus a new head. The head underfits or the backbone wrecks.

Key takeaways

• AdamW is the default modern optimizer. Remember 2× moment memory.
• Warmup plus cosine beats a constant LR for long runs.
• Init so forward variance is stable. Verify loss ≈ ln(num_classes) at step 0.
• LayerNorm or RMSNorm for transformers. BatchNorm still common in CNNs.
• Residuals make depth trainable. Weight decay and data beat clever tricks.
• Overfit one batch first, then read train/val curves systematically.

Further learning

Karpathy's recipe is short and worth re-reading before every new training run.

A Recipe for Training Neural NetworksMusings of a Computer Scientist.karpathy.github.io Decoupled Weight Decay Regularization (AdamW)L$_2$ regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is \emph{not} the case for adaptive gradien…arXiv.org

Stanford lecture on training nets — solid on init, BatchNorm and schedules:

Neural Networks: Zero to HeroNeural Networks: Zero To Herokarpathy.ai