Deep learning
Debugging loss curves
Check loss at initialization, overfit one batch, then read train versus val systematically. Most "broken training" is a shape, LR or data bug, not a new optimizer.
Karpathy's recipe is a disciplined debugging loop: become one with the data, overfit one batch, set a sensible learning rate, then add regularization.
Concept
For a balanced K-way classifier with a near-zero last layer, softmax is about 1/K, so loss should start near ln K. If it does not, something is wrong before you blame the optimizer.
The big idea
L ≈ −log(1/K) = ln K| K | ln K (nats) |
|---|---|
| 2 | 0.69 |
| 10 | 2.30 |
| 1000 | 6.91 |
| 50,000 (big BPE vocab) | ≈ 10.8 |
If your LM loss starts at 4 with a 50k vocab, the head is already oddly peaked or labels are wrong. If it starts at 25, logits are exploding. Check residual scales and init. Log this number on every new run before celebrating a "fast" drop from a broken start.
Walking through the recipe
- Become one with the data — visualize labels, find duplicates, check imbalance.
- Overfit one batch — train until loss ~0 on 1–2 batches. If you cannot, the bug is in code, loss or shapes, not hyperparameters.
- Set a sensible LR — range test, or start from known defaults (
1e-3AdamW for small nets,3e-4class for many transformers). - Generalize — add regularization gradually. Do not jump from nothing to heavy dropout plus decay plus aug.
- Measure — plot train/val curves. Distrust vibes.
Also: confirm decreasing train loss before trusting validation, and change one thing at a time.
| Pattern | Likely cause | Try |
|---|---|---|
| Loss → NaN / explodes | LR too high, bad init, div by zero | Cut LR 10×, grad clip, check the loss API |
| Flat from step 0 | LR too low, forgot train(), grads blocked | Raise LR, check requires_grad, zero_grad order |
| Train ↓ val ↑ | Overfitting | More data/aug, weight decay, early stop |
| Both flat after progress | Plateau | LR schedule, longer train, data diversity |
| Val << train | Data leak or aug-only train difficulty | Audit splits. Eval without heavy aug |
The table's "try" column is the underfitting and overfitting checklist. See overfitting for the bias–variance version. If both curves look stuck, check shuffle, constant labels, a scheduler that set LR to 0, or eval using train-mode BatchNorm.
Train loss is noisy even with correct code. Smooth with an exponential moving average. Do not change LR because of one spiky step. A 50-example val set will jitter more than your real gains.
Mismatching "scheduler step per batch" versus "per epoch" is a classic way to accidentally use 100× fewer decay steps than you intended. Plot opt.param_groups[0]["lr"] over time on every serious run.
Example
Multiclass net, K=1000, init loss ≈ 6.9. You measure 0.4 at step 0: labels may be wrong or the head is over-confidently init. You measure 40: logits exploding — check weight init scale or input normalization.
Common mistakes
- Tuning the optimizer before you can overfit one batch. If one batch will not go to ~0, the code is wrong.
- Celebrating a drop from loss 25. You started broken.
- Changing three knobs after one spiky step. Smooth the curve first.
Check yourself
- A 10-way classifier starts at loss 0.01. What is wrong? Answer: Init loss should be about
ln 10 ≈ 2.3. 0.01 means the head is already peaked or labels leak. - You cannot drive loss to ~0 on two batches. Is the learning rate the first suspect? Answer: No. Shapes, loss, labels or blocked grads come first.
Key takeaways
- Verify loss ≈
ln Kat step 0. - Overfit one batch before you tune hyperparameters.
- Read train versus val as a table of patterns, not a vibe.
- Change one thing at a time. Plot the learning rate as well as the loss.
Further learning
Karpathy's recipe is short and worth re-reading before every new training run.
A Recipe for Training Neural NetworksMusings of a Computer Scientist.karpathy.github.io Neural Networks: Zero to HeroNeural Networks: Zero To Herokarpathy.aiRelated
Training neural networks well
Optimizers, schedules, initialization, normalization, regularization and a practical recipe for debugging loss curves.
Output heads and loss functions
The last layer and the loss must match the question. Regression uses a linear output with MSE, binary classification uses one logit with binary cross-entropy, and multiclass uses softmax with cross-entropy.
Overfitting, underfitting and bias–variance
A model that memorizes its training data looks brilliant until it sees new data. A model too simple for the pattern fails on both. Two learning curves tell you which problem you have.