Conditioning, warmup and learning-rate schedules

The same learning rate can work for one parameter and fail for another. When parameters or features live on very different scales, the loss surface becomes a narrow valley, and a single step size cannot suit both directions. Scaling reshapes the valley. Schedules then change the step size as the run goes on.

Walking through a narrow valley

Take the loss L(w₁, w₂) = (w₁ − 1)² + 100 · (w₂ − 1)², with its minimum at (1, 1). The gradient is [2(w₁ − 1), 200(w₂ − 1)]. At the start point (0, 0), the gradient is [−2, −200].

Two learning rates show the problem:

η = 0.01:   step to [0.02, 2.0]
            w₂ overshoots the target 1 by the same amount. At this rate, it never settles.
η = 0.001:  step to [0.002, 0.2]
            w₂ moves safely, but w₁ barely moves. The steep direction sets the safe rate.

The steep direction needs a small rate, and the flat direction needs a large one. One global rate cannot satisfy both.

The fix: scale the features

Standardize the features so that each one has a similar spread. The valley becomes round, and one rate works for all directions. Adam helps too, because it scales each parameter separately. Scaling is still cheap, and it helps regardless of the optimizer.

Warmup, decay and early stopping

Real training rarely uses a constant learning rate.

• Warmup. Start near zero, and ramp up over the first few hundred or thousand steps. Early updates use Adam's averages before they have settled, and deep residual networks can spike without warmup.
• Decay. Lower the rate over time with a step schedule, a cosine curve, or a linear decline to a floor. Late in training, a small rate settles the parameters into a good region.
• Early stopping. Watch the validation loss, and stop when it stops improving.

A common diagnostic: if the training loss falls while the validation loss rises, you are overfitting. If both are poor, check the learning rate, the features and the pipeline.

Common mistakes

• Assuming Adam makes scaling irrelevant. Scaling still helps, and it costs almost nothing.
• Keeping one rate for the whole run without a reason. A schedule usually helps.
• Stopping on the training loss. Early stopping uses the validation loss.

Check yourself

1. With L = (w₁ − 1)² + 100(w₂ − 1)² at (0, 0) and η = 0.001, where does w₂ go in one step? Answer: To 0.2, since the step is η · 200 · 1 = 0.2.
2. Why use warmup? Answer: Early updates are the least reliable, and a small starting rate prevents spikes while the statistics settle.

Key takeaways

• Features on different scales create narrow valleys, and one rate fails for some directions.
• Standardizing features reshapes the valley and helps any optimizer.
• Warmup protects early training, and decay settles it late.
• Stop on validation loss, not training loss.

Further learning

Learning-rate schedules and the full training recipe are covered in training neural networks well.

Back to the overview: optimization and gradient descent.