Gradient descent and the learning rate

Gradient descent is the basic move behind nearly all model training. Compute how the loss changes with each parameter, then step the other way. One number, the learning rate, controls how far each step goes. It is the most important hyperparameter you will tune.

The big idea

The update rule is one line:

θ ← θ − η · g

Here θ is the parameters, g is the gradient, and η (eta) is the learning rate. A bigger η means a bigger step.

Walking through one parameter

Minimize L(w) = (w − 3)². The gradient is 2(w − 3). Start at w = 0 with η = 0.1.

step 1: g = 2(0 − 3)   = −6,    w = 0 − 0.1·(−6)   = 0.6
step 2: g = 2(0.6 − 3) = −4.8,  w = 0.6 + 0.48     = 1.08
step 3: g = 2(1.08 − 3) = −3.84, w = 1.08 + 0.384  = 1.464

The distance to the answer, 3, shrinks from 3 to 2.4 to 1.92 to 1.536. Each step multiplies the distance by 0.8 = 1 − 2η. For this function, the rule distance × (1 − 2η) holds for any learning rate.

That multiplier explains the whole behavior:

A tiny linear regression

This script fits y ≈ w·x + b on five points with gradient descent. It runs as written.

import numpy as np

x = np.array([0.0, 1.0, 2.0, 3.0, 4.0])
y = np.array([1.0, 2.9, 5.1, 7.0, 9.1])   # roughly y = 2x + 1

w, b, eta = 0.0, 0.0, 0.05
for step in range(40):
    err = (w * x + b) - y
    w -= eta * np.mean(2 * err * x)       # dL/dw
    b -= eta * np.mean(2 * err)           # dL/db

final_err = (w * x + b) - y
print(f"w={w:.3f} b={b:.3f} loss={np.mean(final_err ** 2):.4f}")

Output:

w=2.053 b=0.894 loss=0.0052

The first step shows the mechanics. At w = 0 and b = 0, the loss is 33.446. The gradients are −28.2 for w and −10.04 for b. One step with η = 0.05 moves w to 1.41 and b to 0.502. After 40 steps, the fit is close to the true line of slope 2 and intercept 1.

Choosing the learning rate

Try a log grid, such as 1e-3, 1e-2, 5e-2, 0.2, for a few hundred steps. Keep the largest value that does not diverge. On this five-point problem, the results were:

• η = 0.001: loss 14.9 after 30 steps. Too slow.
• η = 0.01: loss 0.021 after 30 steps.
• η = 0.05: loss 0.0065 after 30 steps. The best of the four.
• η = 0.2: the loss explodes to about 10¹⁵. This is past the stability limit.

Common mistakes

• Picking a learning rate once and never checking. A short sweep takes minutes and saves hours.
• Reading a flat loss as a local minimum. Check the learning rate and the code first.
• Raising the learning rate to speed up a diverging run. A larger step makes divergence worse.

Check yourself

1. For L = (w − 3)² and η = 0.25, what is the multiplier on the distance to 3? Answer: 1 − 2 · 0.25 = 0.5, so the distance halves each step.
2. The loss becomes NaN at step 2. Which knob should you change first? Answer: The learning rate. It is too large, so lower it.

Key takeaways

• Gradient descent steps against the gradient, scaled by the learning rate.
• Too small a rate crawls. Too large a rate oscillates or diverges.
• Sweep the learning rate on a log grid before a long run.
• Mini-batches make each step cheaper, and they are the next topic.

Further learning

Next, estimate the gradient from a random subset of the data: mini-batch SGD and batch size.