Momentum and Adam

Plain gradient descent zigzags in narrow valleys. The steep direction keeps flipping sign, while the useful direction creeps along. Momentum and adaptive methods fix this by remembering past gradients. Adam combines both ideas, and it is the usual default for training neural networks.

Momentum

Momentum keeps a running velocity, which is a decaying sum of past gradients:

v ← β · v + g       with β ≈ 0.9
θ ← θ − η · v

Picture a heavy ball rolling down a bowl. It keeps moving in a consistent direction and loses speed when the direction flips. Here is a constant gradient of 1 for three steps:

step 1: v = 0.9 · 0 + 1 = 1
step 2: v = 0.9 · 1 + 1 = 1.9
step 3: v = 0.9 · 1.9 + 1 = 2.71

The velocity builds toward 1 / (1 − 0.9) = 10. If the gradient alternates between +1 and −1, the velocity stays between −1 and 1, so the zigzag mostly cancels.

Adam

Adam keeps two running averages: the gradient m, and the squared gradient v. Each parameter's step is scaled by its own history:

m ← β₁ · m + (1 − β₁) · g
v ← β₂ · v + (1 − β₂) · g²
m̂ = m / (1 − β₁ᵗ),   v̂ = v / (1 − β₂ᵗ)     bias correction
θ ← θ − η · m̂ / (√v̂ + ε)

The defaults are β₁ = 0.9, β₂ = 0.999 and ε = 10⁻⁸. They work often enough to be the starting point.

Take the first step with gradient g = 2, so t = 1:

m = 0.1 · 2 = 0.2,     m̂ = 0.2 / 0.1  = 2
v = 0.001 · 4 = 0.004, v̂ = 0.004 / 0.001 = 4
step = η · 2 / √4 = η

The first step is about η in size, whatever the gradient's magnitude. Adam measures the typical gradient size and scales the step by it. That is why Adam's learning rate has a different meaning than SGD's.

Trade-offs

• SGD with momentum often generalizes slightly better on vision tasks when it is tuned well. It needs more learning-rate tuning.
• Adam and AdamW converge quickly with defaults, which makes them the usual choice for language models and quick experiments.
• AdamW decouples weight decay from the gradient update. Prefer it for transformers.

Common mistakes

• Using Adam defaults without a learning-rate check. The defaults do not fit every problem.
• Confusing Adam with AdamW. In Adam, L2 regularization mixes with the adaptive scaling. AdamW applies weight decay directly.
• Expecting momentum to fix a rate that is too high. Momentum amplifies large steps.

Check yourself

1. With β = 0.9 and a constant gradient of 1, what is the velocity after 3 steps? Answer: 2.71.
2. Why is Adam's first step about the learning rate in size? Answer: The bias-corrected average gradient and the root of the average squared gradient are both about the gradient's size, so their ratio is about 1.

Key takeaways

• Momentum averages recent gradients, which speeds up consistent directions and damps zigzags.
• Adam scales each parameter's step by its own gradient history.
• Adam's first step is roughly the learning rate in size.
• Prefer AdamW for transformers, because it applies weight decay directly.

Further learning

Next, reshape the loss surface and schedule the rate over time: conditioning, warmup and learning-rate schedules.