Second derivatives and curvature

The first derivative tells you the slope. The second derivative tells you how that slope is changing, which is the curvature. A loss surface near a minimum looks like a bowl. Curvature tells you how steep the walls of that bowl are, and that controls how large a learning step can be before it overshoots.

A worked example

Take f(x) = x⁴ − x²:

f'(x)  = 4x³ − 2x = 2x(2x² − 1)
f''(x) = 12x² − 2

The slope is zero where f'(x) = 0, so the critical points are x = 0 and x = ±√(1/2) ≈ ±0.707.

• At x = 0: f''(0) = −2 < 0, so this is a local maximum.
• At x = ±0.707: f'' = 12 · 0.5 − 2 = 4 > 0, so both are local minima.

The curvature at the minima is positive, so they are bowls. The curvature at zero is negative, so it is a hilltop.

Why this matters for learning rates

Imagine a loss that is gently curved in one direction and sharply curved in another. A step size that is safe along the gentle direction is too large along the sharp one, and the update overshoots and bounces back and forth.

In a single variable, a learning rate must be small enough that the step does not jump past the minimum. In many variables, the sharpest direction sets the limit. This is why adaptive optimizers and careful learning-rate schedules exist. See optimization for those methods.

The Hessian

For many parameters, the curvature is a matrix of all second partial derivatives, called the Hessian. For P parameters it has P × P entries. A neural network with millions of parameters cannot store that matrix, so training methods work with cheaper approximations to the curvature.

Common mistakes

• Treating every zero slope as a minimum. A zero slope can be a minimum, a maximum or a flat spot. Check the second derivative.
• Expecting training to stall at zero gradient. In full-batch toy problems this can happen. With noisy mini-batches, it is rare in practice.
• Forgetting the learning rate depends on curvature. A rate that works on one loss can diverge on another with sharper curvature.

Check yourself

1. Which kind of point is x = 0 for f(x) = x⁴ − x²? A local maximum, since f''(0) = −2 < 0.
2. Why can a sharply curved direction force a smaller learning rate? The step overshoots the minimum along that direction, so the update bounces instead of settling.

Key takeaways

• The first derivative is the slope. The second derivative is the curvature.
• Positive curvature means a bowl, negative curvature means a hilltop.
• Sharp curvature limits how large a learning step can be.
• The full Hessian is too large to build for big models, so training uses approximations.

Further learning

The optimization page covers how optimizers use these ideas in practice.