Partial derivatives and gradients

A loss depends on many weights at once. A partial derivative measures how the output changes when one input moves and the others are held fixed. Stack all the partial derivatives into one vector and you get the gradient. The gradient points in the direction in which the function rises fastest.

∇f = [∂f/∂x₁, ∂f/∂x₂, …, ∂f/∂xₙ]

Working one example

Take f(x, y) = x² + 3xy + y². To get ∂f/∂x, treat y as a constant. To get ∂f/∂y, treat x as a constant:

∂f/∂x = 2x + 3y
∂f/∂y = 3x + 2y

At the point (x, y) = (1, 2):

∇f = [2·1 + 3·2, 3·1 + 2·2] = [8, 7]

The function rises fastest in the direction [8, 7]. Check the first entry by nudging x from 1 to 1.001 with y = 2. f goes from 11 to 11.008001, a change of 0.008001 over a step of 0.001. That is a slope of about 8.

Taking a descent step

Gradient descent moves against the gradient. With learning rate η = 0.1:

[1, 2] − 0.1 · [8, 7] = [0.2, 1.3]

The point moves down the slope, and f drops. The learning rate sets how far one step goes.

In machine learning

In training, the function is the loss L(θ), and the variables are the parameters θ. Computing ∇θ L is the goal of backpropagation. Each entry of the gradient says how the loss would change if that one parameter moved.

Common mistakes

• Mixing up partial and total derivatives. A partial derivative holds the other inputs fixed. If y also depends on x, the total derivative includes that path too.
• Thinking the gradient is the loss. The gradient is a vector of slopes, one per parameter, not a number.
• Forgetting the minus sign. Gradient descent subtracts the gradient, because the gradient points uphill.

Check yourself

1. At (1, 2), which direction does f increase fastest? Along [8, 7].
2. Why does gradient descent subtract the gradient? The gradient points uphill, and we want the loss to go down.

Key takeaways

• A partial derivative is the slope along one input, holding the others fixed.
• The gradient is the vector of partial derivatives, and it points uphill.
• Gradient descent takes a step of size η in the opposite direction.
• In training, each entry of the gradient is one parameter's share of the loss slope.

Further learning

Next, how slopes combine through a chain of operations: the chain rule.