The chain rule on a graph

Neural networks are long chains of simple operations. The chain rule says how slopes combine along such a chain: if y depends on u and u depends on x, then the slope of y with respect to x is the product of the two slopes. Applied to a whole graph, this rule gives the gradient for every input.

The chain rule states:

if y = f(u) and u = g(x), then   dy/dx = (dy/du) · (du/dx)

A three-node graph

Let a = 2, b = 1 and x = 3, and define:

u = a · x
v = u + b
y = v²

The forward pass gives u = 6, v = 7 and y = 49.

The backward pass starts at ∂y/∂y = 1 and walks left:

∂y/∂v = 2v = 14                 # from y = v²
∂y/∂u = ∂y/∂v · ∂v/∂u = 14 · 1  # v = u + b, so ∂v/∂u = 1
∂y/∂b = ∂y/∂v · ∂v/∂b = 14 · 1
∂y/∂a = ∂y/∂u · ∂u/∂a = 14 · x = 14 · 3 = 42
∂y/∂x = ∂y/∂u · ∂u/∂x = 14 · a = 14 · 2 = 28

Each step multiplies the slope arriving from the right by the local slope. That is the whole method.

Common mistakes

• Forgetting a factor. ∂y/∂a is ∂y/∂u · x, not just ∂y/∂u. The local slope of u = a·x with respect to a is x.
• Treating addition as having a slope other than 1. For v = u + b, both ∂v/∂u and ∂v/∂b are 1.
• Reading the chain from the wrong end. The product starts at the output and moves back to the input.

Check yourself

1. Why is ∂y/∂b equal to ∂y/∂v here? Because v = u + b, and the slope of addition with respect to either input is 1.
2. If a were 5 instead of 2, what would ∂y/∂x be? 14 · a = 14 · 5 = 70.

Key takeaways

• The chain rule multiplies slopes along a path.
• On a graph, you multiply the incoming gradient by each node's local slope.
• Addition passes gradients through unchanged. Multiplication scales them by the other factor.
• Checking one derivative with a small nudge catches most algebra mistakes.

Further learning

The next question is what happens when a layer outputs a vector, not a single number: Jacobians and vector-Jacobian products.