Computational graphs and reverse mode

A neural network's forward pass is a chain of simple operations: add, multiply, apply a nonlinearity. Each operation becomes a node in a computational graph. Backprop walks that graph from the loss back toward the inputs, applying a local rule at each node. One backward sweep gives the gradient for every input.

The forward pass builds the graph

Take the expression L = (a·b + c) · d with scalars. The forward pass computes it in steps:

e = a * b
f = e + c
L = f * d

Each step is a node. Its inputs are the parents, and the value it produces is stored, because the backward pass will need it.

The backward pass walks it in reverse

Start at the loss with ∂L/∂L = 1. Then move backward. At each node, multiply the gradient arriving from above by the local derivative, and send the result to each parent.

Use a = 2, b = 3, c = 4, d = 5. The forward values are e = 6, f = 10 and L = 50. The backward values are:

∂L/∂f = d = 5            # L = f · d
∂L/∂d = f = 10
∂L/∂e = ∂L/∂f · 1 = 5    # f = e + c, so the local derivative is 1
∂L/∂c = 5
∂L/∂a = ∂L/∂e · b = 15   # e = a · b
∂L/∂b = ∂L/∂e · a = 10

Check ∂L/∂a by nudging it. Change a to 2.001: e = 6.003, f = 10.003, L = 50.015. The change in L is 0.015, divided by 0.001, which is 15. It matches.

Shared values add their gradients

If a value feeds two operations, both paths contribute. Its gradient is the sum of the two incoming gradients. Forgetting to add is one of the most common bugs in hand-written autodiff.

Why the forward values are stored

The local rules need the forward values. For z = x · y, the gradient for x is y times the incoming gradient, so y must still be available. This is why training memory scales with activations, not just weights. The backprop overview covers the memory trade-off.

Common mistakes

• Treating the graph as built once. The graph is rebuilt on every forward pass. A new batch means a new graph.
• Forgetting to add gradients on a shared node. Each path adds its own contribution.
• Calculating each gradient separately. The backward pass reuses the gradient flowing from above. Recomputing it from scratch for each parameter wastes the whole point of reverse mode.

Check yourself

1. In L = (a·b + c)·d, why is ∂L/∂c = d? Changing c by 1 changes f by 1, and L = f·d, so L changes by d.
2. Why is reverse mode the right choice for training? The loss is one number, and there are millions of parameters. One backward sweep from the loss gives all the gradients.

Key takeaways

• The forward pass records a graph of operations and stores their values.
• The backward pass multiplies each local derivative by the gradient flowing in from above.
• Shared values add gradients from every path.
• Reverse mode needs one backward sweep for a scalar output, which is why it suits training.

Further learning

The next page turns these rules into code: a tiny autograd class.