A worked training step

Here is one training step for a tiny network, by hand. It has two inputs, two hidden units with ReLU, and one output with squared-error loss. The numbers are small enough to check on paper. Once you can trace this, you can trace a real network the same way, one layer at a time.

The setup

x  = [1.0, 2.0],   target y = 1.0
W1 = [[0.1, 0.2],  b1 = [0, 0]     (two hidden units)
      [-0.1, 0.3]]
W2 = [0.5, -0.4],  b2 = 0

Forward pass

z1 = W1 x + b1 = [0.5, 0.5]
h  = ReLU(z1)  = [0.5, 0.5]
z2 = W2 · h + b2 = 0.5·0.5 + (−0.4)·0.5 = 0.05
L  = (z2 − y)² = (0.05 − 1)² = 0.9025

The prediction is 0.05, and the target is 1.0, so the loss is large.

Backward pass

Work from the loss back toward the inputs. Each line multiplies the incoming gradient by the local derivative.

∂L/∂z2 = 2(z2 − y)        = 2(0.05 − 1)   = −1.9
∂L/∂W2 = ∂L/∂z2 · h       = −1.9 · [0.5, 0.5]    = [−0.95, −0.95]
∂L/∂h  = ∂L/∂z2 · W2      = −1.9 · [0.5, −0.4]   = [−0.95, 0.76]
∂L/∂z1 = ∂L/∂h · 1[z1 > 0] = [−0.95, 0.76]   (both units are active)
∂L/∂W1 = outer(∂L/∂z1, x) = [[−0.95, −1.9], [0.76, 1.52]]

Each unit's gradient for W1 is its ∂L/∂z1 times the input x. The ReLU term 1[z1 > 0] is 1 for both units here, so the gradient passes through.

One update

With learning rate η = 0.1, each weight moves against its gradient: W ← W − η ∂L/∂W.

W2 ← [0.5, −0.4] − 0.1 · [−0.95, −0.95] = [0.595, −0.305]

The output weights move in the direction that raises the prediction. Repeating forward, backward and update is training.

A dead unit

Now shift the first hidden unit's bias to b1 = [0, −1]. The first unit still works, but the second one is pushed below zero:

z1 = [0.5, −0.5]     h = [0.5, 0]
z2 = 0.25            L = (0.25 − 1)² = 0.5625
∂L/∂z2 = −1.5        ∂L/∂W2 = [−0.75, 0]
∂L/∂h  = [−0.75, 0.6]
∂L/∂z1 = ∂L/∂h · 1[z1 > 0] = [−0.75, 0]
∂L/∂W1 = [[−0.75, −1.5], [0, 0]]

The second row of ∂L/∂W1 is zero. ReLU blocks the gradient wherever its input is not positive, so that unit cannot learn from this example. If it stays negative on every example, it is a dead unit. Its weights never move.

Common mistakes

• Forgetting the ReLU mask. The backward pass must zero the gradient wherever the forward input was not positive.
• Mixing up the outer product. ∂L/∂W1 is the outer product of the gradient at z1 with the input x, so its shape matches W1.
• Changing the learning rate and the gradient together. Update one number at a time when you debug.

Check yourself

1. Why is the second row of ∂L/∂W1 zero in the dead-unit example? Because z1 for that unit is negative, so ReLU's local derivative is 0 and blocks the gradient.
2. After one step with η = 0.1, why does W2 move toward [0.595, −0.305]? Its gradient is negative, so subtracting η times the gradient increases both entries.

Key takeaways

• Forward: compute each layer's values and the loss. Backward: multiply each local derivative by the incoming gradient.
• An update subtracts the learning rate times the gradient from every weight.
• ReLU blocks gradients wherever its input is not positive, so a unit can go dead.
• Tracing one example by hand shows exactly where each gradient comes from.

Further learning

The training neural nets page covers what happens across many steps, including learning rates and initialization.