Deep learning
Backpropagation from scratch
How training computes the gradient of the loss for every weight in one backward pass. Covers computational graphs, reverse-mode autodiff and the bugs that break training.
Training a neural network means adjusting every weight so a single number, the loss, gets smaller. To do that, you need the slope of the loss with respect to every weight. The efficient method is reverse-mode automatic differentiation. In neural networks it is called backpropagation. Frameworks hide it behind loss.backward(), but if you cannot picture what that call does, you will struggle to debug training.
The problem is scale. A model can have millions of weights. Estimating each slope by nudging one weight and re-running the forward pass would take one pass per weight. Backprop gets all of them from one forward pass and one backward pass, because it works backward from the single scalar loss.
Concepts in this topic
Read in this order. Each page uses the one before it.
| Step | Concept | What it answers |
|---|---|---|
| 1 | Computational graphs and reverse mode | How does a forward pass record what to differentiate? |
| 2 | A tiny autograd class | How does the backward walk work in code? |
| 3 | Gradients for linear layers and softmax | What are the matrix forms for real layers? |
| 4 | A worked training step | What does one full forward and backward step look like with numbers? |
The chain rule behind all of this is on the calculus page.
Common bugs
Most training failures from backprop fall into a few patterns.
| Bug | Symptom | Fix |
|---|---|---|
| Forgot to zero gradients | Updates look like the sum of many past batches | Zero the gradients before each backward pass |
| In-place op on a tensor needed for backward | Runtime error, or silently wrong gradients | Use out-of-place ops on tracked tensors |
detach() or rebuilding a tensor from a Python float | Gradients stop flowing upstream | Detach only for metrics, not for the loss |
sum where you meant mean on the loss | Effective learning rate changes with batch size | Pick one reduction and stay consistent, usually mean |
| Train and eval modes mixed up | Validation looks strange | Call model.train() and model.eval() correctly |
Important
Gradients accumulate by design. Every call to backward() adds to the existing .grad. If you forget to reset them, you train on the sum of every past batch.
Memory: why activations dominate
Backprop needs the forward values, called activations, for the whole batch. For a deep network with long inputs, storing them costs more memory than the weights.
Adam keeps two moment buffers per weight, so its optimizer state is about twice the size of the weights. Activations often outgrow both as batch size and sequence length rise.
Trade-off
Store or recompute. Storing every activation makes the backward pass fast but uses a lot of memory. Gradient checkpointing keeps only some activations and recomputes the rest during the backward pass. The source's rough estimate is O(√L) memory for L layers, at about 33% extra compute. Reach for it when you run out of memory before you run out of compute.
Common mistakes
- Reusing a value without adding its gradients. When a node feeds two paths, both paths contribute to its gradient. Use
+=, not=. - Ignoring shapes. The gradient of a tensor always has the same shape as the tensor. If a shape is wrong, the bug is usually in a broadcast or a transpose.
- Stopping the graph by accident. Calling
.item()or converting to NumPy severs the connection. Your loss should stay connected.
Key takeaways
- Backprop is reverse-mode autodiff on a scalar loss: one forward pass and one backward pass give every gradient.
- The forward pass stores the graph and the intermediate values. The backward pass multiplies local derivatives by the gradient flowing in.
- Always zero gradients before the next backward pass, and avoid in-place ops on tensors that backward needs.
- Activation memory, not weight memory, often limits training. Checkpointing trades compute for memory.
Further learning
Karpathy builds micrograd live, which is the best companion to this topic. Pause and type along.
3Blue1Brown's intuition pass before the calculus:
Related
Calculus and gradients
Derivatives, partial derivatives, the chain rule and gradients. This is the math that turns a loss number into parameter updates.
PyTorch fundamentals
Tensors, autograd, modules, data loaders and a clean training loop. The daily-driver API for deep learning. Start here for the map.
Training neural networks well
Optimizers, schedules, initialization, normalization, regularization and a practical recipe for debugging loss curves.