Calculus and gradients

Training a model means repeatedly answering one question: if I nudge this weight a little, does the loss go up or down? The answer is a derivative. With millions of weights you need a gradient, a vector of derivatives, and the chain rule computes it efficiently. Autograd libraries hide this arithmetic. This topic makes it visible, so you can read backprop code, spot exploding gradients and check your math.

Linear algebra is the language of the forward pass. Calculus is the language of the backward pass. Every optimizer step assumes you can compute the gradient of the loss. Any custom loss you design must be differentiable, at least almost everywhere.

Concepts in this topic

Read in this order. Each page builds on the one before it.

Backpropagation applies the chain rule to a whole network. See backpropagation from scratch.

Common mistakes across the topic

• Trusting autograd silently. A detached tensor gives a zero gradient with no error. After backward(), print param.grad for a few parameters.
• Blaming autograd for NaNs. NaN losses usually come from exploding gradients, log(0) in cross-entropy, or division by zero.
• Using numerical gradients in training. They need one forward pass per parameter. Use them to check code, not to train.

In practice

• Vanishing gradients through deep stacks of sigmoids or tanh pushed the field toward ReLU, residual connections and normalization.
• A residual connection y = x + F(x) adds an identity path, so ∂L/∂x includes a +1 term. Gradients have a route that avoids many weight matrices.
• When writing a custom op, check its gradient against finite differences on a small input before you trust it.

Key takeaways

• A derivative is a local slope. A gradient is the vector of partial derivatives, pointing uphill.
• The chain rule, applied on a computational graph, is backpropagation.
• Frameworks compute vector-Jacobian products, not full Jacobians. That is why reverse mode scales.
• Know the derivatives of sigmoid, tanh and ReLU, and the p − y result for softmax with cross-entropy.

Further learning

3Blue1Brown's essence of calculus builds the idea that slopes multiply under composition:

The chain rule chapter is the direct prerequisite for backprop:

Mathematics for Machine Learning: calculus chaptersCompanion webpage to the book “Mathematics for Machine Learning”. Copyright 2020 by Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong. Published by Cambridge University Press.Mathematics for Machine Learning CS231n: Backpropagation notesCourse materials and notes for Stanford class CS231n: Deep Learning for Computer Vision.cs231n.github.io