PyTorch autograd

To train a network you need the gradient of the loss with respect to every weight. Autograd is PyTorch's engine for that. You write the forward computation, and PyTorch records each operation. Calling backward() then applies the chain rule back through the record. It is the backpropagation algorithm, run for you.

Walking through a tiny graph

Let w hold two weights and x one input row. The output is a scalar:

import torch

w = torch.tensor([[1.0], [2.0]], requires_grad=True)
x = torch.tensor([[3.0, 4.0]])

y = (x @ w).sum()          # 3 * 1 + 4 * 2 = 11
y.backward()
print(y.item())
print(w.grad)             # dy/dw = x transposed

Output:

11.0
tensor([[3.],
        [4.]])

Trace it by hand. y = 3·w₁ + 4·w₂, so dy/dw₁ = 3 and dy/dw₂ = 4. Those are the numbers in w.grad. Autograd did the same work for a network with 200,000 weights.

Turning tracking off

Tracking costs memory, so turn it off when you do not need gradients:

• torch.no_grad() is a block where nothing is tracked. Use it for evaluation and manual weight updates.
• tensor.detach() returns a tensor with the same values and no history. Use it when you only want to log a value.
• torch.inference_mode() is a stricter, faster version of no_grad for pure inference.

with torch.no_grad():
    z = w * 2
print(z.requires_grad)    # False

d = (w * 2).detach()
print(d.requires_grad)    # False

Two rules that catch people

backward() on a non-scalar needs an extra gradient= argument. Reduce to a scalar loss first, which is what training does anyway.

Gradients accumulate. Each backward() adds to .grad rather than replacing it. Two passes without zeroing give double the gradient:

w = torch.ones(2, requires_grad=True)
(w * 3).sum().backward()
(w * 3).sum().backward()
print(w.grad)

Output:

tensor([6., 6.])

Each pass contributed 3. This is why training loops call zero_grad() before every step.

Common mistakes

• Forgetting to zero gradients. Each step adds to the last one, so updates grow wrong.
• Detaching by accident. A .detach() in the middle of the model cuts the path, and no gradient reaches the earlier layers.
• Leaving tracking on in evaluation. It wastes memory. Wrap validation in inference_mode().

Check yourself

1. After two backward passes without zeroing, what is w.grad compared with one pass? Answer: the sum of both gradients, so it is doubled when both passes are identical.
2. Why call detach() before logging a loss? Answer: it removes the autograd history, so logging does not keep the graph alive in memory.

Key takeaways

• Autograd records operations on tensors with requires_grad=True and computes gradients with backward().
• .grad is the gradient of the scalar loss with respect to each tracked tensor.
• Use no_grad, detach or inference_mode when you do not need gradients.
• Gradients accumulate, so zero them every step.

Further learning

After the micrograd video, the same ideas appear in real PyTorch:

Next, see how weights and data are organized in modules and data.