A tiny autograd class

The whole idea of backprop fits in one small class. Each value remembers its parents and a rule for passing gradients back to them. A backward function then visits the values in the right order and runs those rules. This is the same design as Karpathy's micrograd, written from scratch.

Step 1: the forward pass records the graph

This version supports addition and multiplication. Each result stores its parents and a backward_fn closure that knows how to pass gradients back.

class Value:
    def __init__(self, data, parents=()):
        self.data, self.grad, self.parents = data, 0.0, parents
        self.backward_fn = lambda: None

    def __add__(self, other):
        out = Value(self.data + other.data, (self, other))
        def back():
            self.grad += out.grad
            other.grad += out.grad
        out.backward_fn = back
        return out

    def __mul__(self, other):
        out = Value(self.data * other.data, (self, other))
        def back():
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad
        out.backward_fn = back
        return out

a, b, c, d = Value(2.0), Value(3.0), Value(4.0), Value(5.0)
L = (a * b + c) * d
print("L =", L.data)

Output:

L = 50.0

Each back function uses +=. That matters when a value is used more than once, as you will see below.

Step 2: run the rules backward

Add this to the same file. The backward function first orders the graph so that each value appears after its parents. Then it walks that order in reverse, starting from the loss.

def backward(root):
    order, seen = [], set()
    def visit(v):
        if v not in seen:
            seen.add(v)
            for p in v.parents:
                visit(p)
            order.append(v)
    visit(root)
    root.grad = 1.0
    for v in reversed(order):
        v.backward_fn()

backward(L)
print("dL/da", a.grad, "dL/db", b.grad, "dL/dc", c.grad, "dL/dd", d.grad)

Output:

dL/da 15.0 dL/db 10.0 dL/dc 5.0 dL/dd 10.0

These match the hand calculation on the backprop overview: ∂L/∂a = d · b = 15, ∂L/∂b = d · a = 10, and so on.

Why the ordering matters

The reverse order guarantees that a node's gradient is complete before its backward_fn runs. If a node feeds two operations, both of them must add their contributions first. Visiting children before parents would pass along an incomplete gradient.

Common mistakes

• Calling backward twice without resetting. Gradients accumulate. Running the example twice gives a.grad = 60, not 15. Reset the grads to 0 before each backward pass.
• Using = instead of +=. For y = x * x with x = 3, there are two paths from x to y. With =, the second path overwrites the first, and x.grad comes out as 3 instead of 6.
• Forgetting that this is scalar-only. Real frameworks do the same work on tensors. The class is for learning the mechanism.

Check yourself

1. Why does visit append a node after visiting its parents? Parents go into the list before their children. Reversing the list then puts each node before its inputs, so its gradient is complete when it is used.
2. What happens if you call backward(L) twice without resetting? The gradients keep adding up, so the numbers grow with each call.

Key takeaways

• Each operation stores its parents and a rule for passing gradients back.
• backward orders the graph, sets the loss gradient to 1, and runs the rules in reverse.
• Shared values need += so that every path contributes.
• Reset gradients before each backward pass, or they keep growing.

Further learning

Real layers work on matrices rather than scalars. The next page gives those formulas: gradients for linear layers and softmax.