Derivatives as slopes

Training asks one question over and over: if this weight moves a little, does the loss go up or down? The answer is a derivative. A derivative is the slope of a function at a point, meaning how much the output changes for a tiny change in the input.

f'(x) = lim (h → 0)  [f(x + h) − f(x)] / h

For f(x) = x², the derivative is f'(x) = 2x. At x = 3, the slope is 6. A small step Δx changes the output by about 6 · Δx.

Two ways to get the number

Analytical derivatives come from rules. For example, d/dx xⁿ = n xⁿ⁻¹. Training uses these, because they are exact and fast.

Numerical derivatives approximate the slope with a small step. The central difference is the usual form:

f'(x) ≈ [f(x + h) − f(x − h)] / (2h)

f(3.001) = 9.006001
f(2.999) = 8.994001
(9.006001 − 8.994001) / 0.002 = 6.0

Checking a gradient

Numerical gradients are slow, because each parameter needs its own pair of forward passes. Their value is as a check. If the analytical gradient from your backward code disagrees with the numerical one, the backward code is wrong.

import numpy as np

def grad_check(f, x, analytical, h=1e-5):
    numerical = np.zeros_like(x)
    for i in range(x.size):
        xp, xm = x.copy(), x.copy()
        xp.flat[i] += h
        xm.flat[i] -= h
        numerical.flat[i] = (f(xp) - f(xm)) / (2 * h)
    return np.linalg.norm(numerical - analytical) / (
        np.linalg.norm(numerical) + np.linalg.norm(analytical) + 1e-12
    )

w = np.array([0.5, -1.2, 3.0])
rel = grad_check(lambda v: np.sum(v ** 2), w, 2 * w)   # f(w) = ||w||^2, gradient 2w
print(f"relative error {rel:.1e}")

Output:

relative error 4.4e-12

A relative error near 1e-7 or smaller means the two methods agree. Check in float64 when you can. Mixed precision and non-smooth ops such as ReLU make the error larger, so use a looser tolerance there.

Common mistakes

• Using a step size that is too large. The slope estimate is then the slope of a chord, not the tangent. Shrink h until the answer stops changing.
• Using a step size that is too small. Floating-point rounding dominates, and the estimate becomes noise.
• Checking only one point. A backward pass can be right at one input and wrong elsewhere. Test a few random inputs.

Check yourself

1. What is the slope of f(x) = x² at x = −2? 2x = −4, so the output decreases as x increases.
2. Why use the central difference rather than (f(x + h) − f(x)) / h? The central difference cancels the first-order error term, so it is more accurate for a smooth function at the same h.

Key takeaways

• A derivative is the local slope: how much the output moves per unit change in the input.
• Training uses exact, analytical derivatives. Numerical ones are for checking.
• The central difference is the standard finite-difference formula.
• A gradient check compares analytical and numerical gradients on a small input.

Further learning

Next, the same idea with many inputs: partial derivatives and gradients.