Derivatives of common functions

Neural networks reuse a small set of functions, and their derivatives are short. Knowing them lets you check backprop code by eye. This page covers the activation functions, then softmax with cross-entropy, which is the most common loss for classification.

Sigmoid

σ(z)  = 1 / (1 + e^(−z))
σ'(z) = σ(z) · (1 − σ(z))

Saturation is why deep stacks of sigmoids train slowly. The gradient shrinks at every layer that is saturated.

Tanh

tanh'(z) = 1 − tanh(z)²

Tanh has the same saturation problem at large |z|. Its outputs are centered on zero, which is why it was preferred over sigmoid in hidden layers for a long time.

ReLU

ReLU(z)  = max(0, z)
ReLU'(z) = 1 if z > 0, else 0

ReLU does not saturate for positive inputs, so gradients pass through unchanged. The cost is that a unit whose input is always negative gets no gradient at all. It is a dead unit. The worked training step shows one. At exactly z = 0 the subgradient can be anything from 0 to 1, and implementations usually choose 0.

Softmax with cross-entropy

For classification, logits z become probabilities p = softmax(z). The loss against a one-hot label y, with true class k, is:

L = −log p_k

After the chain rule, the gradient with respect to each logit collapses to:

∂L/∂z_i = p_i − y_i

Deriving this from scratch means multiplying the softmax Jacobian, diag(p) − p pᵀ, by the loss gradient. Most of the terms cancel, which is why frameworks fuse the two operations.

Summary table

Common mistakes

• Implementing softmax and cross-entropy separately in float32. The result can be correct and still unstable. Use log_softmax or the fused cross-entropy from your framework.
• Ignoring saturation. A sigmoid unit with |z| = 5 passes back almost no gradient. Check the activation statistics when a deep network stops learning.
• Assuming the derivative of ReLU at zero is well defined. It is not, so implementations pick a convention.

Check yourself

1. What is σ'(0)? 0.25, the steepest point of the sigmoid.
2. Why does a saturated sigmoid slow learning? Its derivative is near zero, so the gradient reaching earlier layers is tiny.

Key takeaways

• Sigmoid and tanh saturate, which shrinks gradients. ReLU does not saturate on the positive side, but a unit can die.
• Softmax with cross-entropy has the gradient p − y.
• Print p − y on one example when debugging. A confident correct prediction gives a small gradient.

Further learning

Curvature is the next idea, and it explains how big a learning step can safely be: curvature and second derivatives.