Deep learning
Activation functions and vanishing gradients
Activations decide how each neuron bends. Sigmoid and tanh flatten out and starve deep networks of gradient. ReLU and its relatives keep the signal flowing, with a few failure modes to watch.
Every hidden layer ends in an activation function. The choice looks cosmetic, but it decides whether gradients survive a deep stack. A function that flattens out at both ends stops learning in the flat regions. The usual default today is ReLU or one of its smooth relatives.
The big idea
| Activation | Formula | Range | Notes |
|---|---|---|---|
| Sigmoid | 1 / (1 + e^(−z)) | (0, 1) | Saturates at both ends. Rarely used in hidden layers now. |
| Tanh | (e^z − e^(−z)) / (e^z + e^(−z)) | (−1, 1) | Centered at zero, but still saturates. |
| ReLU | max(0, z) | 0 and up | Cheap, and the default for years. Can "die". |
| GELU | Smooth, close to z · σ(1.702z) | Close to ReLU | Common in transformers. |
| SwiGLU | A gated product of two linear maps | Varies | Used in many LLM feed-forward blocks. Costs extra parameters. |
Saturating functions flatten for large inputs. Their slope goes toward zero there. Gradients flow backward by multiplying slopes through the layers, so a series of small slopes shrinks the gradient quickly. That is the vanishing gradient problem.
Walking through saturation
The sigmoid's slope is largest at z = 0, where it equals 0.25. At z = 6, the sigmoid is 0.9975, and its slope is about 0.0025.
Suppose the best case, a slope of 0.25 at every layer, and ten layers:
0.25¹⁰ ≈ 0.00000095The gradient reaching the first layer is about a millionth of its size at the output. The early layers barely learn.
ReLU's slope is 1 for every positive input. Ten layers of slope 1 give 1¹⁰ = 1, so the signal passes through depth when units are active.
Dying ReLU
A ReLU unit whose pre-activation is negative for all of your data outputs zero and has zero slope. It gets no gradient, so it never recovers. Lower learning rates, careful initialization, and variants like Leaky ReLU or GELU reduce the problem.
Trade-off
For new vision or tabular MLPs, start with ReLU or GELU. For transformer feed-forward blocks, match the architecture you are copying, which is often GELU or SwiGLU. Avoid sigmoid and tanh in deep hidden stacks, unless the output must lie between 0 and 1.
Common mistakes
- Using sigmoid in deep hidden layers. Gradients vanish quickly. Reserve sigmoid for probability outputs.
- Blaming the optimizer for vanishing gradients. The activation often causes them. Check the activation first.
- Assuming ReLU never fails. Dead units stay dead. Watch the fraction of units that output zero.
Check yourself
- What is the largest slope of the sigmoid, and where does it occur? Answer: 0.25, at
z = 0. - Why does ReLU help signals pass through depth? Answer: Its slope is 1 for positive inputs, so the gradient does not shrink at each layer.
Key takeaways
- Saturating activations flatten, and gradients shrink through many layers.
- ReLU keeps slope 1 for active units, which helps deep stacks.
- A dead ReLU unit gets no gradient and stays dead.
- Match the activation to the architecture you are building on.
Further learning
Training recipes, including initialization and normalization, are covered in training neural networks well.
Next, pick the output head and loss for the task: output heads and loss functions.
Related
Multilayer perceptrons and the forward pass
A multilayer perceptron stacks fully connected layers. Each layer is a matrix multiply plus a bias and an activation, and a whole batch moves through the stack as matrices.
Training neural networks well
Optimizers, schedules, initialization, normalization, regularization and a practical recipe for debugging loss curves.
Backpropagation from scratch
How training computes the gradient of the loss for every weight in one backward pass. Covers computational graphs, reverse-mode autodiff and the bugs that break training.