Vanishing gradients and LSTMs

Train an RNN on a long sequence and two things go wrong. Gradients can explode into NaNs, or they can vanish so early tokens get almost no learning signal. Both come from the same fact: the backward pass multiplies the same Jacobian once per timestep.

The big idea

Suppose each step multiplies the backward signal by a Jacobian with norm 1.1. Over T = 100 steps you get 1.1^100 ≈ 13780 amplification. Clip or truncate, or the run dies. If the norm is 0.9, you get 0.9^100 ≈ 2.7×10⁻⁵. Early tokens receive almost no gradient.

Gradient clipping caps the global grad norm and tames explosions. Vanishing needs architectural help: gates, or later non-recurrent paths (attention).

Walking through LSTM gates

LSTMs add a cell state highway and three gates:

• Forget gate: what to drop from the cell state.
• Input gate: what new information to write.
• Output gate: what to expose as the hidden state.

The cell state updates by addition, not by a long chain of multiplies. Local derivatives on that path can sit near 1. That is the mathematical content of the "constant error carousel" in the original papers.

Gates are soft switches, not hard bits. You do not need to memorize the four equations on sight. You do need the picture: a highway that can carry a signal across many steps, with learned on-ramps and off-ramps.

GRUs simplify to reset and update gates, with fewer parameters. Quality is often similar. GRU is cheaper. LSTM sometimes wins on long, noisy sequences. As of the mid-2010s both were standard. For new NLP work, skip both and use a transformer unless you have a streaming or tiny-footprint constraint.

An analogy: a vanilla RNN is a leaky bucket passed down a line. Each person squeezes it. An LSTM is a bucket with a lid the line can choose to keep shut. The analogy breaks for really long dependencies, where even LSTMs struggle and attention later gives a direct path between distant positions, so the product of many Jacobians is unnecessary.

Bidirectional encoders

For classification and labeling, a bidirectional RNN runs a forward and a backward recurrence and concatenates hidden states, so each position sees left and right context. Classic for named-entity recognition and as the encoder in older speech recognition. Autoregressive generation cannot peek rightward, so decoders stay unidirectional. ELMo-style contextual embeddings were bidirectional-LM-based. BERT later replaced that niche with bidirectional transformers.

Common mistakes

• Clipping and thinking vanishing is solved. Clipping stops explosions. Vanishing needs gates or attention.
• Using a deep LSTM as a default for new NLP. Transformers parallelize better on GPUs.
• Letting a bidirectional encoder leak into a generator. Future context at train time is cheating at generate time.

Check yourself

1. Jacobian norm 0.9 over 100 steps. Roughly how much of the early-token gradient survives? Answer: About 0.9^100 ≈ 2.7×10⁻⁵, so almost none.
2. Which LSTM path is designed so local derivatives can stay near 1? Answer: The cell state, because it updates by addition.

Key takeaways

• Long unrollings multiply Jacobians. Norms below 1 vanish, norms above 1 explode.
• Clipping tames explosions. LSTMs and GRUs mitigate vanishing with gated cell highways.
• GRU is the cheaper cousin. For most new NLP, use a transformer.
• Attention later replaces the long product with a direct path between positions.

Further learning

Next: seq2seq and the fixed-vector bottleneck.

Understanding LSTM Networks (Chris Olah)Understanding LSTM Networks -- colah's blogcolah.github.io