Bahdanau attention

The seq2seq bottleneck asked one vector to hold a whole sentence. Attention (Bahdanau et al., 2015) lets the decoder, at each step, look at all encoder hidden states. The decoder state queries the encoder states. A weighted sum becomes the context for predicting the next target token.

The big idea

score_{t,i} = score(s_t, h_i)
α_{t,i} = softmax_i(score_{t,i})
context_t = Σ_i α_{t,i} h_i

s_t is the decoder state at step t. h_i is encoder state i. The scores become weights α, and the context is a weighted average of the encoder states.

This is the conceptual parent of self-attention: the same query-key-value idea, later applied inside a single sequence without recurrence. See self-attention.

A photocopier picture: instead of one summary memo, every decoder step gets a fresh highlight of the source page. The picture breaks because the highlights are learned, not chosen by a human.

Walking through a 20-token sentence

Translating a 20-token French sentence: without attention, token 20 must encode everything in one vector. With attention, when producing the English word for a French adjective, the decoder can put mass on the encoder positions of that adjective. Soft alignment, learned end to end.

Classical statistical machine translation kept explicit word-alignment tables. Bahdanau attention learns those alignments as a byproduct of differentiable scoring. Visualizing α_{t,i} over a sentence pair was a staple debugging tool: you could see the decoder looking at the right source word. Self-attention maps in transformers are the same object without the encoder/decoder split.

Why recurrence lost to transformers

GPUs train fast when work is parallel. RNNs require h_t before h_{t+1}, so sequence length becomes wall-clock depth. Attention plus transformers process all positions in parallel within a layer, using masked self-attention for causality in language models.

Other RNN pains: limited path length for distant token pairs even with LSTMs, and harder tooling. By about 2018–2020, transformers won most NLP benchmarks. RNNs remain in niches: some streaming speech recognition, tiny on-device models, and teaching.

Tokenization still defines what "tokens" are. That problem never belonged to RNNs uniquely.

At very long T, attention's T² cost hurts, which is why later work on efficient attention and state-space models exists. See modern architectures. The historical hinge remains: recurrence serializes the forward pass, and attention does not.

Common mistakes

• Calling this self-attention. Bahdanau attention is cross-attention from decoder to encoder. Self-attention uses Q, K and V from the same sequence.
• Assuming the alignment is a hard pointer. It is a softmax distribution, so several source words can share the mass.
• Keeping LSTMs for new NLP by default. Parallel attention won the hardware argument at scale.

Check yourself

1. What does the decoder query in Bahdanau attention? Answer: All encoder hidden states, scored against the current decoder state.
2. Why did transformers train faster than RNNs on GPUs? Answer: Attention computes all positions in a layer in parallel. An RNN must finish h_t before h_{t+1}.

Key takeaways

• Attention lets each decoder step read every encoder state, which breaks the fixed-vector bottleneck.
• The weights are a softmax over scores. That is a soft alignment.
• Self-attention is the same idea inside one sequence, without recurrence.
• Recurrence's sequential dependency wasted GPUs. Parallel attention won at scale.

Further learning

Read self-attention next while the bottleneck story is fresh.

Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau)Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that c…arXiv.org