Seq2seq and the fixed-vector bottleneck

Encoder–decoder RNNs mapped one sequence to another, which made neural translation possible. The encoder read the source tokens and handed the decoder a single vector. That vector had to hold the entire sentence. Long inputs degraded. That crush is the fixed-vector bottleneck.

The big idea

Around 2014, encoder–decoder RNNs (Sutskever et al., Cho et al.) did translation in two steps:

1. An encoder RNN reads source tokens and finishes with a hidden state c (one vector).
2. A decoder RNN generates target tokens conditioned on c.

One fixed-size vector must hold every detail of the source. A 5-word sentence might fit. A 40-word sentence does not. Information is crushed through a bottleneck, and then the decoder has to unpack it token by token.

Walking through teacher forcing

Seq2seq decoders are trained with teacher forcing: at training time the decoder receives the ground-truth previous token, not its own prediction. That stabilizes training. It also creates exposure bias. At generation time the model must consume its own mistakes and can drift.

Scheduled sampling (sometimes feed model predictions during training) is one mitigation. Modern language models mostly accept teacher forcing and rely on scale plus decoding tricks.

RNN decoders for translation traditionally used beam search: keep the top-k partial hypotheses by log-probability. For open-ended generation (stories, chat), beam search produces dull, repetitive text. Sampling with temperature or top-p fits better. That decoding story continues unchanged into transformer LMs. See sampling.

A mailroom picture: the encoder stuffs a whole letter into one envelope. The decoder has to reconstruct the letter from that envelope. The picture breaks once attention lets the decoder look back at every source word instead of only the envelope.

Why this still matters

If you maintain an old LSTM seq2seq, adding attention is usually the highest-ROI upgrade before a full rewrite. For new production NLP, use a pretrained transformer unless constraints force otherwise (tiny microcontroller, strict streaming with tiny state, legacy stack).

The objective never changed. Next-token (or next-target-token) cross-entropy runs from n-grams through RNNs to GPT. What changed is the inductive bias and the hardware fit.

Common mistakes

• Blaming the decoder when the encoder state is the limit. A better decoder cannot recover facts that never fit in c.
• Training with teacher forcing and evaluating only with teacher forcing. Generation is free-running. Measure that.
• Using beam search for chat. It is for tasks with a small set of good targets, such as translation.

Check yourself

1. In vanilla seq2seq, what must a 20-token source sentence fit into? Answer: One fixed-size vector, the encoder's final hidden state.
2. What is teacher forcing? Answer: Feeding the ground-truth previous token into the decoder at train time, rather than the model's own prediction.

Key takeaways

• Seq2seq encodes a sequence, then decodes another sequence from that encoding.
• A single context vector bottlenecks long inputs.
• Teacher forcing stabilizes training and creates exposure bias at generation time.
• Beam search fits translation. Sampling fits open-ended text.

Further learning

Next: Bahdanau attention, the fix for the bottleneck.