Sequence models and the road to attention

Language, audio and time series are sequences: the next element depends on what came before. Before transformers, the workhorse was the recurrent neural network (RNN) and its gated cousins. That lineage makes self-attention feel inevitable instead of magical.

A language model estimates P(x_t | x_1, …, x_{t−1}). Train by maximizing the likelihood of real text, which is minimizing next-token cross-entropy. Sample to generate. An n-gram baseline counts frequencies of token n-tuples. It is simple, sparse, and does not generalize to unseen contexts. The objective stays the same from counts through recurrence to transformers. What changes is the inductive bias and the hardware fit.

Concepts in this section

Read them in order. Each one builds on the one before.

A timeline

Rough mental timeline: n-grams → RNNs → LSTM/GRU → seq2seq → Bahdanau attention → self-attentional transformers (2017) → decoder-only LMs at scale. Each step fixed a pain the previous generation felt: sparsity of counts, vanishing gradients, fixed-vector bottlenecks, sequential GPU underuse, then scaling.

You still meet LSTMs in older production and in streaming niches. The design vocabulary now centers on attention. For learning, implement a character RNN or follow makemore before jumping to GPT. You will understand losses, embeddings and generation without attention machinery.

For production NLP, use a pretrained transformer unless constraints force otherwise. If you maintain an old LSTM seq2seq, adding attention is usually the highest-ROI upgrade before a full rewrite.

Interview angle: vanishing gradients, why LSTMs gate, the seq2seq bottleneck, what Bahdanau attention computes, why transformers parallelize better, and how teacher forcing differs from free-running generation.

Common mistakes

• Jumping to transformers without the bottleneck story. Q/K/V algebra clicks faster once you have felt the one-vector crush.
• Using packed-sequence bugs as "the model is bad". Padded RNN steps waste compute. Transformers use attention masks instead.
• Treating RNNs as obsolete everywhere. Edge speech recognition sometimes still likes constant-size state. Cloud LLMs accept KV-cache cost for quality.

Key takeaways

• Next-token prediction is the core language-model objective. N-grams are the baseline.
• RNNs compress history into h_t with shared weights. BPTT trains them.
• Vanishing and exploding gradients plague long unrollings. LSTMs and GRUs mitigate via gates.
• Seq2seq with a single context vector bottlenecks long inputs.
• Attention lets decoders read all encoder states. That is the bridge to transformers.
• Recurrence's sequential dependency wastes GPUs. Parallel attention won at scale.

Further learning

Karpathy's makemore intro builds language modeling from counts upward:

Understanding LSTM Networks (Chris Olah)Understanding LSTM Networks -- colah's blogcolah.github.io The Unreasonable Effectiveness of Recurrent Neural Networks (Karpathy)Musings of a Computer Scientist.karpathy.github.io Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau)Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that c…arXiv.org