Encoder vs decoder transformers

"Transformer" is a family, not one diagram. The original 2017 model was encoder–decoder. BERT dropped the decoder. GPT dropped the encoder. Those three attention patterns still explain almost every open checkpoint.

The big idea

Decoder-only won for LLMs because one stack serves pretraining, SFT and chat. Scaling recipes and inference stacks optimized for it. Next-token prediction transfers surprisingly far.

Walking through GPT-2 versus Llama

Same skeleton, different defaults. If you can redraw both, you can read almost any open LLM modeling_*.py.

Follow a single new decode token through a Llama-style stack:

1. Id t → row E[t] (d,).
2. RoPE rotates Q/K using position index n.
3. For each layer: RMSNorm → attention (query against cached K/V plus itself) → add → RMSNorm → SwiGLU → add.
4. Final RMSNorm → W_U → logits (V,).
5. Sampler picks next id. Append. Repeat.

Training differs only in that all positions run in parallel with a causal mask and loss on every position's next-token target. If you can narrate those five steps with shapes, interviews and code review get easier.

Prefer reading one reference implementation (nanoGPT or Llama) end to end over collecting blog diagrams.

After L layers, take h ∈ R^{n × d} (or just the last position at decode time). The LM head is a linear map R^d → R^V. Training loss is cross-entropy of softmax(logits) versus the next-token id. At inference, sampling turns logits into a token.

Common mistakes

• Calling BERT a language model in the GPT sense. BERT is bidirectional. It does not generate left to right by default.
• Comparing a T5 to a Llama by parameter count alone. One has an encoder. The attention patterns differ.
• Dropping the causal mask in a decoder-only trainer. The model then cheats by reading the future.

Check yourself

1. Why did decoder-only win for chat LLMs? Answer: One stack serves pretrain, SFT and chat, and the inference stack optimized for causal decode.
2. Name two Llama-style replacements for GPT-2 defaults. Answer: RMSNorm instead of LayerNorm, RoPE instead of learned absolute positions, SwiGLU instead of GELU 4d (any two).

Key takeaways

• BERT looks both ways. T5 has encoder plus decoder. GPT looks only left.
• Decoder-only is the default generative LLM.
• GPT-2 versus Llama is the same skeleton with different norms, positions, attention and MLP.
• Narrating one decode step with shapes is the fluency test.

Further learning

Karpathy builds a GPT end to end in code — watch after you can sketch the block on a whiteboard.

The Annotated Transformer (Harvard NLP)The Annotated Transformernlp.seas.harvard.edu

Back to the map: the transformer architecture.