Transformers and language models
The transformer architecture
Every component of a modern decoder-only LLM, from embeddings to the LM head, with parameter and FLOP counts.
Self-attention is one sublayer. A transformer is the full stack: token and position embeddings, repeated blocks of attention and MLPs with residuals and norms, and an LM head that turns the final residual stream into vocabulary logits. Modern open models (Llama, Mistral, Qwen, and others) are decoder-only: causal attention, no encoder.
If self-attention is the verb, the architecture is the sentence. Practitioners who only memorize "stack of layers" struggle to estimate VRAM, read model cards, or spot when a paper quietly switched from multi-head attention to grouped-query attention.
Concepts in this section
Read them in order. Each one builds on the one before.
| Step | Concept | What it answers |
|---|---|---|
| 1 | Embeddings and rotary positions | How does a token id become a vector with order? |
| 2 | The transformer block | What is inside one layer? |
| 3 | Parameter and FLOP counts | How big is the model, and how many FLOPs per token? |
| 4 | Encoder vs decoder transformers | BERT, T5 or GPT — which attention pattern? |
The data path in one paragraph
Id t becomes row E[t]. RoPE rotates Q and K by position. Each layer does RMSNorm, attention, residual add, RMSNorm, MLP, residual add. A final norm and the LM head produce logits over the vocabulary. Training runs all positions in parallel with a causal mask. Decode runs one token at a time against a KV cache.
Non-embedding parameters are about 12 L d² for classic blocks. Forward is about 2N FLOPs per token. Training is about 6N. A 7B-class model with d=4096 and L=32 lands near 6.4B non-embedding parameters, plus about 0.5B for a 128k embedding table.
When a paper says "7B transformer," decode L, d, heads, vocab, tied embeddings, GQA and MoE before comparing to another 7B. Residual plus pre-norm is why deep stacks train. If you remove residuals, depth collapses. Width (d) buys a richer residual stream. Depth buys more serial computation inside one forward pass. Frontier recipes grow both together.
GPT-2-style blocks use LayerNorm, learned absolute positions and a GELU 4d MLP. Llama-style blocks swap in RMSNorm, RoPE, SwiGLU and often grouped-query attention. Same skeleton, different defaults. Prefer reading one reference implementation end to end over collecting blog diagrams.
Common mistakes
- Comparing two 7B models by the badge. The layout (GQA, SwiGLU, vocab, tying) moves the real count.
- Using 2N as the training cost. Training is about 6N per token.
- Skipping a reference implementation. nanoGPT or Llama labeled against this map beats a pile of blog diagrams.
Key takeaways
- Decoder-only LLMs: embed → stack of (RMSNorm→attn→residual→RMSNorm→MLP→residual) → LM head.
- RoPE encodes relative position by rotating Q/K. It replaced absolute learned positions in most new models.
- The residual stream is the information highway. Sublayers read and write additives.
- Non-embedding params ≈ 12 L d² for classic blocks. Forward ≈ 2N FLOPs/token. Train ≈ 6N.
- BERT versus T5 versus GPT differ mainly in attention visibility and whether there is an encoder.
Further learning
Karpathy builds a GPT end to end in code — watch after you can sketch the block on a whiteboard.
Related
Self-attention
The operation at the core of every modern language model. Each position builds a weighted average of the others, with weights set by query-key matches.
How transformers work (practitioner view)
Just enough transformer internals to predict model behavior — attention, context, and why the KV cache shapes pricing.
Inference arithmetic
Napkin math for any model on any GPU: weight memory, decode and prefill time, batch limits and KV budget.