The transformer block

Self-attention is one sublayer. A transformer block wraps it with a norm, a residual add, an MLP and another residual add. Stack L of those blocks and you have the body of a decoder-only LLM.

The big idea

A modern Llama-style block (one layer) looks like:

h = h + Attention(RMSNorm(h))
h = h + MLP(RMSNorm(h))          # MLP often SwiGLU

That is pre-norm: normalize, then sublayer, then residual add. Post-norm (norm after the add) was the 2017 default. Pre-norm trains more stably at depth.

RMSNorm scales by the root-mean-square of the vector (no mean centering, unlike LayerNorm):

RMSNorm(x) = x / RMS(x) * γ
RMS(x) = sqrt(mean(x²) + ε)

γ is a learned per-channel scale (d parameters per norm). See normalization layers.

Without residuals, stacking 32 nonlinear layers tends to destroy gradient signal. Residuals make each layer learn a refinement F(h) added to h, so the identity path carries gradients. See residual connections.

Walking through SwiGLU

Classic transformer FFN: σ(x W₁) W₂ with d_ff ≈ 4 d. Llama-style SwiGLU uses a gated linear unit:

SwiGLU(x) = (Swish(x W_gate) ⊙ (x W_up)) W_down

Three matrices instead of two. To keep parameter count similar to a 4d FFN, width is often set near d_ff ≈ (8/3) d (rounded to a multiple of 256). Most parameters in a dense transformer live in these MLPs. They are a natural place for mixture of experts to sparsify.

Modern stacks often drop attention biases and sometimes MLP biases entirely. SwiGLU gates are not optional decorations. They are first-class parameters in the W_gate matrix. Norm γ vectors are tiny in count but critical for scale.

Karpathy's nanoGPT is small enough to hold in your head:

1. wte / wpe — token and (absolute) position embeddings. GPT-2 style, not RoPE.
2. h — a list of blocks. Each block is LayerNorm → causal attention → residual → LayerNorm → MLP → residual (pre-norm).
3. MLP — expand 4× with GELU, project back (not SwiGLU).
4. Final LayerNorm plus linear LM head. Optional weight tying with wte.

Then compare to a Llama implementation: swap in RMSNorm, RoPE, SwiGLU, and maybe grouped-query attention.

Width (d) buys richer residual streams and bigger MLPs. Depth buys more serial computation inside one forward pass. Frontier recipes grow both roughly together.

Common mistakes

• Removing residuals and expecting depth to train. It usually will not.
• Copying GPT-2 GELU MLP math onto a Llama config. SwiGLU has three matrices and a different width.
• Forgetting norm γ at export. Generation breaks in ways that look like sampler bugs.

Check yourself

1. Write the two residual updates in a Llama-style block. Answer: h = h + Attention(RMSNorm(h)) then h = h + MLP(RMSNorm(h)).
2. Why is SwiGLU width often near (8/3) d rather than 4d? Answer: Three matrices instead of two. (8/3) d keeps parameter count close to a classic 4d FFN.

Key takeaways

• Pre-norm plus residual is why deep stacks train.
• The residual stream is the highway. Sublayers read and write additives.
• RMSNorm is LayerNorm without mean centering.
• SwiGLU is a gated MLP with three matrices. Most dense parameters live there.

Further learning

Next: parameter and FLOP counts.