Self-attention

Every modern language model is built around one operation: self-attention. It takes a sequence of token vectors and lets each position pull in information from the other positions it needs. The weights are not fixed. They depend on the content, so the model decides at each step which tokens matter.

Older sequence models read tokens one after another. Information from far back had to pass through every step in between. Attention lets any position connect to any other in one step, and all positions can be computed in parallel. That is why transformers train quickly and scale well.

The mechanism in one line: each position builds a query, every position offers a key, and the query-key match sets the weights used to mix the values.

Concepts in this topic

Read in this order. Each page builds on the one before it.

When attention looks wrong

Check these first. Most bugs are in one of them.

1. Did you scale by the head size √d_k, not by √d_model?
2. Is the causal mask upper-triangular with diagonal=1, so each token can still see itself?
3. Does softmax run over the keys (the last axis), not over heads?
4. Are you training in FP16? Prefer BF16, or a stable softmax kernel. Raw FP16 softmax is a common source of NaNs.
5. Did you compare your op with a library implementation on random inputs before blaming the optimizer?

Common mistakes

• Mixing up the three names. Self-attention takes Q, K and V from the same input. Cross-attention takes Q from one sequence and K and V from another. Causal attention is self-attention with future positions masked.
• Assuming the full n × n matrix exists. Fused kernels such as FlashAttention compute the result without storing that matrix, so you cannot always inspect the weights.
• Leaving training mode on in production. Dropout should be off at inference. Call model.eval(), or quality drops silently.

Key takeaways

• Self-attention is a content-based weighted average of value vectors.
• Weights come from query-key scores, scaled by √d_k and normalized with softmax.
• Masks act on scores before softmax: causal masks stop future peeking, padding masks ignore filler.
• Multi-head attention splits the width across heads, then concatenates and projects back.
• Cost grows with n², which is the root of long-context pressure.

Further learning

Grant Sanderson's visual walkthrough of attention is the best intuition before the paper.

Attention Is All You NeedThe dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an…arXiv.org The Illustrated Transformer (Jay Alammar)Discussions: Hacker News (65 points, 4 comments), Reddit r/MachineLearning (29 points, 3 comments) Translations: Arabic, Chinese (Simplified) 1, Chinese (Simplified) 2, French 1, French 2, Italian, Japanese, Korean, P…jalammar.github.io