How transformers work (practitioner view)

You can ship LLM products without deriving attention from scratch. You cannot debug cost, latency, "lost in the middle," or prompt caching without a practitioner model of what the network is doing. This page is that map. For layer norms, RoPE and training math, use the transformer architecture notes. For GPU-level prefill and decode, use inference.

A decoder-only model scores the next token given the tokens so far. Generation is a loop: encode, score, sample, append, repeat. Everything product-shaped — chat, tools, agents — is this loop with different strings in the prompt and different stop rules.

Why the internals leak into the product

Attention lets every earlier token influence the next one. That is why leaked PII and hostile retrieved text matter. Position is uneven: facts at the edges of a long prompt are used more reliably than facts in the middle. Prefill sets time to first token; decode sets stream speed and is why output tokens cost more. The KV cache reuses keys and values, which is why a stable prefix is cheap and a request ID at the front is not.

Stop imagining the model as a search engine with a chat skin. It is a next-token mixer over the current context plus frozen weights.

Concepts in this section

Read them in order. Each one builds on the one before.

A loop, in one picture

1. Your app assembles roles and content into a token sequence.
2. The serving stack prefills that sequence, writes K and V into the cache, and produces logits for the next position.
3. Sampling picks a token; decode appends it using the cache.
4. Repeat until a stop rule fires.
5. On tool-use, your app runs the tool, appends the result, and resumes — often with most of the prefix still cached.

That loop explains product bugs. A stuck agent is often unbounded decode. A slow first paint is prefill. A slow stream is decode throughput. A cost spike after "adding memory" is a longer prefill every turn.

Common mistakes

• Stuffing the window because attention can look anywhere. Dilution and prefill cost both get worse.
• Putting a UUID at the start of the system prompt. The KV / prompt cache misses every call.
• Asking the model for a live or private fact with no retrieval and no tool. You will get a fluent guess.
• Blaming the model name for TTFT without checking input length and cache hit rate.

Key takeaways

• LLMs are next-token predictors in a loop. Products wrap that loop.
• Attention lets every context token influence the output — for better or worse.
• Long-context quality is uneven. Prefer edges over the middle for critical facts.
• Prefill sets TTFT. Decode sets stream speed. Output tokens are the expensive phase.
• KV / prompt caching rewards identical prefixes. Put stable content first.

Further learning

Best one-hour intuitive talk on what LLMs are:

Longer Karpathy deep dive into training and behavior:

The Illustrated Transformer (Jay Alammar)Discussions: Hacker News (65 points, 4 comments), Reddit r/MachineLearning (29 points, 3 comments) Translations: Arabic, Chinese (Simplified) 1, Chinese (Simplified) 2, French 1, French 2, Italian, Japanese, Korean, P…jalammar.github.io Lost in the Middle (arXiv:2307.03172)While recent language models have the ability to take long contexts as input, relatively little is known about how well they use longer context. We analyze the performance of language models on two tasks that require …arXiv.org

Start with the next-token loop.