Transformers and language models
Tokenization
Byte-pair encoding in depth: how vocabularies are built and why tokenization causes strange model behavior.
Models never see raw characters. They see tokens: integer IDs from a fixed vocabulary built once before training. Tokenization is the bridge between Unicode text and the embedding table. Get it wrong and you waste context, inflate non-English cost, and create failure modes that look like "the model can't spell" when letters were never separate symbols.
Treat the tokenizer as frozen infrastructure for the life of a base model. Pretraining, mid-training, SFT and inference must agree on the exact same byte-to-id map. Teams that "upgrade" tokenizers casually orphan checkpoints and invalidate eval fixtures. The right mental model is closer to a wire protocol than to a preprocessing nicety. The AI track covers token counts for product work. Here the internals are BPE mechanics, vocab-size trade-offs and pathologies.
Concepts in this section
Read them in order. Each one builds on the one before.
| Step | Concept | What it answers |
|---|---|---|
| 1 | Why subword tokenization | Why not characters or words? |
| 2 | Byte-pair encoding | How is the vocabulary built and how is a new string encoded? |
| 3 | WordPiece, SentencePiece and special tokens | What are the cousins of BPE, and how do templates fit in? |
| 4 | Tokenizer pathologies | Why does the model fail at spelling and arithmetic? |
Working in practice
Always count tokens with the model's tokenizer before sizing prompts or bills. When fine-tuning, freeze the tokenizer unless you have a deliberate vocab expansion plan. For multilingual UX, measure fertility on real user languages. Commit golden encode() fixtures next to any chat template.
A one-character mistake in a chat template changes token IDs and can tank a fine-tune. Fine-tunes that "don't follow instructions" are often template mismatches, not weak SFT data.
English-centric BPE often sits near 1.3 tokens per word. Other scripts can run 2–4× worse, which 4×s attention cost for the same character content because attention scales with tokens squared. Vocab size is the other lever: Llama 2 used about 32k, Llama 3 moved to about 128k for multilingual and code efficiency, at the cost of a larger embedding table.
Common mistakes
- Upgrading a tokenizer under a trained checkpoint. Embedding rows no longer match.
- Counting words and calling them tokens. Context windows and API bills are in tokenizer IDs.
- Scoring digit tasks as reasoning without controlling tokenization.
Key takeaways
- Subwords balance sequence length against vocab size. Pure chars are too long, pure words are too open.
- BPE greedily merges frequent adjacent pairs. Byte-level BPE covers all Unicode from 256 base bytes.
- WordPiece and Unigram/SentencePiece are cousins with different training criteria.
- Vocab size trades embedding parameters for compression. 128k-class vocabs favor multilingual and code efficiency.
- Many "dumb model" demos are tokenizer artifacts: spelling, arithmetic, trailing spaces, glitch tokens.
Further learning
Karpathy builds a GPT-2-compatible tokenizer from scratch:
minbpe (Karpathy)Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization. - karpathy/minbpeGitHubRelated
Tokens and context windows
How text becomes tokens, what that means for cost and limits, and how to budget a context window.
Pretraining a language model
Data, objective and compute: how a base model is trained on trillions of tokens.
The transformer architecture
Every component of a modern decoder-only LLM, from embeddings to the LM head, with parameter and FLOP counts.