Why subword tokenization

Models never see raw characters. They see tokens: integer IDs from a fixed vocabulary built once before training. Tokenization is the bridge between Unicode text and the embedding table. Get it wrong and you waste context, inflate non-English cost, and create failure modes that look like "the model can't spell."

Treat the tokenizer as frozen infrastructure for the life of a base model. Pretraining, mid-training, SFT and inference must agree on the exact same byte-to-id map. Teams that "upgrade" tokenizers casually orphan checkpoints. The right mental model is closer to a wire protocol than to a preprocessing nicety.

The AI track covers token counts for product work. Here the internals start with why subwords exist at all.

The big idea

Subword tokenization is a compression scheme. Frequent patterns become single IDs. Rare strings still factor into known pieces. Every modern LLM uses some flavor of it.

Walking through fertility and vocab size

Fertility is tokens per word (or per byte). English GPT-4-class fertility is often about 1.3 tokens/word. Some languages run 2–4× worse on English-centric tokenizers, so they pay more latency and cost per token for the same content.

If English averages 4 characters/token, a 10,000-character doc is about 2,500 tokens. Attention cost scales with tokens², so a language that averages 2 characters/token on the same tokenizer doubles sequence length and can 4× attention FLOPs for the same character content. That is the multilingual fairness issue hiding under flat per-token API pricing.

Providers price tokens because that is what the model consumes. Product folks thinking in words under-estimate non-English and code. Measure fertility on your traffic sample.

Llama 2 used about 32k. Llama 3 moved to about 128k partly for multilingual and code efficiency. Bigger vocab costs O(V d) parameters and makes the LM-head matmul heavier, but cuts sequence length. For multilingual products, larger vocabs often win on net FLOPs per meaning. For tiny on-device models, smaller vocabs can be preferable.

Raising V from 32k to 128k 4× the LM-head matmul (≈ 2 d V FLOPs per token). At large d the MLPs still dominate, but embedding memory grows too.

Common mistakes

• Counting words and calling them tokens. Bills and context windows are in tokenizer IDs.
• Mixing tokenizers across a fine-tune. Embedding row 50256 means something else now.
• Assuming English fertility holds for every language. Measure on real user text.

Check yourself

1. Why not feed raw bytes to a transformer with a long context? Answer: Attention is O(n²), so a byte sequence is much more expensive than a subword sequence for the same text.
2. English at 4 characters/token vs another language at 2 characters/token. What happens to attention FLOPs for the same character count? Answer: Sequence length doubles, so attention FLOPs can 4×.

Key takeaways

• Subwords balance sequence length against vocab size.
• Pure characters are too long. Pure words are too open.
• Fertility drives multilingual cost. Measure it on your traffic.
• Vocab size trades embedding parameters for compression. 128k-class vocabs favor multilingual and code efficiency.

Further learning

Next: byte-pair encoding.