Tokenizer pathologies

Get tokenization wrong and the failure looks like a dumb model. Letters were never separate symbols. Numbers split inconsistently. A trailing space changed the next-token distribution. This page is the catalog of those traps, plus a short debugging checklist.

The big idea

The model only sees IDs. If a task needs character-level structure that the tokenizer smashed, the model has to infer that structure from fragments. Sometimes it can. Often it cannot, and the demo goes viral.

Walking through the usual traps

• Arithmetic. Numbers split inconsistently (12345 might be one token or several). The model never sees clean place-value digits.
• Spelling / letter counting. Asking how many r's are in "strawberry" fails when the model sees straw + berry, not characters.
• Trailing spaces and invisible Unicode. A trailing space changes the next-token distribution. Zero-width chars and NFC versus NFD normalization change IDs.
• Non-English inflation. Models trained with English-heavy BPE under-segment English and over-fragment other scripts.
• Glitch tokens. Rare vocab entries that barely appeared in training can produce bizarre outputs when elicited.

Evaluating "reasoning" on digit tasks without controlling tokenization confounds the metric. Prefer character-level probes or carefully chosen numeric formats when the claim is about arithmetic competence.

Symptoms of a mismatch: a fine-tuned model ignores stop tokens; a chat model "speaks" raw special-token strings; eval scores crater after a library upgrade.

Checklist:

1. Print tokenizer.name_or_path and vocab size versus checkpoint config.
2. Encode a golden chat transcript. Assert the exact id list against a fixture committed in CI.
3. Verify add_bos / add_eos flags match training.
4. For byte-level BPE, confirm decoding round-trips on emoji and CJK samples.

Tokenizer bugs are silent. Loss still decreases while product behavior is wrong.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")
a = tok.encode("strawberry")
b = tok.encode("strawberry ")
print(tok.convert_ids_to_tokens(a))
print(tok.convert_ids_to_tokens(b))
print(a == b)

Typical output: the trailing space becomes its own leading-space marker on the next piece, so the ID lists differ and False prints. That one space is enough to change the next-token distribution.

In practice

• Always count tokens with the model's tokenizer before sizing prompts or bills.
• When fine-tuning, freeze the tokenizer unless you have a deliberate vocab expansion plan and an embedding init strategy.
• For multilingual UX, measure fertility on real user languages. It dominates cost more than people expect.
• Commit golden encode() fixtures next to any chat template.

Common mistakes

• Scoring arithmetic as a reasoning failure without checking tokenization. The digits may never have been atomic.
• Skipping round-trip tests on emoji and CJK. Byte-level decoding has sharp edges.
• Changing add_bos between train and eval. A single extra ID shifts every position.

Check yourself

1. Why might a model fail "how many r's in strawberry"? Answer: It likely sees pieces such as straw + berry, not the letters.
2. Loss fell after a tokenizer library upgrade, but chat quality collapsed. What do you check first? Answer: Vocab size, template, and a golden encode() fixture against committed IDs.

Key takeaways

• Many "dumb model" demos are tokenizer artifacts: spelling, arithmetic, trailing spaces, glitch tokens.
• Silent ID mismatches keep the loss falling while the product breaks.
• Count tokens with the model's tokenizer. Commit encode fixtures next to templates.
• Measure fertility on real languages before you talk about cost.

Further learning

Back to the map: tokenization.