Tokenization quirks

Several "model bugs" are tokenizer bugs. The network never sees letters or digits as you do. It sees the pieces the vocabulary chose. When evals fail on string games or arithmetic, a bigger model is usually the wrong fix.

Walking through a letter-count trap

Ask "how many r's in strawberry?" The model may see something like "straw" + "berry", not nine letters. It was not trained to count characters inside a merge. The same thing happens with "replace the 3rd character": the edit fights the tokenizer. Ask for a whole-word rewrite instead, or do the edit in ordinary code.

Long integers fragment in unpredictable ways. 123456789 may become several tokens, which is one reason models stumble at digit-level arithmetic. Hand the sum to a calculator tool.

    # Sketch of why character jobs should not go to the model
word = "strawberry"
print(word.count("r"))
print(sum(int(c) for c in "12345"))

Output:

3
15

Product specs that say "validate exactly 12 alphanumeric characters" should run that check in code. Specs that say "add these account numbers" should call a calculator. Reserve the model for language and judgment.

Languages, code and payloads

A customer-support prompt that is 80 tokens in English may be 150+ tokens in a script that tokenizes poorly. If you ship the same feature worldwide, measure counts per locale before you promise uniform unit economics.

Code-heavy tenants blow past the 0.75-words rule. Identifiers, operators and indentation all become tokens. JSON burns counts on brackets. Log real averages per feature; do not apply the English-prose heuristic globally.

When you compare two APIs on the "same" prompt, compare token counts, not character counts. A 2,000-character system prompt might be 480 tokens on one vocabulary and 560 on another. That moves both cost and how much room remains for retrieved docs. Rebuild the context budget per tokenizer.

Keep a small fixture file of representative prompts — an English ticket, a German ticket, a Python diff, a JSON blob — and record counts whenever you adopt a new model family. That fixture is cheap insurance against a surprise bill after a "drop-in" migration.

Trailing whitespace in prompts can change the next-token distribution. Keep few-shot examples tidy.

Images and pages

Vision and document models turn pixels and pages into tokens with vendor-specific formulas (tile grids, tokens-per-page, low/high detail). A single high-resolution image can cost thousands of tokens. Budget it like a document, not like "one attachment."

PDF pipelines that rasterize every page to an image explode cost versus extracting text where a text layer exists. Mixing high-detail images with long retrieved context is how teams accidentally spend a dollar on one customer message.

Common mistakes

• Asking the model to count letters or digits. Use code or a calculator tool.
• Applying the English heuristic to code or CJK text. Measure those payloads.
• Comparing character counts across providers. Compare tokens, then rebuild the budget.

Check yourself

1. Why is "how many r's in strawberry?" hard for a subword model? It sees merges such as "straw" + "berry", not nine letters.
2. A localized prompt jumps from 80 to 150 tokens. What do you do before promising cost? Measure token counts per locale and redo the unit-economics model.

Key takeaways

• Tokenization quirks look like reasoning failures. They are not.
• Use tools or ordinary code for character edits, letter counts and long arithmetic.
• Measure tokens per locale, per code-heavy feature and per image mode.
• Keep a fixture of representative prompts when you change model families.

Further learning

See the OpenAI Tokenizer and tokenization. Next, how these counts fill a context window.