What a token is

Language models do not read characters or words. They read tokens: integer IDs from a fixed vocabulary of text chunks. Every limit you hit — request size, monthly bill, time to finish a reply — is counted in tokens. If you budget in words, you will undershoot cost and silently cut context.

The big idea

Most modern models use byte-pair encoding (BPE) or a close cousin (SentencePiece, tiktoken-style BPE). The recipe is simple:

1. Start from bytes so any string can be represented.
2. Repeatedly merge the most frequent adjacent pairs in a training corpus into new vocabulary entries.
3. Stop at a target vocabulary size, often tens or hundreds of thousands of entries.

The split is model-specific. The same sentence can be 12 tokens on one API and 15 on another. A leading space is usually part of the token, so " world" is not the same token as "world".

A kitchen analogy: BPE invents shortcuts for frequent ingredient pairs, the way "peanut butter" becomes one jar. The analogy breaks because two kitchens invent different shortcuts, and you cannot guess one kitchen's count from the other's.

Walking through a split

"Tokenization is unbelievably useful"
→ illustrative BPE pieces:
   ["Token", "ization", " is", " unbelie", "vably", " useful"]

Six pieces, not four words. "unbelievably" split because that full word was rarer than the pieces in the training corpus.

For English prose on common Western tokenizers, these planning heuristics hold:

They fail in predictable ways. Code tokenizes denser or weirder than prose. Chinese, Korean, Arabic and many other scripts can cost 2–3× English for the same meaning. Long numbers fragment (123456789 may become several tokens). Whitespace and odd Unicode change the count.

    # Planning heuristic only — replace with the model's tokenizer before you bill
text = "Hello, tokens!"
approx = max(1, round(len(text) / 4))
print(len(text), "characters, about", approx, "tokens")

Output:

14 characters, about 4 tokens

The same string on another vocabulary can be 3 or 5. Measure both if you A/B two APIs.

Common mistakes

• Budgeting in words. A 2,000-character system prompt might be 480 tokens on one model and 560 on another.
• Assuming one tokenizer for every locale. An 80-token English support prompt can become 150+ tokens once localized.
• Treating the ~4-character rule as a bill. It is a planning heuristic, not a billing function.

Check yourself

1. Why can the same sentence cost more on API B than API A? Each model has its own vocabulary, so the same text splits into a different number of tokens.
2. A page of English prose is about 600 tokens. About how many words is that? About 450 words, using ~0.75 words per token.

Key takeaways

• Tokens, not words, set cost, latency and limits.
• BPE builds a model-specific vocabulary by merging frequent pairs.
• ~4 characters or ~0.75 English words per token is a planning heuristic only.
• Count with the model's own tokenizer before you ship.

Further learning

Paste text and see exact splits for OpenAI models:

Karpathy builds a GPT-style tokenizer from scratch:

Deeper notes on BPE and vocabularies: tokenization. Next, see how tokens fill a context window.