Tokens and context windows

Language models do not read characters or words. They read tokens: integer IDs from a fixed vocabulary. Every limit you hit — max request size, monthly bill, time to finish a reply — is denominated in tokens. If you only think in words, you will mis-budget cost and silently truncate context.

A token is a chunk of text the model consumes and produces. Common English words are often one token; rarer words split. The split is model-specific, so the same sentence can be 12 tokens on one API and 15 on another. Count with the model's tokenizer, not with a word estimate.

Why this matters

Tokens sit under three product decisions. First, what fits: the context window must cover input plus output. Second, what it costs: vendors bill USD per million tokens, and output is often 3–5× input. Third, what breaks: letter-counting, long numbers and some languages fail because the model sees pieces, not characters.

Long windows are not free. Prefill cost grows with input. Attention can lose a fact in the middle. You pay for every input token on every call unless a stable prefix is cached. Bigger is a capability. Context engineering is how you spend the budget.

Concepts in this section

Read them in order. Each one builds on the one before.

For tokenizer internals see tokenization. For spending the window well see context engineering.

A billing shape to remember

cost ≈ (in_tokens × in_price + out_tokens × out_price
        + cached_tokens × cached_price) / 1_000_000

A support-bot window that stays near 17,000 tokens (2,500 system+tools, 6,000 history, 6,000 retrieved docs, 500 question, 2,000 reserved output) is cheaper and more reliable than stuffing 80K retrieved chunks. If retrieval wants 80K, fix ranking.

Before shipping a feature, log input_tokens, output_tokens, cache_read_tokens and dollars per request class. Alert when average input drifts up. When a request overflows, log the per-slice breakdown. One slice is usually the villain.

Common mistakes

• Gating production on word-count estimates. Count with the same tokenizer the model uses.
• Forgetting that the window includes output. A full input leaves no room to answer.
• Resending every tool dump. Chat APIs are usually stateless. History that grows without a cap makes cost quadratic in conversation length.
• Putting a request ID or timestamp at the front of a stable prompt. That destroys prefix caching.

Key takeaways

• Tokens — not words — set cost, latency and limits.
• The context window covers input plus output. Reserve space for the answer.
• Output tokens and uncached input dominate bills. Caching rewards a stable prefix.
• Tokenization quirks explain letter-counting and number bugs. Use tools instead of hoping.
• Budget the window like memory: instructions, tools, history, retrieval, output.

Further learning

Karpathy builds a GPT-style tokenizer from scratch:

Tokenization

Start with what a token is.