Context windows

The context window is the maximum number of tokens the model can consider in one call. Hit it and the request is truncated or rejected. Miss it by stuffing everything in, and you still pay for every extra token and often lose the fact buried in the middle.

The rule

context_window ≥ input_tokens + output_tokens

Providers also enforce a separate max output cap per response, for example 4K or 8K tokens, even when the window is 128K+. Hitting max output stops generation mid-thought with a max_tokens stop reason. See LLM APIs.

What fills the input side:

• System prompt and developer instructions
• Tool and function definitions
• Conversation history (every prior turn, resent in full on typical chat APIs — the API is usually stateless)
• Retrieved documents and other RAG context
• Images, files and tool results

A long window is not free memory. Prefill cost and latency grow with input length. Attention can get "lost in the middle." You pay for every input token on every call unless prompt caching applies. Bigger is a capability, not a strategy. Context engineering is how you spend the budget well.

Walking through a 128K support bot

System + tools          2,500
Last 8 turns history    6,000
Retrieved docs (4×1.5K) 6,000
User question             500
Reserved for answer     2,000
─────────────────────────────
Total budgeted         17,000   (comfortable headroom)

17,000 tokens in a 128K window looks wasteful. It is not. The leftover is insurance against a long answer, a fat tool result and a history spike. If retrieved docs alone want 80K tokens, you do not have a "use the long context" problem. You have a ranking problem. Stuffing the window raises cost, latency and middle-of-context miss rates together.

When debugging "we randomly hit context length errors," log the per-slice breakdown (system / tools / history / rag / user) for failing requests. One slice is usually the villain — often unbounded history or a retriever set to top_k=20 with huge chunks.

Common mistakes

• Forgetting to reserve output. A 128K window filled with 128K of input has no room to answer.
• Treating a long window as a strategy. More tokens raise cost and bury the useful fact.
• Assuming the API remembers the chat. Typical chat APIs are stateless. You resend history every turn.

Check yourself

1. A model has a 128K window and an 8K max output. You send 125K input tokens. What happens? The request is over the window (125K + up to 8K), so it is rejected or truncated. You also left almost no room to answer.
2. Retrieved docs want 80K tokens in that same 128K bot. What is the real problem? Retrieval ranking. Stuffing 80K docs raises cost and miss rates; keep a few good chunks.

Key takeaways

• The window covers input + output. Always reserve space for the answer.
• Instructions, tools, history, retrieval and files all share one budget.
• Long windows cost latency and money. They are not a substitute for ranking.
• Log the per-slice breakdown when a request overflows.

Further learning

How to spend the budget well: context engineering. Next, what those tokens cost in token pricing.