Budgeting a context window

Treat the window like RAM for a single request. Allocate each slice on purpose. If you do not, history and retrieved docs will fill it, the answer will get cut, and the bill will grow every turn.

The slices

A 200-token tool definition × 8 tools is 1,600 tokens on every request before the user message. Descriptions that ramble cost real money at scale. Descriptions that omit "when not to use" cost wrong calls. Budget both.

History is the silent killer

Chat UIs that resend the full transcript every turn create quadratic cost in conversation length. Turn 1 costs 500 tokens. By turn 30 you may resend 15,000 tokens of history on every call — mostly old tool dumps.

Mitigations that work:

1. Hard cap — keep last N turns (for example 10) plus a rolling summary.
2. Drop tool payloads — store "called get_invoice, status=paid" instead of the full JSON after the model has used it.
3. Summarize asynchronously — a cheap model compresses older turns into a 300-token memo.
4. Branch threads — do not make every side question inherit a 40-turn coding session.

Walking through a leak

A support bot starts each ticket at 2,500 tokens of system+tools. After 12 turns of raw tool JSON, history is 14,000 tokens. Each new turn now costs:

system + tools     2,500
history           14,000
user + rag         1,000
reserved output    2,000
────────────────────────
per-turn input    17,500   (was 3,500 at turn 1)

At 80 output tokens and $3 / $15 per M, turn 1 was cheap. Turn 12 is five times the input. Cap history at 8 turns, replace old tool JSON with one-line summaries, and the same ticket stays near 8,000 input tokens.

Before shipping, log input_tokens, output_tokens, cache_read_tokens and dollars per request class. Alert when average input drifts up — prompt regressions, history leaks, retrieval dumping. In design reviews, make someone say the monthly token forecast out loud with arithmetic. "We'll just use the 200K window" is not a forecast.

Common mistakes

• No history cap. Cost grows with the square of conversation length.
• Leaving full tool JSON in later turns. The model already used it; send a one-line result.
• Putting a unique timestamp at the start of the system prompt. That kills prefix caching.

Check yourself

1. Why does cost grow faster than "one more turn"? Typical APIs resend the whole history, so turn N pays for turns 1 through N.
2. After a tool returns a 40 KB invoice JSON, what should later turns see? A one-line summary such as "get_invoice: status=paid", not the raw payload.

Key takeaways

• Allocate the window: instructions, tools, history, retrieval, output.
• Cap history and drop spent tool payloads, or cost grows every turn.
• Keep raw transcripts in your database; send a budgeted view to the model.
• Log per-slice token counts so a leak shows up as a number.

Further learning

Context engineering is the broader practice. For the bill, see token pricing.