Context ordering

The tokens in a request are not a bag. Order changes what the model uses, and it changes what a provider can cache. A robust default for agentic apps is a stable prefix, then memory, then retrieved docs, then recent history, then the current task.

The big idea

The system prompt is the constitution. Retrieval and memory are the case files. The latest user turn is the motion before the court. A short constitution plus the right exhibits beats a constitution restated forty ways plus a warehouse of unsorted paper.

Why the stable prefix comes first: if you put a changing timestamp or user id at byte 0, you bust the cache every request and pay full prefill cost. Put volatile content late. Design the first N thousand tokens to be identical across users when you can.

Walking through a support turn

A 128k-window model, 2,000 tokens reserved for the reply. Remaining input budget: 126k. Allocate on purpose:

System + safety + escalation rules:     1,800 tokens  (stable, cached)
Tool schemas (8 tools):                 2,200 tokens  (stable, cached)
Customer profile memory:                  400 tokens
RAG: top 5 help-center chunks:          2,500 tokens
Open ticket fields + order JSON:          900 tokens
Compacted earlier conversation:         3,000 tokens
Last 6 turns verbatim:                  2,200 tokens
Latest user message:                      180 tokens
-----------------------------------------------
Total:                                 13,180 tokens

You are nowhere near the ceiling. The failure is not "hit 128k." It is letting tool results grow until RAG and memory sink into the mushy middle. Cap each tool result (for example 1,500 tokens). If list_orders returns 40 rows, return the newest five and a next_page tool.

Moving request_id=... from the system prompt into a trailing debug footer dropped input cost about 40% on a chat product that already qualified for prompt caching. Same model, same answers, cheaper prefills.

Watch p95 input tokens per turn and the cache hit rate on the stable prefix. If p95 climbs and cache hits fall, someone put per-request entropy at the top (timestamps, request ids, randomized few-shots).

Common mistakes

• A timestamp or request id in the system prompt. The cache never hits.
• Randomized few-shots in the prefix. They look clever and destroy prefix reuse.
• Retrieved docs after the user question. The live task is no longer at the end.

Check yourself

1. Where should request_id go? Late, in a footer. Never at the start of a cached prefix.
2. Why keep tool schemas in the prefix? They are stable, high-leverage and cacheable. Write them like API docs.

Key takeaways

• Order is prefix, memory, retrieval, compacted history, current task.
• Stable tokens first help both attention and prompt caches.
• Budget the window with real token numbers, then cap tool results so the middle stays thin.
• Instrument p95 tokens per turn and prefix cache hit rate.

Further learning

See caching for LLM apps for break-even math. Next, compact what does not fit.