Compaction and just-in-time retrieval

Sessions grow. Tool outputs grow faster. A git diff or a JSON API dump can be tens of thousands of tokens. Compaction shrinks older turns into a short narrative. Trimming cuts tool results on the way in. Just-in-time retrieval stores the full payload elsewhere and lets the model fetch it again if needed.

The big idea

When a session nears the limit, summarize older turns, keep recent turns verbatim and continue. Claude Code and similar agents do this continuously.

Good summaries keep the goal and constraints, decisions already made, open questions, identifiers (ticket ids, file paths) and tool outcomes that still matter. Bad summaries keep chatty acknowledgments and drop the one stack trace that mattered.

Tool outputs are the fastest way to explode context:

• return top-N rows, not whole tables
• truncate long strings with clear markers
• store full payloads in object storage and let the model re-fetch via a tool

Just-in-time beats "load everything upfront" when the agent can navigate (search, open file, query a database). Upfront loading wins when the working set is small and known.

After each tool result, run a cheap policy in code, not another model call, unless the result is huge and unstructured:

• Keep verbatim if the next step must cite exact strings (error messages, SQL, short diffs).
• Keep a hash plus path if the blob lives in object storage.
• Keep a summary if only the conclusion matters ("migration applied; 3 warnings").
• Drop acknowledgments, progress spinners and duplicated file reads.

Agents that re-read the same file five times should cache it by path and mtime in the runtime, or stop writing five copies into the prompt.

Walking through a budget

Max context:           128,000 tokens
Reserve for output:      4,000
Tools + system:          6,000
Memory + profile:        1,500
RAG chunks:              3,000
Remaining for history: 113,500  (theoretical)

After 40 tool calls:
  Raw history:          90,000
  After compaction:     12,000 summary + 8,000 recent turns = 20,000

You freed about 70k tokens of headroom and removed obsolete tool noise. That is usually a quality win, not only a cost win.

Memory worth persisting

Persist durable preferences and facts. Do not persist every intermediate thought. Let the user see and delete personal memories. For agents, a small set of memory files the model can read and write (with permissions) often beats an invisible vector swamp. Inspectability matters.

Common mistakes

• Dropping identifiers in the summary. The next turn cannot find the ticket or the file.
• Appending forever and hoping a 200k window will save you.
• Reloading the full raw transcript after compaction. Keep the full log in cold storage for debugging. Reload the working set.

Check yourself

1. After 40 tool calls the raw history is 90k tokens. What do you keep hot? A short "what we know" summary plus the last few turns, not all 90k.
2. When should you load a whole file into context up front? When the working set is small and known. Otherwise fetch just in time.

Key takeaways

• Compact old turns. Keep recent turns and the goal verbatim.
• Trim tool results on the way in. Store blobs and re-fetch.
• Persist durable memory on purpose. Do not treat the raw transcript as sacred.
• Full traces belong in cold storage. The model sees the hot working set.

Further learning

Effective context engineering for AI agents (Anthropic)Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.anthropic.com

When the working set is a few documents versus a whole corpus, see long context vs RAG.