Prompting and context
Context engineering
Deciding what goes into the context window: ordering, compaction, memory and long context vs retrieval.
Prompt engineering shapes the instructions. Context engineering shapes the entire token budget the model sees on each call: system rules, tool schemas, retrieved docs, conversation history, memories, scratchpads and the latest user turn. In a support agent, coding agent or research assistant, the hard problem is rarely a missing adjective in the system prompt. It is packing the right high-signal tokens into a finite window, turn after turn.
People hear "prompt" and think of a short task description. Production systems assemble a context window at runtime. Curating that assembly is the job. Anthropic's framing: find the smallest set of high-signal tokens that maximizes the chance of the desired outcome.
Concept
The context window is the maximum tokens per request (input plus reserved output, depending on the API). Every token competes for limited attention. More tokens are not free: you pay latency, money and often accuracy.
| Prompt engineering | Context engineering | |
|---|---|---|
| Focus | Wording of instructions and examples | Which tokens enter, in what order, at each step |
| Cadence | Relatively static artifacts | Dynamic per turn and per tool result |
| Failure mode | Ambiguous goals | Noise, staleness, lost-in-the-middle, cache misses |
You still need clear prompts. Perfect retrieval will not save a contradictory task description. You need both layers. Production outages teach you which one is on fire.
Concepts in this section
Read them in order. Each one builds on the one before.
| Step | Concept | What it answers |
|---|---|---|
| 1 | Context rot and lost-in-the-middle | Why can more tokens make answers worse? |
| 2 | Context ordering | What goes first, and what must stay last? |
| 3 | Compaction and just-in-time retrieval | How do you keep a long session inside the window? |
| 4 | Long context vs RAG | When do you paste, and when do you retrieve? |
An agent turn usually includes a tight system prompt, tool definitions written like API docs, only the examples that still teach, an optional plan that you rewrite rather than append forever, and trimmed observations. Stopping conditions belong in context too: max tool calls, "ask the user when uncertain," and a definition of done. Without them, agents wander until the window fills.
Common mistakes
- Rewriting the system prompt while each turn still injects a 20k scrape "just in case."
- Per-request entropy at byte 0 (timestamps, request ids) that busts the prefix cache.
- Treating "the model forgot" as a model bug before you dump the exact payload.
Key takeaways
- Context engineering manages the whole token budget. Prompting is one slice.
- Attention is finite. More tokens can lower quality.
- Stable, cacheable prefix first. Live question last.
- Compact history, trim tool results and fetch just in time when the agent can navigate.
- Choose long context vs RAG with working-set size, cost and latency. Hybrid is common.
Further learning
Effective context engineering for AI agents (Anthropic)Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.anthropic.com
Lost in the Middle: How Language Models Use Long ContextsWhile recent language models have the ability to take long contexts as input, relatively little is known about how well they use longer context. We analyze the performance of language models on two tasks that require …arXiv.org
Token math pairs with tokens and context windows.
Start with context rot and lost-in-the-middle.
Related
Tokens and context windows
How text becomes tokens, what that means for cost and limits, and how to budget a context window.
Retrieval-augmented generation
The core RAG pipeline: ingest, chunk, embed, retrieve, generate with citations.
Caching for LLM apps
Prompt caching, exact-match and semantic caches: where each fits, and how to avoid serving the wrong answer.