Lost in the middle

Attention can look at any earlier position. In practice, models are uneven. The "Lost in the Middle" research (Liu et al., 2023) found U-shaped use of long contexts: performance is often best when the crucial fact sits near the beginning or the end, and worse when it is buried in the middle of a long prompt.

Product consequences

• Put durable instructions and non-negotiable constraints early (and keep them stable for caching).
• Put the user question and the most relevant evidence near the end, close to where generation starts.
• Do not assume that stuffing 100 retrieved chunks helps. Ranking and truncation beat volume.
• Recency in chat history matters. The latest tool results usually deserve to stay; ancient turns may not.

A filing cabinet is a fair picture. People find the folder on top or the one they just put back. The middle drawer is where things go to be forgotten. The analogy breaks because the model can retrieve from the middle — it just does so less reliably as the pile grows.

Walking through a packing test

You have four retrieved chunks. Chunk B contains the refund window. You try two packs in the same 8K-token prompt:

Pack 1 (B in the middle):
  system (2,000) | A | B | C | D | question

Pack 2 (B at the end):
  system (2,000) | A | C | D | B | question

If pack 2 cites the 30-day window and pack 1 invents 14 days, the retriever already did its job. The loss was position. That is cheaper to fix than a new embedding model: rerank, keep top-k small, and place the best evidence last.

Softmax over thousands of tokens also dilutes. Critical facts compete with boilerplate. The fix is fewer, better tokens. A sermon in the system prompt that restates policy already sitting in chunk B just adds middle-of-context noise.

When quality drops after adding "more context," A/B the same evidence at the top versus the bottom of the prompt before you blame the model name.

Common mistakes

• Assuming more retrieved chunks are safer. Extra chunks bury the good one.
• Putting the question first and the evidence last. Generation starts after the last token; keep the question and the best evidence together at the end.
• Blaming embeddings when a position swap fixes the case.

Check yourself

1. Where should the user question and the best retrieved chunk sit? Near the end of the prompt, close to where generation starts.
2. You move the golden chunk next to the question and the answer becomes correct. What failed? Context packing, not retrieval. The chunk was already found.

Key takeaways

• Long-context quality is often U-shaped: edges beat the middle.
• Put stable instructions first. Put the question and the best evidence last.
• Rank and truncate. Volume is not a substitute for position.
• A/B the same fact at the top versus the bottom before you retune retrieval.

Further learning

Lost in the Middle (arXiv:2307.03172)While recent language models have the ability to take long contexts as input, relatively little is known about how well they use longer context. We analyze the performance of language models on two tasks that require …arXiv.org

Context engineering. Next, why output tokens feel slower and cost more: prefill, decode and the KV cache.