Long context vs RAG

Both long context and RAG solve "the model needs facts it was not trained on." They are not mutually exclusive. The choice is about working-set size, cost and the shape of the question.

The big idea

Pasting a handbook is like carrying every file to a meeting. Retrieval is like pulling the three pages you need. The first wins when you must cross-reference a small set end to end. The second wins when the library is huge and each question touches a little of it.

Always-on long context (paste the whole thread forever) is simpler and works well under about 20k to 40k useful tokens. Beyond that, explicit memory plus RAG usually beats hoping the model attends correctly to turn 3 of 80.

Walking through the bill

Rough order of magnitude, using the source's 2025 to 2026 pricing band of about $3 per million input tokens:

Paste a 100k-token handbook every request:
  100,000 × $3 / 1,000,000 ≈ $0.30 / request

RAG: retrieve 3k tokens of chunks:
  3,000 × $3 / 1,000,000 ≈ $0.01 / request
  (plus a shared index, amortized)

At 10,000 requests per day, the paste approach is thousands of dollars per day before output tokens. RAG is about two orders of magnitude cheaper if retrieval quality is high enough.

When the task is "compare these two 30-page PDFs end to end," long context or staged summarization may still win.

Decision sketch:

• Corpus much larger than the window, questions sparse → RAG.
• Few documents, deep cross-reference → long context.
• Agents over codebases → hybrid: search tools plus selective file reads, not a full-repo paste.

Common mistakes

• Pasting the handbook on every turn because the window is large enough. You pay, and the needle can still sink.
• RAG for two short PDFs the user already uploaded. Retrieval can miss a paragraph that a full paste would have included.
• Treating the two as rivals. Many products retrieve a short list, then expand the chosen documents in full.

Check yourself

1. A 100k-token handbook at $3 / MTok, pasted every request. What is input cost per request? $0.30, before output tokens.
2. When does pasting beat RAG? When the working set is small and the task needs the whole documents, such as comparing two 30-page PDFs.

Key takeaways

• Long context wins on a small, deep working set. RAG wins on a large, sparse corpus.
• Cost scales with tokens in. Retrieval is cheaper when it is accurate.
• Always-on transcripts work until about 20k to 40k useful tokens. Then compact and retrieve.
• Coding agents should search and read files, not paste the repo.

Further learning

The assembly rules sit on context engineering. For the retrieval half, start at RAG.