Retrieval-augmented generation

Language models are frozen at training time. They do not know your private wiki, yesterday's incident doc or the SKU list that changed this morning. Retrieval-augmented generation (RAG) fixes that without fine-tuning. At question time, you retrieve relevant passages from your corpus and generate an answer with those passages in the prompt. The model becomes a reader, not the only source of truth. That is why answers stay current, can use private data and can be cited.

RAG is the workhorse behind internal copilots, support bots and document Q&A. Hybrid search, rerankers and query rewriting live in advanced RAG. Embeddings and nearest-neighbor indexes are covered in embeddings and vector databases.

Fine-tuning can teach style and format, but it is a poor cache for facts that change weekly. Stuffing every document into every prompt blows up the context window and the bill. RAG splits the job: a retriever finds a small set of evidence, and a generator reads it.

Why RAG, and when not

Reach for RAG when knowledge is private, fresh, large or access-controlled. Retrieval can filter by user; a model's weights cannot.

If the whole corpus fits comfortably in the context window, putting all of it in the prompt can be simpler and more accurate. See context engineering for that trade-off. Many products use both: retrieve a short list, then expand the chosen documents in full.

Concepts in this section

Read them in order. Each one builds on the one before.

The two pipelines

Ingestion runs offline, in the background. Query time runs on every question. Most RAG bugs come from the first pipeline, but they show up in the second one as bad answers.

Common mistakes

• Tuning the prompt while parsing is broken. Multi-column PDFs and table text can scramble facts before retrieval ever runs. Check the parser first.
• A sync path that inserts but never deletes. Stale content then gets cited with confidence.
• Access control only in the prompt. Enforce permissions in the retrieval query, not as an instruction the model may ignore.
• No "I don't know" path. Without one, a retrieval miss becomes a confident hallucination.
• Measuring only the final answer. Measure retrieval and generation separately, or you cannot tell which one failed.

Key takeaways

• RAG retrieves evidence into the prompt, so answers use private, fresh and large knowledge, with citations.
• Ingestion (parse, chunk, embed, index, sync) decides what can be found.
• Chunk on structure, size for the embedder and overlap the boundaries.
• Prompt for grounding and abstention, then check the citations in code.
• Start simple. Add hybrid search or reranking only when evals show a specific miss.

Further learning

Lewis and colleagues introduced this pattern for knowledge-intensive NLP in 2020.

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and prec…arXiv.org

Eugene Yan's patterns post covers how RAG fits into a wider system.

Patterns for Building LLM-based Systems & Products (Eugene Yan)Evals, RAG, fine-tuning, caching, guardrails, defensive UX, and collecting user feedback.eugeneyan.com

Start with the first concept page: ingestion pipeline.