Retrieval
Advanced RAG
Hybrid search, reranking, query rewriting, contextual retrieval and measuring retrieval quality.
Basic RAG — chunk, embed, top-k, generate — gets you a demo. Production quality usually needs a sharper retrieval stack: keywords for exactness, dense vectors for paraphrase, rerankers for precision, query transforms for messy questions, and metrics so you know which change helped.
Advanced RAG is not one architecture you install. It is a toolbox ordered by evidence. Pick from it with metrics, not vibes.
Concept
Recall asks whether you surfaced the evidence at all. Precision asks whether the top results are useful. Early retrieval casts a wide net for recall. Reranking and a tighter prompt raise precision before the model spends tokens.
If you have not yet built a grounded prompt with citations and a syncing index, start with RAG. Here we assume that pipeline exists and is under eval.
Concepts in this section
Read them in order. Each one builds on the one before.
| Step | Concept | What it answers |
|---|---|---|
| 1 | Retrieval metrics | Did the right chunk appear, and how do you know? |
| 2 | Hybrid search | How do you catch both paraphrases and exact IDs? |
| 3 | Reranking | How do you shrink 50 hits to 5 useful ones? |
| 4 | Query transformation | What if the user's question is a bad search query? |
| 5 | Contextual retrieval | How do you stop chunks from losing their document? |
A strong default path: contextualize chunks at ingest (title and section at minimum), index dense plus BM25, optionally rewrite chat queries, retrieve 50 to 100, rerank to 5 to 10, generate with citations. Gate every change on recall@k, MRR and faithfulness.
When the toolbox is not enough
Agentic RAG lets the model decide whether to search, what to search and whether to search again. Use it when questions are multi-hop, tools already exist or single-shot retrieval fails evals. Cost: extra round-trips, harder debugging, a need for hard stops. Cap tool calls (for example 3), force an answer or escalation after two empty searches, and log every query string the agent issued.
Sometimes the right retrieval is not a text chunk. SQL or warehouse tools handle exact aggregates and permissions you already modeled. Knowledge graphs help with "how are X and Y connected?" Mixed systems vector-search for candidate entities, walk the graph, then pull prose chunks as evidence.
Trade-off
Graphs and SQL pay off when relationships or exactness dominate and you can maintain the structure. For mostly prose knowledge bases, invest in hybrid plus contextual chunks plus rerank before you build a graph.
| Dominant failure in the sample | Next lever |
|---|---|
| Gold doc missing from top 50 | Chunking, contextualization, hybrid, embedder |
| Gold in top 50, missing from top 5 | Reranker, diversity |
| Gold in the prompt, answer wrong | Prompt, model, fewer chunks |
| Multi-hop or compare questions | Decomposition or an agentic loop |
| Exact ID misses only | BM25 or a routing heuristic |
| Relational "how connected" | Graph or SQL tool |
Common mistakes
- Adding agent loops while gold is already in the top 50. Rerank first.
- A weighted score mix across BM25 and dense. Use reciprocal rank fusion, then route special query shapes if needed.
- Tuning the prompt when recall@k is weak. The evidence never arrived.
Key takeaways
- Hybrid BM25 plus dense plus RRF catches paraphrases and exact IDs.
- Rerank 50 to 100 down to 5 to 10 before you add exotic loops.
- Query transforms help messy questions. Measure the extra call.
- Situate chunks before you embed them.
- Track recall@k, MRR and faithfulness. Let the failure table pick the next lever.
Further learning
LangChain's RAG from scratch series continues into query translation.
ColBERT: Efficient and Effective Passage SearchRecent progress in Natural Language Understanding (NLU) is driving fast-paced advances in Information Retrieval (IR), largely owed to fine-tuning deep language models (LMs) for document ranking. While remarkably effec…arXiv.org
What We Learned from a Year of Building with LLMsA practical guide to building successful LLM products, covering the tactical, operational, and strategic.Applied LLMs
Pinecone Learning Center — retrieval topicsPinecone is the trusted AI knowledge company. Its AI knowledge platform—Database, Nexus, and Marketplace—powers accurate, fast, cost-effective AI for 10,000+ customers and 1M developers worldwide.pinecone.ioStart with retrieval metrics.
Related
Retrieval-augmented generation
The core RAG pipeline: ingest, chunk, embed, retrieve, generate with citations.
Search systems
How inverted indexes, analyzers and BM25 scoring work, and how to run Elasticsearch-style search at scale with sharding, indexing pipelines and hybrid vector retrieval.
Ranking and learning to rank
Second-stage ranking for search, feeds and ads. Learning to rank, ranking metrics, LambdaMART, CTR models, position bias and calibration. Start here for the map.