Retrieval metrics

A wrong RAG answer has two common causes. The right passage never entered the prompt, or the model read the passage and still wrote something else. Retrieval metrics measure the first cause. Without them you are tuning in the dark.

The big idea

Build a set of real questions with known relevant chunk ids. Fifty to a few hundred graded queries beat thousands of unlabeled ones.

Walking through three queries

Take k = 5.

Q1 relevant at rank 2 → recall@5 = 1,  RR = 1/2 = 0.50
Q2 relevant at rank 8 → recall@5 = 0,  RR = 0
Q3 relevant at ranks 1 and 4 → recall@5 = 1,  RR = 1/1 = 1.00

Macro-average recall@5 = (1 + 0 + 1) / 3 = 0.67
MRR                   = (0.50 + 0 + 1.00) / 3 = 0.50

Q2's gold chunk sat at rank 8. Expanding retrieval to top 50 would catch it. Dumping those 50 chunks into the prompt would still hurt the answer. That is why you measure retrieval and generation separately.

Once retrieval is healthy, score answers with faithfulness (claims supported by retrieved text), answer relevance, and citation accuracy (cited ids exist and support the sentence). Use code checks, entailment models or an LLM judge.

Building a labeled set

1. Sample about 100 real production questions or support tickets.
2. For each, have a human paste the URL or section that should answer it, or mark it unanswerable.
3. Map those anchors to chunk ids after your chunker runs. Chunk ids change when you rechunk, so store doc_id plus a character offset as well.
4. Freeze a v1 set. Later add slices for hard intents: IDs, multi-hop, other languages.

Do not wait for 5,000 graded queries. A sharp 100 beats a noisy 5,000.

Track recall@k and MRR on every change that touches chunking, embeddings, hybrid fusion or rerankers.

Common mistakes

• Scoring only the final answer. You cannot tell whether search or generation failed.
• Rechunking without remapping labels. Yesterday's gold ids point at the wrong text.
• Waiting for a huge labeled set. A hundred real questions is enough to start.

Check yourself

1. Gold sits at rank 8 and you report recall@5. What is that query's recall@5? 0. The relevant chunk is outside the top 5.
2. Recall@50 is 0.91 but answers are still wrong. What do you fix first? The prompt, the model or the number of chunks, not the embedder. Retrieval already found the evidence.

Key takeaways

• Recall@k, MRR and nDCG measure whether the right chunk was found.
• A labeled set of 50 to a few hundred real questions is enough to start.
• Measure retrieval and generation separately.
• Store source anchors, not only chunk ids, so rechunking does not orphan labels.

Further learning

Use these numbers when you add hybrid search or a reranker.