RAG failure modes and debugging

When a RAG answer is wrong, the fix depends on the stage that failed. A missed chunk needs better retrieval. A right chunk the model ignored needs a better prompt. Changing the wrong stage wastes weeks. This page gives you a taxonomy and a debugging order to find the stage quickly.

The big idea

Each failure leaves a different trail in the trace: the question, the retrieved chunk IDs and the answer. Read those three things before you change anything.

A medical triage table is a useful picture: symptoms point to the likely cause, and the first test is chosen by the symptom. The analogy stops at one point: in RAG, the same symptom ("wrong answer") can come from any row, so you need the trace to choose.

Walking through a real debugging session

Symptom: users say the bot invents refund windows. Follow the order:

1. Pull one failing trace. Read the question, the retrieved chunk IDs and the answer.
2. Check whether the gold policy chunk is in the top-k. If not, it is a retrieval miss. Look at chunking, hybrid search and k.
3. If it is there, read the chunk. If the chunk is a partial sentence that cuts off the window, the problem is chunking.
4. If the chunk is correct and the model ignored it, the prompt is weak, too many chunks compete, or the temperature is too high.
5. If the model cited [4] but only three sources were sent, the failure is a fake citation. Tighten the code that checks citations.

This five-minute order beats speculative changes. It also keeps you from reaching for agents before the basics work.

Measure each stage

Score retrieval and generation separately:

• Retrieval: recall at 5, on a labeled set of questions with known source chunks. Many internal document bots aim for 0.8 to 0.9 before they add polish.
• Generation: whether the answer is faithful to the retrieved sources, and whether it refuses when the sources are silent.

A perfect generator cannot recover from a miss. A perfect retriever still needs a faithful reader.

Common mistakes

• Tuning the prompt first. Most "model" bugs are retrieval or parsing bugs.
• Changing several things at once. You cannot tell which change helped. Change one stage at a time.
• Scoring only the final answer. It hides which stage failed.

Check yourself

• A user gets a wrong refund window, and the correct policy chunk is not in the top 5. Which stage failed? Retrieval. Fix chunking, search or k before the prompt.
• The right chunk is in the top 5, but the answer ignores it. What do you change first? The prompt and the number of competing chunks, then the temperature.

Key takeaways

• Read the trace first: question, chunk IDs, answer.
• Match the symptom to one failure type before you change anything.
• Measure retrieval recall and generation faithfulness separately.
• Fix stages in order: retrieval, then the prompt, then the citation checks.

Further learning

Return to the RAG hub for the pipeline overview, or read the wider evals page for how to build the labeled set this debugging needs.