Grounded answers with citations

At query time, RAG has to do three things: find the right passages, put them in the prompt in a way the model will use, and return an answer the user can check. A grounded answer ties each claim to a passage you supplied. A citation is a pointer, such as a chunk ID or URL, that lets a person verify the claim.

The big idea

The query path has six steps:

1. Optionally rewrite the question. Skip this at first.
2. Embed the question with the same model used at ingestion.
3. Retrieve the top-k chunks. Start with k between 5 and 10.
4. Assemble the prompt: instructions, sources and the question.
5. Generate, telling the model to use only the sources and to say "I don't know" when they are silent.
6. Map the citations back to metadata, and return the answer with links.

A research assistant who answers from a folder of papers is a fair picture. Good assistants quote the page and say "not in the folder" when the answer is missing. The analogy stops at one point: the assistant can refuse, but code has to check that each cited number really exists.

Walking through it

Take the question "How do refunds work?" The steps look like this:

1. embed("How do refunds work?")
2. nearest chunks: refund-v3#2, refund-v3#3, faq#11, ... (top 5)
3. build prompt: rules + 5 sources + question
4. model answers: "Damaged items are refundable if the order shipped
   less than 14 days ago [1]."
5. check: [1] is one of the 5 sources given, so the citation is valid
6. return answer plus links to refund-v3#2

The prompt template keeps the rules and the sources apart:

Answer the question using only the sources below.
If the sources do not contain the answer, say you don't know.
Cite sources by id like [2].

<sources>
<source id="1" title="Refund policy" url="https://example.com/refunds">
...</source>
<source id="2" title="Billing FAQ" url="https://example.com/billing">
...</source>
</sources>

<question>{{QUESTION}}</question>

Three practical rules follow from this template:

• Cap the total source tokens. Five strong chunks beat twenty weak ones.
• Log the retrieved chunk IDs with every answer. You need them for debugging and evals.
• Check citations in code. If the model cites [4] but you sent three sources, reject the answer.

Common mistakes

• "Use the context if relevant." Without an explicit "I don't know" path, a retrieval miss turns into a confident guess.
• Trusting citations without a check. A model can cite a source number that does not exist.
• Dumping too many chunks. More text dilutes the useful evidence and costs more.
• Logging nothing. Without retrieved IDs, you cannot tell retrieval failures from generation failures.

Check yourself

• Why reject an answer that cites [4] when only three sources were sent? The citation cannot be verified, so the answer is not grounded in the evidence you provided.
• Why must the query embed use the same model as ingestion? Otherwise the question and the chunks sit in different spaces, and the nearest matches are meaningless.

Key takeaways

• The query path is embed, retrieve top-k, assemble, generate and cite.
• Put the rules first, then the sources, then the question.
• Require an "I don't know" path and make it easy to take.
• Check citations in code and log the chunk IDs with each answer.

Further learning

Next, see how to tell which stage broke when an answer is wrong, in failure modes.