Reranking

First-stage retrieval is fast and approximate. It is built to catch the right chunk somewhere in the top 50, not to put it first. A reranker then reads the query and each surviving passage together and reorders them. You keep 5 to 10 for the prompt.

The big idea

This is the same retrieve-then-rank shape as web ranking. Stage one casts a wide net. Stage two judges a shortlist carefully.

Latency cost is linear in candidates times passage length. Reranking 75 passages might add tens to a few hundred milliseconds, depending on model and hardware. Generation already costs seconds, so the extra wait is often worth it.

If eval shows the gold chunk is frequently in the top 50 but not the top 5, rerank before you invent agent loops. It is one of the highest-return upgrades in RAG.

A published pattern from the source: dense top-5 recall@5 = 0.62. Expanding to top 50 lifts gold coverage to 0.91, but 50 chunks in the prompt hurt faithfulness. A MiniLM cross-encoder down to top 8 restores prompt size and lifts end-to-end correctness more than switching the LLM.

ColBERT-style late interaction sits between bi-encoders and cross-encoders: richer than a single vector, cheaper than full cross-attention over the whole corpus. Hosted rerank APIs (Cohere, Voyage and others, as of 2025 to 2026) trade ops work for per-query cost. Self-hosted cross-encoders win at high QPS.

Diversity matters after rerank. If the top 8 chunks are near-duplicates of one paragraph, the model still lacks coverage. Maximal marginal relevance (MMR) or an embedding-diversity penalty helps multi-aspect questions.

Common mistakes

• Using a cross-encoder as the only retriever over a large corpus. It is too slow.
• Dumping the whole fused pool into the prompt instead of reranking. Faithfulness drops as noise rises.
• Skipping diversity. Eight copies of one paragraph look like a strong ranking and still miss the second fact.

Check yourself

1. Gold is usually in the top 50 but not the top 5. What is the next lever? A cross-encoder reranker, not a new agent loop.
2. Why retrieve 50 to 100 if the prompt only holds 8 chunks? The first stage optimizes recall. The reranker turns that recall into a small, precise prompt.

Key takeaways

• Retrieve 50 to 100 with a bi-encoder or hybrid search, then rerank to 5 to 10.
• A cross-encoder scores query and passage together and usually lifts precision a lot.
• If gold is in the wide pool but not the prompt, rerank first.
• After rerank, drop near-duplicates so the prompt covers more than one facet.

Further learning

ColBERT: Efficient and Effective Passage SearchRecent progress in Natural Language Understanding (NLU) is driving fast-paced advances in Information Retrieval (IR), largely owed to fine-tuning deep language models (LMs) for document ranking. While remarkably effec…arXiv.org

Messy questions need a better query before this stack runs. That is query transformation.