Hybrid search

Dense embeddings shine on paraphrases: "cancel my plan" sits near "terminate subscription." They stumble on exact identifiers: error codes (ECONNRESET), SKUs (SKU-4491), emails, API field names and rare proper nouns. Sparse lexical search, classically BM25, does the opposite. Hybrid search runs both, then fuses the ranks.

The big idea

Think of two clerks. One knows synonyms. The other has a perfect filing system for codes and names. You ask both, then merge their piles.

The picture breaks if the two indexes use different chunk ids. Fusion needs a shared id so the same passage can get credit from both lists.

Reciprocal rank fusion (RRF) combines ranked lists without calibrating incompatible scores:

score(doc) = Σ over lists of  1 / (k + rank)

Conventionally k = 60. Rank is 1-based. A document at rank 1 in both lists beats a document at rank 1 in only one.

def reciprocal_rank_fusion(lists: list[list[str]], k: int = 60) -> list[str]:
    scores: dict[str, float] = {}
    for ranking in lists:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return [d for d, _ in sorted(scores.items(), key=lambda x: -x[1])]


bm25 = ["orch-210", "orch-overview", "timeouts"]
dense = ["timeouts", "orch-overview", "retries"]
print(reciprocal_rank_fusion([bm25, dense])[:3])

Output:

['orch-overview', 'orch-210', 'timeouts']

orch-overview wins because it ranks 2 in both lists. orch-210 is first in BM25 only. That is the query what does error ORCH-210 mean? in miniature: dense-only may retrieve generic orchestration docs; BM25 hits the page titled ORCH-210; RRF keeps both.

People like weighted mixes such as 0.7 * dense + 0.3 * bm25. Those weights are brittle across query types and need calibrated scores. RRF ignores score magnitudes and usually gets you most of the benefit with one constant. If BM25 should dominate for code-like queries, route by a cheap heuristic (hyphens, digits, CamelCase) rather than a global weight.

OpenSearch, Elasticsearch, Weaviate and Qdrant can run both legs in one system. Otherwise fuse in the application. Index both channels with the same chunk ids. If BM25 indexes whole documents and dense indexes chunks, map hits to the same grain before fusion.

Common mistakes

• Fusing different grains. Whole-document BM25 plus chunk-level dense needs a mapping step first.
• A global score mix. Weights that win on FAQs lose on error codes.
• Skipping hybrid because the embedder is "strong." Strength on paraphrases does not fix a missing SKU.

Check yourself

1. Why does RRF use rank instead of raw scores? BM25 and cosine live on different scales. Rank is comparable; the scores are not.
2. A query is SKU-4491. Which leg is more likely to hit the right page? BM25, because the exact token is rare and distinctive.

Key takeaways

• Dense search finds paraphrases. BM25 finds exact tokens.
• Fuse with RRF (k = 60) instead of fragile score weights.
• Give both indexes the same chunk ids.
• Add hybrid as soon as IDs, code or proper nouns show up in real queries.

Further learning

The fused pool is still 50 to 100 hits. Next, rerank it down to 5 to 10.