Bi-encoders and cross-encoders

Every embedding search makes a trade-off between speed and judgment. A bi-encoder reads the query and each document separately, which keeps search fast across a large corpus. A cross-encoder reads the query and one document together, which judges relevance more carefully but is far too slow to run on everything. Production systems use both, one after the other.

The big idea

Think of a library. A bi-encoder is a librarian who files every book once, by topic. When a question arrives, the librarian looks up the nearest shelves in seconds. A cross-encoder is a reader who takes the question and each candidate book and reads them side by side. The reader judges better, but can only check a few books.

The analogy breaks in one place. The librarian files each book using a compressed summary, a single vector. Details that the summary drops can't be recovered at search time. The cross-encoder sees the full text, so it can catch them.

Walking through the cost

Suppose a corpus holds 1,000,000 chunks.

• Bi-encoder, index time: embed each chunk once. This happens in advance.
• Bi-encoder, query time: embed the question once, then search the stored vectors for the nearest 100. That is 1 model pass plus a fast lookup.
• Cross-encoder, if used on everything: score the question against each chunk. That is 1,000,000 model passes for one user question.
• Cross-encoder, in the pipeline: score only the 100 candidates. That is 100 passes, and the top 8 go to the LLM.

The pipeline runs 100 passes where the naive version runs a million. That is the whole reason for the two-stage design.

Common mistakes

• Using a cross-encoder as the only retriever over a large corpus. It cannot score everything.
• Skipping the reranker when the right chunk is often in the top 50 but rarely in the top 5.
• Reading a bi-encoder score as a verdict. It ranks by similarity of meaning, which is not always the same as "answers the question."

Check yourself

1. Why can a bi-encoder's document vectors be computed ahead of time, but a cross-encoder's scores cannot? The bi-encoder encodes each document without seeing the query, so the vector stays valid for every query. A cross-encoder needs the query and document together, so its score exists only at query time.
2. The right chunk is in the top 50 but not the top 5. Which stage do you change? The reranker. Retrieval already found the chunk, so the problem is ordering.

Key takeaways

• Bi-encoders encode query and document separately, so document vectors are computed once.
• Cross-encoders read both together, so they judge relevance better and cost more per pair.
• The standard pattern is a bi-encoder to fetch about 100 candidates, then a cross-encoder to keep the best few.
• Use a cross-encoder only where the number of candidates is small.

Further learning

Next, learn how to choose the bi-encoder model itself in choosing an embedding model. For where the reranker fits in a full pipeline, see reranking.