Bi-encoders and cross-encoders

Retrieval needs two things at once: speed over millions of documents, and accuracy on the final few. The two main encoder designs trade these off. A bi-encoder embeds the query and the document separately, then compares the vectors. A cross-encoder packs the query and the document into one input and scores the pair directly.

Sentence embeddings

Averaging the token states from BERT is a weak sentence vector. Sentence-BERT fine-tunes an encoder with a contrastive or triplet loss, so pooled sentence vectors compare well with cosine similarity.

Bi-encoders are how many vector database systems and RAG pipelines retrieve. Embed documents offline, embed the query online, then run an approximate nearest-neighbor search.

Comparing the two

The usual pipeline is a bi-encoder to collect candidates, then a cross-encoder to reorder them.

Training a bi-encoder

The training idea is an in-batch softmax. Each query's positive document sits on the diagonal of a similarity grid, and every other document in the batch is a negative. This runnable sketch uses random vectors to show the mechanics:

import torch
import torch.nn.functional as F

torch.manual_seed(0)
B, D, temperature = 4, 8, 0.05
q = F.normalize(torch.randn(B, D), dim=1)            # 4 query embeddings
p = F.normalize(q + 0.1 * torch.randn(B, D), dim=1)  # matching docs, close to their queries
logits = (q @ p.T) / temperature                     # (4, 4) similarity grid
labels = torch.arange(B)                             # positives sit on the diagonal
loss = F.cross_entropy(logits, labels)
print(round(loss.item(), 4))

Output:

0.079

The loss is small because each query is closest to its own document. In a real model, q and p come from two encoder networks, and the vectors are learned.

Hard negatives replace random batch items with documents a weaker model retrieved, which are related but not labeled relevant. Filter these carefully. A near-duplicate that is labeled irrelevant poisons training.

Making vectors cheaper

Matryoshka Representation Learning trains an embedding so its first d′ dimensions already form a good embedding on their own. You can store and search the short vectors, and use the full length only when you need it.

Common mistakes

• Using a cross-encoder for the whole corpus. It must run once per pair, so it cannot scan millions of documents.
• Mixing normalized and unnormalized vectors. Cosine search assumes unit-length vectors. Normalize before indexing.
• Choosing hard negatives that are actually relevant. The model then learns to reject good matches.

Check yourself

1. Why can a bi-encoder search millions of documents but a cross-encoder cannot? Answer: document vectors are computed ahead of time, while a cross-encoder runs on every query-document pair.
2. What does Matryoshka training let you do? Answer: shorten the vector to fewer dimensions and keep most of the quality.

Key takeaways

• A bi-encoder embeds query and document separately, so it searches fast with an ANN index.
• A cross-encoder reads the pair together, so it is accurate but slow, and it reranks a short list.
• The common pipeline is bi-encoder for recall, cross-encoder for precision.
• Matryoshka embeddings can be truncated to save storage and search time.

Further learning

Next, see how the same contrastive idea links images and text in CLIP.