Embeddings and similarity

An embedding is a fixed-length list of numbers that stands in for the meaning of a piece of text. "How do I reset my password?" and "Forgot my login credentials" share almost no words, yet a good model places them near each other. That geometry is what makes semantic search, RAG, clustering and recommendations work.

You do not need the full training story from representation learning to ship. You need a picture of the space, a similarity rule that matches how the model was trained, a way to pick a model, and a habit of never mixing two models in one index.

Older search used sparse bags of term weights (TF-IDF or BM25) over tens of thousands of dimensions. That still wins on exact IDs. It fails when two texts mean the same thing with different words. Dense embeddings compress meaning into a few hundred learned dimensions, so paraphrases meet even when token overlap is zero.

The model is usually a transformer trained with a contrastive objective: pull related pairs together, push unrelated pairs apart. At inference you send text in and get a vector out. No generation, no sampling.

A real space has hundreds or thousands of dimensions. 2D is only for intuition. OpenAI's text-embedding-3-small returns 1,536 floats (you can request fewer). Sentence-Transformers' all-MiniLM-L6-v2 returns 384. Same kind of object; different spaces. Never mix them in one index.

Concepts in this section

Read them in order. Each one builds on the one before.

Common mistakes

• Mixing models in one index. Vectors from model A and model B are not in the same space. Changing models means re-embedding the whole corpus. Store embedding_model=text-embedding-3-small@v1 in metadata and never upsert mixed generations.
• Silent truncation. A 2,000-token chunk into a 512-token embedder indexes only the prefix. Align chunk size with max_seq_length.
• Query and document prefixes mixed. E5 and many instruction-tuned embedders expect query: on questions and passage: on documents. Mixing them collapses quality. Read the model card.
• Training-objective mismatch. A model that tops a sentence-similarity chart may lose on asymmetric retrieval. Prefer retrieval-oriented models for RAG.
• Domain mismatch. A web-text embedder underperforms on clinical notes or contracts. Fix the model before you tune chunking.
• Image and text compared by accident. They sit in one space only for CLIP-style models. A text-only embedder does not understand screenshots.

Key takeaways

• Embeddings turn text into points, so semantic similarity becomes nearest-neighbor search.
• Prefer normalized vectors plus cosine or dot product. Treat raw scores as model-specific.
• Choose models with your own eval. Watch dimension, max length, language and cost.
• Bi-encoders retrieve. Cross-encoders rerank. Use both when quality matters.
• Never mix models in one index. Re-embed and version on every model change.

Further learning

Computerphile's short intuition for word vectors is useful before you touch sentence embedders.

Jay Alammar walks through the Illustrated Word2vec picture.

MTEB LeaderboardEmbedding Leaderboardhuggingface.co Sentence-BERT (Reimers and Gurevych, 2019)BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019) has set a new state-of-the-art performance on sentence-pair regression tasks like semantic textual similarity (STS). However, it requires that both sentences a…arXiv.org OpenAI text embedding modelsLearn how to turn text into numbers, unlocking use cases like search, clustering, and more with OpenAI API embeddings.OpenAI Developers

Start with vector similarity.