Embeddings and representation learning

Raw inputs, such as token IDs, pixels and user IDs, are poor for similarity or prediction until you map them into a space where distance reflects meaning. Representation learning is the art of learning that map. Its outputs are embeddings, and they underpin search, recommendations, RAG and multimodal models.

This section is the ML view: how embeddings are trained. For the productized side of vector search, see AI embeddings.

Concepts in this section

Read them in order. Each step builds on the one before.

Step	Concept	What it answers
1	Embedding lookup tables	How does an ID become a learned vector?
2	word2vec	How do you learn word vectors from context?
3	Contrastive learning and InfoNCE	How do positives and negatives shape the geometry?
4	Bi-encoders and cross-encoders	How do you search fast and then rerank accurately?
5	CLIP and zero-shot classification	How do images and text share one space?

Common mistakes

Trusting nearest-neighbor demos. A few hand-picked neighbors look right, and measured retrieval can still be poor. Measure recall at k, MRR or nDCG on labeled queries held out by time or product.
Mixing vectors from two model versions. An index built from model A and queried with model B returns noise. Version the index with the model.
Ignoring freshness. If documents change hourly but you re-embed weekly, recall drops, and it looks like a model problem.
Assuming embeddings replace keyword search. Exact tokens such as product codes or names are often better served by lexical search. Hybrid retrieval combines BM25 for exact matches with embeddings for paraphrase, then fuses the scores or uses reciprocal rank fusion before a cross-encoder rerank. Log which source won each candidate, so you know where to invest.

Failure	Symptom	Mitigation
False negatives in batch	Popular documents hurt	De-duplicate the batch, use a memory bank, mine negatives carefully
Train and serve tokenizer skew	Sudden recall drop	Freeze the tokenizer and add integration tests
Mixed model versions in one index	Garbage neighbors	Alias indexes by model ID
Shortcut learning	Embeddings encode only length or language	Hard negatives and balanced data
Polysemy in static vectors	One vector per word	Use contextual encoders

Key takeaways

An embedding layer is a learned lookup. The geometry comes from the loss.
word2vec learned vectors from context. Contextual transformer encoders replaced one-vector-per-word.
Contrastive training needs positives, negatives and a temperature. InfoNCE is the common loss.
Bi-encoders retrieve quickly. Cross-encoders rerank a short list accurately.
CLIP aligns images and text, which enables zero-shot classification.
Normalize vectors if you use cosine, version the embedding space, and evaluate with recall and nDCG.