Guide

Deep learning

word2vec

word2vec trains one vector per word so that words used in similar contexts end up close together. Skip-gram with negative sampling is the version to know.

97 of 187 · 2 min read

Words used in similar contexts tend to mean similar things. word2vec (Mikolov et al., 2013) turns that observation into training. It gives each word a vector, and it adjusts the vectors so that words which appear near each other end up close together in the space.

Concept

In skip-gram, the model takes a center word and predicts the words within a small window around it. The window is the context. CBOW does the reverse: it predicts the center word from its context.

Negative sampling

A full softmax over a large vocabulary is expensive, because every step normalizes over every word. Negative sampling replaces it with a yes-or-no question. Is this word a true neighbor, or a random draw? For each true pair, sample a few random words as negatives:

score = σ(v_center · v_context)     push toward 1 for a true pair
      vs σ(v_center · v_negative)   push toward 0 for a random word

Walking through one pair

Take the centre word "cat" with a true context word "sat", and one random negative word, "river".

Suppose the current vectors give the true pair a dot product of 0.8, and the negative pair −0.5.

positive loss = −log σ(0.8)   = −log(0.69)  ≈ 0.37
negative loss = −log σ(0.5)   = −log(0.62)  ≈ 0.47

Training nudges the vectors to reduce both losses. The true pair's dot product rises, and the random pair's falls. Each pass moves the vectors a little closer to the real pattern of neighbors.

What it gives you

The famous demo is king − man + woman ≈ queen. Analogies like that work sometimes. They are a side effect of the geometry, not a reasoning engine.

What works reliably:

  • Nearest neighbors for synonyms and related terms.
  • Good features for a linear classifier on top.

Limits

  • One vector per word type. "Bank" as in a river bank and "bank" as in a loan get the same vector.
  • A fixed window. Only nearby words shape the vector.
  • Superseded for context. Contextual encoders, such as ELMo, BERT and GPT, give a vector per occurrence, so "bank" near "river" differs from "bank" near "loan."

Common mistakes

  • Treating analogies as reasoning. They are a geometry effect that works only sometimes.
  • Reading nearest neighbors for polysemous words. The neighbors mix the senses.
  • Using word2vec where context matters. For a sentence-level task, a contextual encoder is the better fit.

Check yourself

  1. Why does negative sampling beat a full softmax? Answer: it scores a few random words per pair, not the whole vocabulary, so each step is cheap.
  2. What does word2vec give "bank" in a sentence about a river? Answer: the same single vector it gives "bank" in a sentence about a loan.

Key takeaways

  • word2vec learns one vector per word from the words around it.
  • Negative sampling turns the expensive softmax into a few yes-or-no pair checks.
  • Analogies such as king − man + woman ≈ queen are a side effect, not a guarantee.
  • One vector per word type cannot separate word senses, which contextual encoders fix.

Further learning

Next, see the loss that generalizes this idea to any pair of things in contrastive learning.

Embedding lookup tables

An embedding layer is a learned table with one row per ID. Looking up a row turns a discrete ID into a vector that training can shape.

Contrastive learning and InfoNCE

Contrastive learning pulls positive pairs together and pushes negatives apart. InfoNCE is the loss that does it with a softmax over a batch, and temperature controls how sharp it is.

Embeddings and similarity

Turning text into vectors, measuring similarity, and choosing an embedding model.