Contrastive learning and InfoNCE

Contrastive learning trains embeddings without labels for each class. You only need to say which pairs belong together. Two views of the same image, a caption and its photo, or a query and the document it found should land near each other. Everything else in the batch should land away from them. The most common loss for this is InfoNCE.

The InfoNCE loss

For an anchor q, one positive k⁺, and a set of negatives kᵢ:

L = − log [ exp(sim(q, k⁺) / τ) / Σᵢ exp(sim(q, kᵢ) / τ) ]

sim is usually cosine similarity. τ is the temperature. A low τ makes the softmax peaky, so the hardest negatives dominate. A high τ makes everything look alike.

InfoNCE is just a softmax over the batch, where the correct answer is the positive's position.

Walking through one anchor

Take one anchor with a positive similarity of 0.9 and two negatives at 0.2 and 0.1. Use τ = 0.5:

divide by τ:   1.8,  0.4,  0.2
exponentials:  6.05, 1.49, 1.22     (sum = 8.76)

P(positive)  = 6.05 / 8.76 ≈ 0.69
loss         = −log 0.69 ≈ 0.37

If the positive rose to 0.99, the loss would fall toward zero. If a negative rose to 0.9, the loss would rise, because that negative now competes with the positive.

Where the pairs come from

• Positives: two augmented views of one image, a caption and its image, two paraphrases, or a user and an item they engaged with.
• Negatives: other items in the batch, items mined as hard near-misses, or sampled items.

A larger batch gives more negatives. That is why contrastive vision training favors big batches. The gain stops at some point.

Other contrastive losses

• Triplet loss asks that the anchor-positive distance plus a margin is still smaller than the anchor-negative distance.
• SigLIP replaces the softmax over the batch with pairwise sigmoid losses, which scales better.
• Supervised contrastive treats every same-class example in the batch as a positive.

The shared structure is the same: define a similarity, decide who should attract and who should repel, then optimize.

Common mistakes

• Tuning τ blindly. Start from the value in the paper you follow, then check the spread of your similarities.
• Ignoring duplicates in a batch. Two copies of one item become a false negative pair.
• Assuming hard negatives are always better. Near-duplicates labeled as negatives can poison training.

Check yourself

1. What does a lower τ do to the loss? Answer: it sharpens the softmax, so the hardest negatives dominate the gradient.
2. Why can a popular item hurt in-batch training? Answer: it may appear as a negative for an anchor it actually matches, which is a false negative.

Key takeaways

• Contrastive learning pulls positives together and pushes negatives apart, without per-class labels.
• InfoNCE is a softmax over the batch, with the positive as the correct answer.
• Temperature controls how hard the softmax is, and it is a real hyperparameter.
• Batch negatives are free, but duplicates and popular items create false negatives.

Further learning

Next, see how these embeddings are used to search, in bi-encoders and cross-encoders.