CLIP and zero-shot classification

CLIP, from OpenAI in 2021, trains two encoders together: one for images and one for text. For each batch of image-caption pairs, a matching image and caption should score high, and mismatched pairs should score low. That is contrastive learning, applied across two kinds of data.

The two towers differ, because one reads pixels and the other reads text. Both produce vectors in one shared space, which is what makes them comparable.

Zero-shot classification

Once the spaces are aligned, you can classify images without training a classifier. Turn each class name into a text prompt, such as "a photo of a dog." Embed each prompt, embed the image, and pick the class whose text vector is most similar to the image vector.

Walk through three classes, with made-up similarity scores:

image vs "a photo of a dog"  0.31
image vs "a photo of a cat"  0.27
image vs "a photo of a car"  0.12

prediction: dog

The text embeddings act as the classifier's weights. Adding a new class means writing one more prompt, with no retraining.

Where it works and where it breaks

Zero-shot works when the concept appears in the pretraining data and the prompt matches how the concept was described. It is sensitive to wording. "A photo of a {label}" and a domain phrase can give different accuracy, so test the template you will ship.

Known limits from the original work and later studies:

• Bag-of-concepts behavior. Early versions treated images more like a set of concepts than a precise arrangement.
• Weak OCR and counting. Reading text in images and counting objects were weaker.
• Prompt sensitivity. Small wording changes shift the scores.

CLIP-style models power multimodal search, content moderation and conditioning for image generators.

Common mistakes

• Trusting zero-shot accuracy from a benchmark. Measure accuracy on your own class list with the prompt template you will ship.
• Assuming it counts. Ask for three apples and it may not get the number right.
• Comparing raw scores across prompts. Scores from different templates are not on the same scale. Compare within one template.

Check yourself

1. What serves as the classifier's weights in zero-shot CLIP? Answer: the text embeddings of the class prompts.
2. Why must the two towers produce vectors in one shared space? Answer: so an image vector and a text vector can be compared with cosine similarity.

Key takeaways

• CLIP aligns an image encoder and a text encoder with a contrastive loss on matched pairs.
• Zero-shot classification embeds class names as text and picks the closest match.
• New classes need a new prompt, not a retrained model.
• Prompt wording and counting remain weak spots. Test your own templates.

Further learning

The CLIP paper is the original reference:

Learning Transferable Visual Models From Natural Language Supervision (CLIP)State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data i…arXiv.org

The section overview is embeddings and representation learning.