Applied ML systems
Two-tower retrieval
A two-tower model embeds users and items separately, so item vectors can be indexed in advance and a nearest-neighbor search finds candidates fast.
Retrieval has one job: from millions of items, find a few hundred that the user might like, quickly. A two-tower model does this with two separate neural networks. The user tower turns user features into a vector. The item tower turns item features into a vector. Their only interaction is a cheap similarity, usually a dot product or cosine.
Because the towers do not interact until the final dot product, the item vectors can be computed ahead of time. That is the trick that makes retrieval fast.
Why the towers scale
Item embeddings are computed offline and loaded into an approximate nearest-neighbor (ANN) index. Libraries such as FAISS, ScaNN and HNSW do this. At request time you run only the user tower, which takes milliseconds, then ask the index for the top few hundred items by dot product.
You never run a deep network over a billion items. Any heavy cross-feature model is reserved for the ranker, which sees a few hundred candidates.
Training with negatives
For each positive pair, such as a user and an item they watched, the model should score the positive above the negatives. The usual loss is a softmax over items: push the user's vector toward the watched item and away from many others. Each of those others is a negative.
| Negative type | How you get it | Pros | Cons |
|---|---|---|---|
| Random or in-batch | Other items in the same minibatch | Free, scales with batch size | Easy negatives, so the model learns slowly |
| Popularity-sampled | Sampled by log popularity | Closer to what users actually see | Still not hard |
| Hard negatives | Near misses from ANN or an older model | Teaches fine distinctions | Noisy, and can include true positives |
Tip
In-batch negatives are the default. Add a small share of hard negatives once recall at k stops improving. Be careful: an item near the user in embedding space may be a good recommendation the user has not seen yet.
Two-tower retrieval maximizes recall, the share of good items that make it into the shortlist. It does not produce the final order. That is the ranker's job, which can afford richer features on each candidate.
Common mistakes
- Letting the towers see each other. Any cross-feature between user and item breaks precomputation and the fast search.
- Trusting nearest neighbors blindly. Retrieval quality depends on the training objective, the embedding size and the index settings.
- Mixing vectors from two model versions in one index. Results become silent garbage. Version the index with the model.
Check yourself
- Why can item vectors be computed ahead of time? Answer: the item tower does not depend on the user, so its output never changes at request time.
- Why are in-batch negatives mostly easy? Answer: random items from the batch are usually unrelated to the user, so the model quickly learns to separate them.
Key takeaways
- A two-tower model embeds users and items with separate networks and compares them with a dot product.
- Item vectors are precomputed and searched with an ANN index.
- Negatives shape training. In-batch negatives are the default, and hard negatives are added carefully.
- Retrieval is judged by recall. The ranker handles the final order.
Further learning
Next, see what happens to these models when data is thin or feedback loops form in cold start and feedback loops.
Related
Collaborative filtering and matrix factorization
Collaborative filtering recommends items from patterns of who interacted with what. Matrix factorization learns a short vector for each user and item, so a dot product predicts affinity.
Embeddings and representation learning
Learning vectors that capture meaning, from word2vec and contrastive training to CLIP and retrieval. Start here for the map of embedding concepts.
Recommender systems
How feeds and stores pick a short list from millions of items. A cheap retrieval step, a careful ranker and business rules. Start here for the map.