Ranking and learning to rank

After candidate generation hands you a few hundred items, ranking decides their order. Search engines, feeds, ads and RAG re-rankers all solve this problem. Given a query or user context and a set of candidates, produce an ordering that maximizes relevance or value. Position on the page matters, so the top of the list gets most of the attention.

Two things make ranking hard. Models are trained on logged clicks, and those clicks reflect the old ranking. And offline gains often disappear online. The concepts below cover the math for ordering, the metrics that judge it, and the corrections that keep training honest.

Concepts in this section

Read them in order. The first two cover the basics. The rest build on the metrics.

Common mistakes

• Trusting offline gains that ignore bias. A model can improve NDCG on logged clicks while the logs favor the old ranking. Correct for position first.
• Mismatched labels. Offline metrics often use clicks, but the online goal may be watch time. The model then optimizes the wrong thing.
• Feature skew. A feature computed differently at training and serving time breaks the model in production. See ML in production.
• Judging by metric alone. A small NDCG gain on a screen that shows three results may not matter. Match the metric to the UI.

Treat offline metrics as a filter, not as the ship decision. Move a candidate through shadow traffic, then a canary, then a full A/B test. Define the guardrail metrics before the experiment starts.

Key takeaways

• Learning to rank differs in the loss: pointwise, pairwise or listwise. Serving always sorts by score.
• NDCG discounts by position and divides by the ideal DCG. Set k to match the UI.
• LambdaMART is boosted trees trained with NDCG-scaled pairwise gradients.
• Position bias corrupts raw click logs. Log what was shown and correct with propensities or randomization.
• Calibrate scores before using them as probabilities, and blend objectives with weights that are tested online.

Further learning

Wide & Deep Learning for Recommender SystemsGeneralized linear models with nonlinear feature transformations are widely used for large-scale regression and classification problems with sparse inputs. Memorization of feature interactions through a wide set of cr…arXiv.org Introduction to Information Retrieval (Manning et al.)Introduction to Information Retrievalnlp.stanford.edu From RankNet to LambdaRank to LambdaMART (Burges, Microsoft Research)MSR-TR-2010-82.pdfmicrosoft.com XGBoost learning-to-rank tutorialLearning to Rank — xgboost 3.5.0-dev documentationxgboost.readthedocs.io

Sophie Watson's talk walks from RankNet to LambdaMART and trains on Microsoft's MSLR data: