Guide

Applied ML systems

Position bias

Users click the top result more often even when it is worse. Raw click logs therefore reflect the old ranking, and inverse propensity weighting, randomization and logging can correct for it.

180 of 187 · 2 min read

Users look at the top of a results list far more often than the bottom. They click the top result even when a lower one is better. This is position bias. If you train on raw clicks, the model learns the old ranking, because whatever was on top gets the most clicks.

Concept

A click has two causes. The user has to examine the slot first, then decide to click. Position bias is a change in how often a slot is examined. The propensity of a slot is the chance it is examined.

Walking through the correction

Inverse propensity scoring (IPS) weights each click by 1 / P(examine | position). A click in a slot users rarely look at is worth more, because it was less likely to happen by chance.

Suppose, as a hypothetical, slot 1 is examined 90% of the time and slot 4 only 30%:

slot 1 click weight = 1 / 0.9 ≈ 1.11
slot 4 click weight = 1 / 0.3 ≈ 3.33

A click in slot 4 counts about three times as much as one in slot 1. That corrects for the fact that slot 4 gets fewer looks in the first place.

To estimate the propensities, you need randomization or an examination model. You cannot correct for a position you did not record.

Ways to collect unbiased signal

  • Randomized traffic. A small share of traffic sees shuffled or partly random lists. The results estimate the propensity of each slot.
  • Interleaving. Results from model A and model B are merged into one list. Each click is credited to the model that supplied the clicked slot. It needs less traffic than a full A/B test.
  • Logging impressions. Record what was shown, in which slot, and what was not clicked.

Counterfactual evaluation asks what a metric would have been under a different ranking. It reuses the logged clicks with propensity weights. It is approximate and noisy, but far cheaper than launching every idea.

Common mistakes

  • Training on raw clicks. The model reinforces whatever was on top before.
  • Feeding position as a feature at training time. The model learns that the first slot causes clicks. Remove position or debias it.
  • Not logging the slate. Without what was shown and where, you cannot debias the data later.

Check yourself

  1. Why must positions be logged to correct the bias? Answer: IPS needs the examination chance of each slot the item was shown in.
  2. What does interleaving compare? Answer: two models, merged into one list, with each click credited to the model that supplied the clicked slot.

Key takeaways

  • Clicks depend on position as well as relevance. Raw clicks mislead.
  • IPS weights each click by the inverse of its slot's examination probability.
  • Randomized traffic and interleaving give cleaner comparisons.
  • Log what was shown and where. You cannot debias data you did not record.

Further learning

Next, check whether the predicted scores can be trusted as probabilities in calibration and multi-objective ranking.

Ranking and learning to rank

Second-stage ranking for search, feeds and ads. Learning to rank, ranking metrics, LambdaMART, CTR models, position bias and calibration. Start here for the map.

CTR prediction

Click-through rate models estimate the chance a user clicks an item. The model family grew from logistic regression to embedding-and-interaction networks.

Learning to rank basics

Learning to rank trains a model to order a list. Pointwise, pairwise and listwise methods differ in what the loss compares, not in how you serve the sorted list.