Calibration and multi-objective ranking

A ranker's scores are used for two different things. The order matters for every ranker. The scores matter as probabilities when an auction or a threshold uses them. Two ideas cover both needs: calibration, which makes a score mean what it says, and multi-objective ranking, which blends several predicted outcomes into one score.

Checking calibration

Group predictions into buckets and compare each bucket's average prediction with its actual click rate. Suppose, as a hypothetical, 1,000 items are all scored 0.3:

• If 290 are clicked, the bucket is close to calibrated.
• If only 150 are clicked, the model is overconfident by about a factor of two.

Tree and deep models are often poorly calibrated. Ads auctions depend on calibration, because the bid times the predicted click rate decides both rank and price.

Common fixes are fitted on a held-out set:

• Platt scaling fits a sigmoid to the raw scores.
• Isotonic regression fits a flexible, non-decreasing curve.
• Temperature scaling divides the logits by one learned number.

Monitor calibration in production too. It drifts as users change.

Blending several objectives

A feed may need to predict clicks, likes, watch time and reports at once. A common approach blends them into one score:

score = w1 · pCTR + w2 · pLike + w3 · E[watch] − w4 · pReport

The weights are product decisions, not facts. Tune them with online experiments and guardrails. For example, you might never let reports rise. Some teams make the goal a constraint instead: maximize watch time subject to click rate staying at or above a baseline.

Walking through a blend

Take two candidates with made-up predictions. Candidate X has pCTR = 0.10, pLike = 0.04, E[watch] = 0.5 and pReport = 0. Candidate Y has pCTR = 0.06, pLike = 0.12, E[watch] = 0.2 and pReport = 0.01. With w1 = 1, w2 = 1, w3 = 0.5 and w4 = 10:

X = 0.10 + 0.04 + 0.5 · 0.5 − 10 · 0      = 0.39
Y = 0.06 + 0.12 + 0.5 · 0.2 − 10 · 0.01   = 0.18 + 0.10 − 0.10 = 0.18

X wins. Lowering w4 to 1 lifts Y to 0.27, which is still below X's 0.39. Enough changes to the weights would flip the order, which is why the weights need owners and experiments.

Common mistakes

• Treating raw model outputs as probabilities. Check calibration on held-out data before using scores in an auction or threshold.
• Setting blend weights from offline numbers alone. Offline metrics do not capture how users respond. Test blends online.
• Leaving guardrails unwatched. A guardrail metric, such as reports, needs a dashboard and an alert.

Check yourself

1. Why does an ads auction need calibrated click probabilities? Answer: bid times predicted click rate sets the rank and price, so a biased probability mis-prices every ad.
2. Why does the report term carry a negative sign? Answer: reports are bad for the product, so a higher report probability should lower the score.

Key takeaways

• A calibrated score matches the observed frequency. Check it in buckets on held-out data.
• Tree and deep models often need fitting with Platt scaling, isotonic regression or temperature scaling.
• Multi-objective scores blend predictions with weights that are product decisions.
• Test blend weights online, with guardrails defined before the experiment.

Further learning

The section overview, with the full pipeline, is ranking and learning to rank.