Calibration

A probability is useful only if it matches what happens. A calibrated model is one where cases scored at 0.8 are positive about 80% of the time, cases scored at 0.3 are positive about 30% of the time, and so on. Ranking metrics do not check this. A model can rank cases perfectly and still give probabilities that are far off.

Checking with a reliability table

To check calibration, bin the predicted probabilities and compare each bin's average prediction with the observed rate of positives. A calibrated model shows nearly equal numbers in each row.

The code below simulates a model that is calibrated by construction, because each outcome is drawn from its own predicted probability. A real model would replace p and y with its validation predictions and labels.

import numpy as np

rng = np.random.default_rng(2)
p = rng.uniform(0, 1, 2000)                      # predicted probabilities
y = (rng.uniform(0, 1, 2000) < p).astype(int)    # outcomes drawn from those probabilities

edges = np.linspace(0, 1, 6)
bin_id = np.clip(np.digitize(p, edges) - 1, 0, 4)
print("bin          n   predicted  observed")
for b in range(5):
    m = bin_id == b
    print(f"[{edges[b]:.1f}, {edges[b+1]:.1f})  {m.sum():4d}     {p[m].mean():.2f}      {y[m].mean():.2f}")

Output:

bin          n   predicted  observed
[0.0, 0.2)   421     0.10      0.14
[0.2, 0.4)   376     0.29      0.30
[0.4, 0.6)   396     0.50      0.51
[0.6, 0.8)   401     0.70      0.70
[0.8, 1.0)   406     0.90      0.85

Each row matches within sampling noise, which is what calibration looks like. Each bin has only a few hundred cases, so small gaps like 0.14 against 0.10 are expected noise. A systematic gap across all bins would be a real problem.

Fixing a miscalibrated model

Models trained with log loss are often well calibrated. Boosted trees and deep networks often are not, so you may need a post-hoc fix fitted on held-out data:

• Platt scaling fits a logistic regression that maps raw scores to probabilities. It has two parameters, so it needs little data.
• Isotonic regression fits a non-decreasing step function. It is more flexible, but it needs more validation data to avoid overfitting.

Fit either method on validation data, never on the training set. Check the reliability table again afterward.

Common mistakes

• Equating a high AUC with calibration. AUC depends only on the order of scores. Any monotonic rescaling keeps the AUC the same while changing the probabilities.
• Calibrating on the training set. The model has already seen those labels, so the fix will look better than it is.
• Skipping the check. Calibration can drift when the population changes, so re-check it on recent data.

Check yourself

1. A model says 0.9 for 100 cases, and 60 of them are positive. Is it calibrated? No. The observed rate is 0.6, not 0.9.
2. Can a model with perfect ranking be badly calibrated? Yes. Ranking only depends on order, so a monotonic change to the probabilities leaves the ranking alone.

Key takeaways

• Calibration means predicted probabilities match observed frequencies.
• Check it with a reliability table, comparing average predictions and observed rates per bin.
• Platt scaling and isotonic regression are common fixes, fitted on validation data.
• A good AUC says nothing about calibration.

Further learning

Probabilities are only as trustworthy as the labels behind them. The last check is whether the model saw information it should not have: data leakage.