ROC-AUC, PR-AUC and intervals

A threshold picks one point on the trade-off. The ROC and PR curves show every threshold at once, and their areas summarize the whole ranking. The two areas answer different questions, and on rare positives they can disagree sharply. A single area number is also noisy, so it needs an interval.

Two curves, two stories

For each threshold, the ROC curve plots recall against FP / (FP + TN). The false positive rate uses the huge pool of negatives in the denominator, so it stays small even when there are many false alarms.

The PR curve plots precision against recall. Precision uses only the cases the model flagged, so it reacts quickly to false alarms.

Which one to use

An interval for the number

A single AUC from one test set is an estimate. The bootstrap resamples the test set with replacement, recomputes the AUC on each sample, and reports the middle 95% of the results.

The AUC below uses the rank formulation, so it runs with NumPy alone. It is the chance that a random positive outranks a random negative.

import numpy as np

def auc(y, scores):
    order = np.argsort(scores, kind="stable")
    ranks = np.empty(len(scores))
    ranks[order] = np.arange(1, len(scores) + 1)
    pos = y == 1
    n_pos, n_neg = pos.sum(), (~pos).sum()
    return (ranks[pos].sum() - n_pos * (n_pos + 1) / 2) / (n_pos * n_neg)

rng = np.random.default_rng(0)
y = rng.integers(0, 2, 400)
scores = y * 0.6 + rng.normal(0.3, 1.0, 400)
print("AUC", round(auc(y, scores), 3))

boot = []
for _ in range(1000):
    idx = rng.integers(0, len(y), len(y))
    boot.append(auc(y[idx], scores[idx]))
lo, hi = np.quantile(boot, [0.025, 0.975])
print(f"95% interval ({lo:.3f}, {hi:.3f})")

Output:

AUC 0.695
95% interval (0.640, 0.747)

The interval is about 0.11 wide on 400 cases. Two models whose intervals overlap are not clearly different. If model A scores 0.812 and model B scores 0.807 with overlapping intervals, pick on simplicity, latency or a secondary metric.

Common mistakes

• Using ROC-AUC on rare positives. It can look excellent while the top of the list is mostly wrong.
• Comparing two AUCs without an interval. Small differences in a single number are often noise.
• Trusting PR-AUC on a handful of positives. It is also noisy when there are only a few positive cases.

Check yourself

1. Why does a random ranker score about 0.5 on ROC-AUC? A random positive is above a random negative half the time.
2. Why is PR-AUC for a random ranker close to the base rate? Its precision is just the share of positives, which is 0.1% here.

Key takeaways

• ROC-AUC summarizes how well positives outrank negatives. PR-AUC focuses on the positive class.
• With rare positives, PR-AUC is the more honest picture.
• A bootstrap interval shows how much an AUC can move on the same data.
• Compare models with their intervals, not with the third decimal place.

Further learning

A good ranking can still produce poor probabilities. The next page covers that: calibration.