Guide

Classical machine learning

Precision, recall and F1

A classifier can make two kinds of mistakes, false alarms and misses. Precision and recall measure each one, and F1 combines them. Accuracy hides both when positives are rare.

39 of 187 · 2 min read

A binary classifier answers yes or no for each case. It can be wrong in two ways. It can raise an alarm on something harmless, or it can miss something real. Precision measures the first kind of error, and recall measures the second. Together they describe how the classifier behaves on the cases that matter.

Concept

Precision is TP / (TP + FP): of the alarms raised, how many were real. Recall is TP / (TP + FN): of the real positives, how many were caught. F1 is the harmonic mean, 2 · P · R / (P + R), which is high only when both are high.

The confusion matrix

For one threshold, each case lands in exactly one cell:

Predicted positivePredicted negative
Actually positiveTrue positive (TP)False negative (FN)
Actually negativeFalse positive (FP)True negative (TN)

Walking through fraud

Take 1,000,000 transactions, of which 1,000 are fraud. The model flags 800 transactions, and 600 of those are real fraud. So:

TP = 600     FP = 200  (800 flagged, 600 real)
FN = 400     TN = 998,800
precision = 600 / 800       = 75%
recall    = 600 / 1,000     = 60%
F1        = 2 · 0.75 · 0.60 / (0.75 + 0.60) ≈ 0.667
accuracy  = (600 + 998,800) / 1,000,000 = 99.94%

The accuracy looks excellent, but it is barely better than the 99.9% you would get by never flagging anything. Precision and recall tell the real story: the model finds 60% of fraud, and three in four of its alarms are correct.

Accuracy hides the rare class

When positives are rare, accuracy is dominated by the majority class. Prefer precision and recall, or the PR curve, whenever the classes are unbalanced or the errors have different costs.

Common mistakes

  • Reporting only accuracy on imbalanced data. A model that never flags anything can look strong.
  • Looking only at F1. It hides which of precision or recall is weak. Report both.
  • Comparing F1 across thresholds without saying so. Each threshold gives a different pair of numbers.

Check yourself

  1. With the fraud numbers, if the model had caught 800 real frauds instead of 600, what would recall be? 800 / 1,000 = 80%.
  2. A model never flags anything. What is its recall? Zero, because it catches none of the real positives. Its precision is undefined, since it has no alarms to be right about.

Key takeaways

  • Precision measures false alarms, and recall measures misses.
  • F1 is the harmonic mean of precision and recall.
  • Accuracy hides the rare class, so use precision and recall for unbalanced data.
  • Always report the confusion matrix or both numbers, not just one summary.

Further learning

Precision and recall depend on the threshold you choose. The next page covers how to set it: thresholds and costs.

Evaluating models

Choose metrics that match the decision, set thresholds from costs, check calibration, and guard against leakage. A high offline score means little until it holds up on real decisions.

Thresholds and costs

A classifier's score becomes a yes or no only after you pick a threshold. The right threshold comes from the costs of each mistake, or from the review capacity you have.

Framing ML problems

Learning types, business framing, data splits, baselines and when not to use ML at all.