Thresholds and costs

Most models output a score or a probability. A threshold turns that score into a decision: flag the case if the score is at or above the threshold. Moving the threshold slides you along the precision-recall trade-off. A lower threshold catches more real positives, and it also raises more false alarms. The right place for the threshold depends on what each mistake costs.

The default is usually wrong

A threshold of 0.5 is a convention, not a decision. It assumes that a false alarm and a miss cost the same, and that the scores are calibrated probabilities. Neither assumption holds for most real problems.

Setting the threshold from costs

If a miss costs more than a false alarm, lower the threshold. Let c_fp be the cost of a false positive and c_fn the cost of a false negative. For a threshold, the expected cost on validation data is:

expected cost ≈ FP × c_fp + FN × c_fn

threshold 0.3:  FP = 2,000,  FN = 50   →  2,000 × 10 + 50 × 400  = $40,000
threshold 0.7:  FP = 200,    FN = 180  →  200 × 10 + 180 × 400    = $74,000

Setting the threshold from capacity

Sometimes the limit is not money but people. If a review team can handle 500 cases a day, choose the threshold that yields about 500 flags a day, then pick the one with the highest precision among those. The threshold follows the capacity, and the model has to fit it.

Sweeping thresholds in code

The sweep below uses made-up scores and the same costs as the example. It tries 19 thresholds and keeps the cheapest. Run it on a validation set, not on the test set.

import numpy as np

def expected_cost(y_true, scores, threshold, c_fp=10, c_fn=400):
    pred = scores >= threshold
    fp = np.sum(pred & (y_true == 0))
    fn = np.sum(~pred & (y_true == 1))
    return fp * c_fp + fn * c_fn

rng = np.random.default_rng(1)
y = rng.integers(0, 2, 1000)
scores = np.clip(0.3 * y + rng.uniform(0, 0.7, 1000), 0, 1)

thresholds = np.linspace(0.05, 0.95, 19)
costs = [expected_cost(y, scores, t) for t in thresholds]
best = thresholds[int(np.argmin(costs))]
print(f"best threshold {best:.2f}, cost {min(costs)}")

Output:

best threshold 0.30, cost 2970

The cost curve is steep at both ends. Very low thresholds flood the review queue with false alarms, and very high thresholds miss too many positives.

Common mistakes

• Using 0.5 without checking. Check the costs first.
• Tuning the threshold on the test set. The test set is for the final number. Choose the threshold on validation data.
• Assuming the threshold stays valid. When the score distribution shifts, the best threshold moves too. Re-check it on recent data.

Check yourself

1. If a miss costs ten times a false alarm, which way should the threshold move from 0.5? Lower, to catch more positives, which accepts more false alarms.
2. In the cost example, why is threshold 0.3 cheaper even though its precision is lower? Each missed positive costs $400, against $10 for a false alarm, so fewer misses matter more.

Key takeaways

• A threshold turns scores into decisions, and it is a choice about costs, not a default.
• Lower thresholds raise recall and lower precision.
• Set the threshold from expected cost or from review capacity, on validation data.
• Store the threshold with the model, so serving uses the same value.

Further learning

Curves summarize every threshold at once. The next page covers ROC and PR curves, and how sure you can be about the numbers: ROC-AUC, PR-AUC and intervals.