Guide

Classical machine learning

Class imbalance

When positives are 0.1% to 5% of the data, naive training and a 0.5 threshold both fail. Fix the metric and threshold first, then try class weights, and use resampling only as a last resort.

51 of 187 · 2 min read

When the positive class is rare, a model can score well by predicting the majority class every time. Without adjustment, the training loss barely cares about the rare class, and a 0.5 threshold misses most of it. Fixing this starts with the metric and the threshold. Reweighting the loss comes next. Resampling is the last resort.

The toolkit, in order

  1. Metric and threshold. Optimize PR-AUC or a similar metric, and set the threshold from costs or review capacity. See evaluation.
  2. Class weights. Use class_weight="balanced" in scikit-learn, or scale_pos_weight in boosting. This is cheap and effective, and easy to reverse.
  3. Resampling. Undersample the majority or oversample the minority, including SMOTE-style methods. It has more moving parts, so try weights first.
  4. Calibration. If you reweight, recalibrate before you use the scores as probabilities.

Walking through balanced weights

The balanced setting gives each class a weight of n / (classes × count). Take 1,000,000 rows, with 1,000 positives and 999,000 negatives:

weight for positives = 1,000,000 / (2 × 1,000)   = 500
weight for negatives = 1,000,000 / (2 × 999,000) ≈ 0.5

Each positive now counts about 1,000 times as much as a negative in the loss. The model has to pay attention to the rare class, and the gradient is no longer dominated by the majority.

Trade-off

Weights or resampling. Prefer weights for a first pass. They add no duplicate rows and are easy to undo. Resample when the minority class is so small that batches rarely contain it, or when you are bagging and want balanced bootstrap samples. Never judge success by accuracy after either one.

Common mistakes

  • Judging the model by accuracy. On a 0.1% base rate, accuracy barely moves when the model finds fraud. Use precision, recall and PR-AUC.
  • Resampling before the split. Copies of the same rows then land in both training and test data, which is leakage. Split first, then resample only the training data.
  • Reweighting without recalibrating. Weights change the probabilities. Check calibration before you use them for expected-value math.

Check yourself

  1. Why does the balanced weight for the positives come out to about 500 here? Because it is n / (classes × count), and the positive count is only 1,000.
  2. Why must resampling come after the train-test split? Otherwise copies of the same rows appear on both sides, which is leakage.

Key takeaways

  • Start with the metric and the threshold, not with resampling.
  • Class weights are the cheap first fix, and they change the loss rather than the data.
  • Resample only the training split, and only after you have tried weights.
  • Recalibrate after reweighting if you need real probabilities.

Further learning

The threshold you pick after reweighting is a business decision, not a default. The thresholds and costs page covers how to choose it.

Features and data preparation

Turning raw data into model inputs without leaking the future. Covers scaling, encoding, missing values, time windows, target encoding and class imbalance, and keeping training and serving identical.

Evaluating models

Choose metrics that match the decision, set thresholds from costs, check calibration, and guard against leakage. A high offline score means little until it holds up on real decisions.

Thresholds and costs

A classifier's score becomes a yes or no only after you pick a threshold. The right threshold comes from the costs of each mistake, or from the review capacity you have.