Data leakage

Leakage means the model can see something at training time that it will not have when it makes a real prediction. It is the most common way a strong offline score turns into a production failure. The model is not learning the task. It is learning a shortcut that only exists in the training data.

The main kinds

Why churn with a cancellation reason fails

Suppose you predict whether a customer will churn, and one column is cancellation_reason. That column is filled in only after the customer cancels. The model learns "if the reason is filled in, the customer churned." The offline AUC is excellent. At prediction time, the column is empty for everyone, so the model has nothing useful.

The fix is to ask one question of every column: would this value exist at the moment the prediction is made? If not, remove it or move it to the label side.

A detective workflow

When a metric looks too good, work through these steps in order:

1. Train a model on each feature alone. Any feature with an AUC far above the baseline is a suspect.
2. Check timestamps. Compare the feature's time with the label's time.
3. Search for columns that are functions of the label, such as status, closed_at or fraud_tag.
4. Re-split by group or by time. If the metric collapses, the original split was contaminated.
5. Compare the offline lift with a shadow deployment. If the online result is flat, suspect skew or a proxy label.

Common mistakes

• Fitting preprocessing on all the data. Fit scalers, imputers and encoders on the training split only, inside the pipeline.
• Splitting repeated users at random. Split by user, so no user appears on both sides.
• Shuffling time series. Split by time, so the model is always tested on the future.

Check yourself

1. Why is cancellation_reason leakage for predicting churn? It is recorded because the customer cancelled, so it would not exist when the prediction is made.
2. Why does a random split leak for repeated users? The same user appears in both sets, so the model can memorize that user's pattern.

Key takeaways

• Leakage is information the model would not have at prediction time.
• A suspiciously strong feature or score is a signal to investigate, not to celebrate.
• Fit every preprocessing step on the training split only, and split by group or time when rows are related.
• Confirm offline wins with a shadow deployment or an experiment before you ship.

Further learning

Many leakage bugs start in feature code. The feature engineering page covers point-in-time features and how to keep them correct.