Train, validation and test splits

A model's score is only as honest as its split. If the training data contains information from the future, or the same user appears in train and test, the score looks great and then fails in production. The split decides what your metric actually means.

The big idea

Use three sets:

• Train to fit the model.
• Validation to choose models, features, thresholds and hyperparameters.
• Test to report the final number, once.

Generalization is performance on data that was not used for any choice. If you look at the test score while tuning, the test set becomes a second validation set, and the reported number gets optimistic.

Random splits are often wrong

A random split assumes every row is independent of every other. Product data often breaks that assumption.

Walking through a leaky split

Suppose you have 12 months of click logs and want to predict next-day clicks. A random 80/20 split puts Tuesday morning clicks in train and Tuesday afternoon clicks from the same users in test. The model learns each user's habits, and the test score looks like generalization. It is not.

A time split avoids this. Train on January to October, validate on November, test on December. That matches what happens when the model is deployed.

This script builds a year of daily rows and splits them by date. It runs as written.

import pandas as pd

events = pd.DataFrame({
    "event_time": pd.date_range("2025-01-01", "2025-12-31", freq="D"),
})

train = events[events.event_time < "2025-10-01"]
val   = events[(events.event_time >= "2025-10-01") & (events.event_time < "2025-11-01")]
test  = events[events.event_time >= "2025-11-01"]
print(len(train), len(val), len(test))

Output:

273 31 61

Train covers January through September, which is 273 days. Validation is October, 31 days. Test is November and December, 61 days.

Three kinds of leakage

• Temporal leakage. A feature uses data from after the prediction time.
• Target leakage. A feature is a rewrite of the label, such as "account closed" when predicting churn.
• Train–test contamination. The same user, device or document appears on both sides.

Prevent them by choosing the split before you compute features. Fit every statistic, including scalers, on training rows only.

Common mistakes

• Using a random split for time-ordered data. Future rows end up in training.
• Tuning on test. Every look at the test score is a leak. Tune on validation and keep test for one final report.
• Splitting rows instead of entities. Keep a user's rows together when the question is about new users.

Check yourself

1. You have three years of logs and must predict next month. Which split? Answer: A time-based split. Train on earlier months, and validate and test on later ones.
2. Each user has 50 rows. Which split keeps the same user out of both train and test? Answer: A group split by user ID.

Key takeaways

• Train, validation and test each have one job. Use test only once.
• Random splits leak when rows are time-ordered or grouped.
• Match the split to the deployment: time, group or geography.
• Compute every statistic from training rows only.

Further learning

Next, check the model against a simple baseline on the same split: baselines and when not to use ML.