Classical machine learning
Point-in-time features
A feature is valid only if it could have been computed at prediction time. Time windows are the most common source of leakage, and a parity checklist keeps training and serving identical.
Window features are the core of product ML: counts, sums and rates over the last 1, 7 or 28 days, time since the last event, and day of week. They are also where most leakage hides. A feature built from data after the prediction time looks brilliant offline and fails in production.
The rule is simple. For a label defined at time t, every feature must come from data stamped before t. Data stamped at t itself can count only if it was visible to the production system at prediction time.
A churn example
Predict on day t whether a customer will churn, using orders_30d. The correct feature counts orders in [t − 30 days, t).
If the feature also counts orders on day t, after the prediction job ran, that is temporal leakage. The notebook score is high, because the model sees the orders that caused the churn. In production, those orders do not exist yet.
Feature stores solve this with point-in-time joins, which fetch each feature as it was at the prediction time. If you build features by hand, write the window bounds down and test them.
Serving parity checklist
Training-serving skew is more often a feature problem than a model problem. Check each row:
| Check | Failure mode |
|---|---|
| Same join keys and time windows | Training used a 30-day window including today. Serving uses yesterday's batch |
| Same timezone and week start | Weekly totals disagree by one day |
| Same rare-category handling | Training saw __OTHER__, but production sends new raw IDs |
| Same imputation constants | Training filled with a median of 12.4, and production hard-codes 0 |
| Same code path | A notebook in pandas, and a separate rewrite in production SQL |
Prefer one implementation: a feature store, a shared library or a serialized pipeline. Duplicated SQL is where drift starts. The ML in production page covers the serving side.
Trade-off
Batch or real-time features. Batch features, such as daily aggregates, are simpler, cheaper and easier to make point-in-time correct. Real-time features, such as the last five minutes of events, matter for fraud and feed ranking, where freshness is the product. Start with batch. Add streaming features when offline analysis shows that freshness would move the metric.
Common mistakes
- Including the label's own day. Use a window that ends before
t, unless the data was visible at prediction time. - Mismatched timezones. A day boundary in UTC is not the same day as one in local time.
- Hard-coded defaults. A fill value typed into the serving code, and not learned from training, will eventually differ.
Check yourself
- For a label at day
t, which window is correct fororders_30d? Orders in[t − 30 days, t), which excludes dayt. - Why does a notebook feature often break in production? The production path is rewritten, often in SQL, so windows, timezones or defaults can drift.
Key takeaways
- Every feature must be computable at the time of prediction.
- Window bounds are the usual leak. Write them down and test them.
- Keep one implementation of each feature, shared by training and serving.
- Start with batch features, and add real-time ones when freshness is measurably worth it.
Further learning
Encoding a category by its label is the next place leakage tends to appear: target encoding without leakage.
Related
Features and data preparation
Turning raw data into model inputs without leaking the future. Covers scaling, encoding, missing values, time windows, target encoding and class imbalance, and keeping training and serving identical.
Data leakage
Leakage is when a model learns from information it would not have at prediction time. Offline scores look excellent, then production falls apart. Here is how to find it.
ML in production
Feature stores, training-serving skew, monitoring for drift and the lifecycle of a deployed model.