Training-serving skew and point-in-time features

A model can score well offline and fail in production with no error message. The usual cause is skew: the features used at serving time are not the same as the features used in training. Two kinds matter. One is code and defaults that differ between the two paths. The other is training rows that quietly use information from the future.

The big idea

Skew is any systematic difference between the training path and the prediction path. It is silent. The offline AUC was real, but the online features are different variables, so the model is working with inputs it never learned to read.

An analogy: a pilot trains in a simulator with one set of instruments, then flies a plane with the dials in different units. The skill is real. The readings are not.

Walking through point-in-time features

A fraud label arrives 14 days after each transaction. Suppose a training row is for day 12. Its features must use only what was known on day 12.

A user placed 3 orders by day 10 and 5 orders by day 20. For the day-12 row, the feature "orders in the last month" must be 3. Using 5 would leak information from day 20 into a row about day 12.

That is a point-in-time join. For each training row at time t, take the latest feature values with a timestamp at or before t. Take labels only from after the label horizon.

The common mistake is joining today's snapshot, such as "current lifetime value", onto last year's events. The join looks harmless, and the model learns from the future. Feature stores exist largely to make correct point-in-time joins routine.

A skew checklist

Check these before the first launch of any model:

1. Names and types. The same feature names and types in the training data and the serving requests.
2. Defaults. The same value for missing data. Training with 0 and serving with −1 is a common failure.
3. Time windows. "Last 7 days" must end at the request time, not at a batch cutoff.
4. Unknown categories. Handle unseen IDs at serving time the same way training did.
5. Shared tests. Run the training transform and the serving transform on the same fixtures, and assert they match.

Also log a sample of production feature vectors. Compare their distributions with the training data. A weekly job that reports the largest relative difference per column catches skew early.

Common mistakes

• Different defaults for missing values. The model sees a value at serving time it never trained on.
• Windows that end at different times. A batch job and a live request can disagree on "the last week".
• Trusting offline metrics alone. Compare the logged serving features with the training features.

Check yourself

1. Training fills missing income with 0, but serving fills it with −1. What happens? Answer: The model meets a value at serving time that it never learned from, and its predictions shift without any error.
2. Why must a training row for day 12 ignore an order placed on day 15? Answer: The model would learn from information that did not exist at prediction time. That is leakage.

Key takeaways

• Skew is a silent gap between training and serving.
• Point-in-time joins keep training rows honest by using only past information.
• Shared feature definitions and shared transform tests prevent most skew.
• Log serving features and compare them with training distributions.

Further learning

Feature construction is covered in more depth here: features and data preparation.

Next, watch the model after launch: monitoring drift and delayed labels.