Applied ML systems
Monitoring drift and delayed labels
Production models fail quietly. Watch inputs, predictions and business numbers now, and confirm model quality later with labels that arrive days or weeks after each decision.
A broken model does not return a 500 error. Its accuracy slides while every service dashboard stays green. You need signals that move before the labels arrive: the distribution of inputs, the distribution of predictions, and the business numbers the model is supposed to improve. Then you confirm with delayed labels.
The big idea
Each kind of change needs a different check:
| Signal | What it catches | Typical check |
|---|---|---|
| Feature drift | Inputs have shifted | PSI or similar distribution tests on each feature |
| Concept drift | The relationship between inputs and labels has changed | Delayed-label metrics by segment |
| Prediction drift | Output scores have shifted | Mean score, score quantiles, class mix |
| Performance | Latency, errors and timeouts | Standard service monitoring plus model SLOs |
| Business KPIs | The model hurts the product | Conversion, refunds, complaints |
Walking through a PSI check
The Population Stability Index (PSI) compares a baseline distribution, such as training data or last week, with current traffic. Split the feature into bins, then add up the difference in share across bins, weighted by the log of the ratio.
Take one bin pair. The baseline is 50% and 50%. Today's traffic is 70% and 30%.
PSI = (0.70 − 0.50) · ln(0.70 / 0.50) + (0.30 − 0.50) · ln(0.30 / 0.50)
= 0.20 · 0.336 + (−0.20) · (−0.511)
≈ 0.17Common rules of thumb put a PSI below about 0.1 in the fine range, 0.1 to 0.25 as worth investigating, and above about 0.25 as a serious shift. Treat these as starting points. More useful than the number is knowing which feature moved. A new app version that sends nulls is a different incident from real seasonality.
Working with delayed labels
Labels often arrive late. Credit defaults show up after months. Churn shows up after a few weeks. A survey answer about a recommendation may take days.
Example
A churn model scores users every Sunday. Labels, meaning whether the user churned within 28 days, mature four weeks later. On day 0, watch feature null rates, the mean and 95th-percentile score, the share of users flagged, and the volume of outreach emails. On day 28, watch precision at the top decile, and the lift over a holdout group that received no outreach. If the day-0 mean score jumps 30% after a schema change, page someone now. Waiting for day 28 wastes four weeks.
Keep a holdout group that the model does not act on. Without it, you cannot measure real lift, only correlation.
Retraining policies
- Scheduled. Retrain weekly or daily. Simple, and works when data volume is steady.
- Triggered. Retrain when drift or a KPI gate fires.
- Continual. Update the model on every batch. Powerful, easy to corrupt, and rare outside teams with strong guardrails.
Always keep the previous model and its feature configuration, ready to promote again in minutes.
Common mistakes
- Alerting only on a metric you cannot compute yet. If labels arrive in two weeks, alert on null rates and score shifts now.
- Retraining on a drift alert without checking the cause. Check which feature moved, and whether it is a bug first.
- Measuring without a holdout. A metric that rises after a launch might reflect seasonality.
Check yourself
- Labels take 30 days to arrive. What can you monitor on day 1? Answer: Feature null rates, the score distribution and the volume of flagged items.
- One feature shows a PSI of 0.3. What is the next step? Answer: Find out why it moved, such as a pipeline bug, a new app version or real seasonality, before you retrain.
Key takeaways
- Production models degrade quietly, so watch inputs and predictions first.
- PSI measures how far a distribution has moved, and the feature that moved matters most.
- Delayed labels confirm quality, and a holdout group measures real lift.
- Keep a rollback model ready before any retrain.
Further learning
Next, roll out the new model in stages: deploying models safely.
Related
ML in production
Feature stores, training-serving skew, monitoring for drift and the lifecycle of a deployed model.
Training-serving skew and point-in-time features
Training-serving skew happens when the code or data used to predict in production differs from the code or data used to train. Point-in-time joins and shared feature definitions prevent most of it.
Deploying models safely
Roll out a new model in stages, shadow then canary then A/B test. Gate each step on offline slices and product metrics, and keep the previous version one step away.