Deploying models safely

Deploying a model is riskier than deploying most services. A bad model still returns valid responses, so nothing crashes to tell you something is wrong. Roll the new model out in stages that compare it with the old one on real traffic, and keep a rollback ready at every stage.

The three stages

1. Shadow. The new model scores live requests, and its predictions are logged but never shown. Users see no change. You compare the two models on the same requests.
2. Canary. A small slice of real traffic, often 1% to 5%, gets the new model. You watch quality metrics and error budgets. The deployment strategies topic covers the mechanics.
3. A/B test. Users are randomly assigned to the old or the new model. You measure product metrics, not just offline scores. This needs enough traffic and a success metric agreed in advance.

For ranking models, interleaving is a faster online comparison. It mixes results from both rankers into one list and records which ones users click. See ranking.

Walking through a promotion

Take a candidate fraud model, version 4, against the current version 3.

• Shadow for a week. Say the two models agree on 97% of requests. Check where they disagree. If the disagreements cluster on new cards, you have found a risk before any user saw it.
• Canary at 2%. Watch precision and recall at the production threshold, plus error rates and latency.
• A/B test. Compare the decline rate and chargebacks between random groups. Decide only on the metric written down before the test started.

At each step, check the important slices too: new users, rare locales and high-value customers. A model that improves the average but harms one segment should not ship.

Promotion gates

• Offline metrics overall and for each important segment.
• Shadow agreement, and the cases where the models disagree.
• Canary quality and error budgets.
• A written rollback plan: the previous model and its feature configuration, ready to restore.

Promotion should be an explicit action with a checklist, not a file copy. Serving should read a named alias, so rollback is a single change.

Common mistakes

• Shipping because a leaderboard moved. Offline gains do not always become product gains.
• Skipping the holdout. Without a randomized comparison, you cannot separate the model's effect from seasonality.
• No rollback path. If the only way back is rebuilding the service, the rollout is too risky.

Check yourself

1. What does shadow mode protect users from? Answer: Any effect of the new model, because its predictions are logged and never shown.
2. Why is a canary not enough to prove a model is better? Answer: A canary shows the model is safe at small scale. Proving a gain needs a randomized comparison on a metric chosen in advance.

Key takeaways

• Shadow, then canary, then A/B test. Each stage answers a different question.
• Check the important segments, not only the average.
• Decide on the success metric before the test starts.
• Keep the previous model one step away.

Further learning

Back to the production overview: ML in production. For the service-level rollout mechanics, see deployment strategies.