The ML system lifecycle

A deployed model is one part of a loop. Data arrives, features are computed, a model trains, gets evaluated, is registered, is served, and is watched. Then the loop runs again. Knowing the stages, and the artifact each one produces, is the map for everything else in production ML.

The loop

1. Data. Logs, databases, labels and human feedback.
2. Features. Transforms that must match between training and serving.
3. Training. Batch jobs or continual training, tracked as experiments.
4. Evaluation. Offline metrics and slices, with a gate before promotion.
5. Deployment. Batch scores, an online service, or on-device.
6. Monitoring. Inputs, predictions, latency and business numbers, plus delayed labels.
7. Retraining. Scheduled, or triggered when quality drops.

The model weights are one artifact among many. A training run also produces code, configuration, a data snapshot and an evaluation report. All of them matter when something breaks.

Walking through a registry

Here is a realistic sequence. A fraud model, version 3, is in production. Version 4 trains on Monday. Its offline checks pass, so the pipeline pushes it to a Staging state. A person reviews it and moves the Production alias to version 4.

Serving reads the alias, not a file path. If version 4 misbehaves, moving the alias back to version 3 takes minutes. Nobody rebuilds the service.

Record these for every production model:

• The training job ID and the git commit SHA.
• The data snapshot ID and the evaluation report.
• The thresholds and feature list, versioned with the model.

Common mistakes

• Shipping a file from a laptop. A model.pkl with no code, data or evaluation record cannot be audited or reproduced.
• Treating the model as the whole system. Feature pipelines, configuration and monitoring usually contain more code than the model itself.
• Retraining with no fixed evaluation set. Each retrain then automates a new mistake, and nobody can tell which version is better.

Check yourself

1. What makes a rollback fast in a registry setup? Answer: Serving reads a Production alias, so rollback moves the alias to the previous version.
2. Name three things to record next to every production model. Answer: The training job ID, the git commit SHA and the data snapshot ID. The evaluation report and feature list also count.

Key takeaways

• Production ML is a loop, not a single training run.
• Every stage produces an artifact that should be recorded.
• A registry with aliases makes promotion and rollback explicit.
• Most long-term cost sits in pipelines and glue code, not the choice of classifier.

Further learning

Sculley et al. describe why the model code is only a small part of a real system:

Hidden Technical Debt in Machine Learning Systemspapers.nips.cc

Next, decide how fresh each prediction needs to be: batch vs online inference.