Applied ML systems
ML in production
Feature stores, training-serving skew, monitoring for drift and the lifecycle of a deployed model.
A model that scores well in a notebook is not a product. ML in production is the loop that keeps predictions correct, fresh and operable under real traffic. It covers data pipelines, training, evaluation, deployment, monitoring and retraining, plus the glue software around the model weights. Classic papers estimate that the model code is a small fraction of a real system. Most of the rest is data, configuration, serving and monitoring.
This topic is the systems view for classical and deep models. LLM-specific operations, such as prompt versioning and eval harnesses, live in LLMOps. Deployment mechanics overlap with deployment strategies.
Concepts
| Step | Concept | What it answers |
|---|---|---|
| 1 | The ML system lifecycle | What are the stages of the loop, and what does each one produce? |
| 2 | Batch vs online inference | How fresh do predictions need to be, and what does that cost? |
| 3 | Training-serving skew and point-in-time features | Why do offline scores not match production? |
| 4 | Monitoring drift and delayed labels | How do you spot a model going bad before its labels arrive? |
| 5 | Deploying models safely | How do you roll out a new model, and how do you roll it back? |
Hidden technical debt
Sculley et al. (NeurIPS 2015) describe the usual shape of an ML system: a small box for the model code, surrounded by data collection, verification, feature extraction, configuration, monitoring and serving. The debt builds up in the surroundings. The patterns you will meet:
- Entanglement. Changing one feature changes the meaning of the others.
- Pipeline jungles. Brittle joins and scrapes that nobody owns.
- Ghost features. Still computed, no longer used, and still able to break.
- Under-checked data. Schemas drift, enums gain new values, and units change.
- Glue sprawl. Each model has its own one-off deploy path.
- Feedback loops. The model influences the data it is trained on. Recommenders are the textbook case.
Google's Rules of Machine Learning gives the same advice from the other side. Ship a solid heuristic baseline, keep the first model simple, watch for skew, and do not move to complex models until the infrastructure can be trusted.
Common mistakes
- Optimizing the classifier and ignoring the pipeline. Most long-term cost is pipelines and glue, not the algorithm.
- Adding online features without an owner. Each one is an outage risk.
- Launching without a monitoring plan. Decide what you will watch on day one, before the launch.
Key takeaways
- Production ML is a lifecycle loop, and the weights are one artifact among many.
- Training-serving skew and point-in-time leakage cause silent gaps. Shared feature definitions and careful joins prevent them.
- Match the inference mode to the freshness and latency the product needs.
- Monitor inputs and predictions right away, and confirm with delayed labels.
- Deploy models like risky services: shadow, canary, A/B test, and an instant rollback.
Further learning
Designing Machine Learning Systems (Chip Huyen)oreilly.com Hidden Technical Debt in Machine Learning Systemspapers.nips.cc Rules of Machine Learning (Google)Rules of Machine Learning: | Google for DevelopersGoogle for Developers
Stanford MLSys Seminar: Chip Huyen on production myths and ML systems design principles:
Related
Features and data preparation
Turning raw data into model inputs without leaking the future. Covers scaling, encoding, missing values, time windows, target encoding and class imbalance, and keeping training and serving identical.
LLMOps and observability
Tracing, prompt management, feedback loops and online monitoring for LLM applications.
Deployment strategies and safe releases
Rolling, blue-green and canary deploys, automated rollback, and zero-downtime schema changes - ship often without betting the fleet on every commit.