Classical machine learning
Baselines and when not to use ML
Build the simplest predictor that already drives the decision, measure it on the same split, and only then train a model. Sometimes a rule or an LLM call is the better tool.
The model to beat first is a heuristic. A baseline is any predictor you can build in an afternoon that already drives the decision. If your model cannot beat the baseline on an honest split, the problem usually lies in the framing, the labels or the features. A bigger model will not fix that.
Useful baselines
- Majority or mean. Always predict the most common class or the average value. This sets a floor.
- Business heuristic. A simple rule, such as "flag accounts with no logins in 14 days".
- Simple linear or logistic model on a few obvious features.
- The current production model, if one exists.
Here is a worked comparison. Churn is 8% of accounts. A majority classifier that always says "stays" is 92% accurate and useless. The heuristic "zero logins in 14 days" catches 40% of churners at 25% precision. Your first logistic regression must beat that heuristic on the metric that matches the decision, such as recall at a fixed outreach budget. Beating 92% accuracy is not the test.
Rules, classical ML or an LLM call
Not every prediction deserves a trained model.
| Choose | When |
|---|---|
| Rules | The policy must be auditable, labels are scarce, the logic is already known, or latency and cost forbid a model call. |
| Classical ML | Inputs are structured tabular features, you have tens of thousands of labels, and a probability score improves ranking or thresholds. |
| An LLM or API call | Inputs are messy text or images that need judgment, labels are scarce, and volume is low enough that API cost beats building a pipeline. |
Trade-off
Training a classifier wins at high volume with stable schemas, and when you need calibrated probabilities under a tight latency budget. An LLM call wins for one-off text classification, cold-start products, and tasks where the inputs are free-form language. Measure the cost per decision and the error rate on a held-out set before you commit.
Google's rules, briefly
Martin Zinkevich's Rules of Machine Learning is long. A working engineer can use the first few points:
- Do not be afraid to launch without ML. Heuristics come first.
- Keep the first model simple, and get the infrastructure right.
- Watch for training–serving skew, where the training features and the serving features differ.
- Start with interpretable features such as counts, rates and recency.
- Measure the decision metric, and slice it by segment.
Common mistakes
- Comparing to a trivial accuracy. The majority-class score is a floor. Compare against the heuristic that already runs.
- Training before the baseline exists. Without one, you cannot tell whether the model adds anything.
- Picking an LLM for a high-volume, stable decision. Cost and latency can make a small trained model the better choice.
Check yourself
- A majority-class predictor scores 92% on an 8%-positive problem. Is that a good baseline? Answer: It is a floor, not a target. It catches no positives, so judge the model on a metric that matches the decision.
- A rule catches 40% of churners at 25% precision. What should the first model beat? Answer: That rule, measured as recall at the same outreach budget.
Key takeaways
- Build a baseline before a model, and measure both on the same split.
- The baseline is the rule or heuristic that already drives the decision.
- Rules, classical ML and LLM calls each win in different conditions.
- Launch the simplest thing that works, then measure before you optimize.
Further learning
Google's Rules of Machine Learning is the standard checklist before you train anything:
Rules of Machine Learning (Google)Rules of Machine Learning: | Google for DevelopersGoogle for Developers
Google's free introduction is a fast, interactive pass over framing, splits and generalization:
Machine Learning Crash Course (Google)Machine Learning | Google for DevelopersGoogle for Developers
A talk walkthrough of the Rules of ML document, useful if you prefer listening:
Related
Framing ML problems
Learning types, business framing, data splits, baselines and when not to use ML at all.
Trees and gradient boosting
Decision trees, random forests and XGBoost/LightGBM: still the best default for tabular data.
Framing a business problem as a prediction
Turn a business decision into a prediction target, a label source, the decision it drives and the cost of each kind of error. Write these four sentences before any training.