Classical machine learning
Overfitting, underfitting and bias–variance
A model that memorizes its training data looks brilliant until it sees new data. A model too simple for the pattern fails on both. Two learning curves tell you which problem you have.
A model can fail in two opposite ways. It can overfit, learning quirks of the training data that will not repeat. Or it can underfit, being too simple to capture the real pattern. The bias–variance trade-off is the classic name for the tension between the two. You can usually tell which problem you have from two numbers: the training score and the validation score.
The big idea
Overfitting and underfitting are the two ends of a dial, and model flexibility sets the dial:
- Rigid models (a linear model with a few features) have high bias. They are consistently wrong in the same direction, but they barely change between datasets.
- Flexible models (a deep tree, a huge network) have low bias. They can fit almost anything, but they change a lot from one training sample to the next. That change is variance.
An analogy: a tailor who sews each suit to one customer's exact measurements fits that customer perfectly and fails on anyone else. A tailor who makes one size for everyone fits nobody well. The analogy breaks down once you have enough data, because a flexible model with plenty of data and good regularization can generalize well.
Walking through the two curves
Track the training score and the validation score as the model gets more capacity. Two patterns cover most cases. The numbers below are illustrative.
| Training error | Validation error | Diagnosis |
|---|---|---|
| 2% | 30% | Overfitting. The model fits the training data but not new data. |
| 40% | 40% | Underfitting. The model is too simple, or the features or pipeline are wrong. |
For overfitting, the fixes are regularization, more data, or a simpler model. For underfitting, the fixes are richer features, a more flexible model, or a bug. A common bug is a wrong label join that quietly makes the labels noisy.
In product language: overfitting looks like "amazing on last month's random split, falls apart in the new region". Underfitting looks like "the model predicts the majority class, and the segments we care about are ignored".
Common mistakes
- Reading training accuracy as progress. It always goes up with more capacity. Look at the validation curve.
- Counting rows instead of independent people. One million rows from ten thousand users is not a million independent examples. Use a group-aware split to see the real signal.
- Adding capacity before checking the split. An honest split often reveals the overfitting that a random split hid.
Check yourself
- Training error is 2% and validation error is 30%. Overfitting or underfitting? Answer: Overfitting. The model fits the training data but not new data.
- Training and validation errors are both 40%. What is the likely cause? Answer: Underfitting. The model is too simple, or the features or pipeline are wrong.
Key takeaways
- Overfitting fits noise. Underfitting misses the pattern.
- Read the training and validation scores together.
- Rigid models are biased but stable. Flexible models fit more but vary more.
- Check that the split matches deployment before you tune capacity.
Further learning
An Introduction to Statistical Learning covers bias, variance and train/test thinking with clear examples:
An Introduction to Statistical LearningAn Introduction to Statistical LearningNext, see the kinds of learning problems you might be framing: types of learning.
Related
Framing ML problems
Learning types, business framing, data splits, baselines and when not to use ML at all.
Train, validation and test splits
Fit on train, tune on validation, and report once on test. Random splits leak the future and repeated users, so split the way the model will be used.
Evaluating models
Choose metrics that match the decision, set thresholds from costs, check calibration, and guard against leakage. A high offline score means little until it holds up on real decisions.