Overfitting and underfitting are the two ways a model fails. Almost every practical decision in machine learning — how complex a model to use, how much data to gather, when to stop training — is really a decision about where to sit between them.
Overfitting and underfitting, in one picture
Picture fitting a line through scattered points on a graph. There are three outcomes:
Underfitting is the straight line through data that clearly curves. The model is too simple to capture what is there. It performs badly on training data and badly on new data — which at least has the virtue of being obvious.
Overfitting is the wiggling curve that passes through every single point, including the ones that are just noise. On training data it looks flawless. On new data it is worse than the straight line, because it committed to accidents of the sample rather than the underlying pattern.
Diagnosing overfitting and underfitting from two numbers
You do not need to plot anything. Compare performance on your training set against your held-out test set:
| Training error | Test error | Diagnosis |
|---|---|---|
| High | High | Underfitting. The model cannot even fit what it has seen. |
| Low | High | Overfitting. Learned the sample, not the pattern. |
| Low | Low | A good fit. This is the target. |
The gap between the two is what separates overfitting and underfitting. A model at 99% on training and 71% on test is overfitting badly. One at 86% and 84% is healthy — a small gap is normal and not worth chasing.
Why overfitting and underfitting happen
🧪 Signal and noise
A flexible model cannot tell them apart. Given the freedom, it will contort itself to accommodate that hurried seller, because doing so lowers training error. But that discount will not recur, so the contortion actively hurts on new data.
Underfitting has the opposite cause: too little capacity, or features that simply do not carry the answer. Overfitting becomes more likely when the model has many parameters relative to the data, when you have few rows, when features are noisy or irrelevant, or when you simply train for too long.
The bias–variance tradeoff behind it
This is the formal framing, and the terms appear everywhere.
Traditionally these trade off: reducing one raises the other, and total error is lowest somewhere in the middle rather than at either extreme. In very large modern neural networks the picture is more complicated — see the work on double descent — but the intuition remains the right starting point.
Six proven fixes for overfitting
- Get more data. As above.
- Simplify the model. Fewer parameters, fewer layers, a shallower tree — less capacity to memorise.
- Regularisation. Add a penalty to the loss function proportional to parameter size, so the model is charged for complexity. L2 shrinks weights smoothly; L1 drives some to exactly zero, doubling as feature selection. That difference is really the difference between L1 and L2 norms.
- Early stopping. Watch validation performance during training. Training error falls continuously, but validation error falls then rises — that turning point is where memorisation begins.
- Drop useless features. Every irrelevant column is another opportunity to find a coincidence.
- Dropout, in neural networks: randomly switch off a fraction of neurons each pass, so the network cannot lean on any single one.
Fixing underfitting: the opposite moves
The opposite moves, and underfitting is generally the easier problem to solve:
- Use a more capable model — more parameters, more layers, a higher-degree polynomial.
- Add or improve features. Often the model is fine and the inputs simply do not contain the answer. No algorithm can predict house prices from the front door colour.
- Reduce regularisation if you have over-applied it.
- Train longer. Sometimes it has just not converged yet.
A practical order of work
⚠️ Do not reach straight for a bigger model
- Check your split is sound and nothing leaked. A surprising share of “overfitting” is really a bug.
- Get training error down first. If the model cannot fit what it has seen, that is underfitting and it is your only problem.
- Only then look at the gap to test error. That gap is the overfitting.
- Change one thing at a time, judged on validation data — never the test set.
🔑 Key Takeaways
- Overfitting and underfitting are opposite failures — diagnose them from the gap between training and test error.
- High error on both means underfitting; low training error with high test error means overfitting.
- Bias is rigidity, variance is sensitivity to the sample. Total error is lowest between the extremes.
- More data is the most reliable fix for overfitting; regularisation and early stopping are the cheap ones.
- Fix underfitting first — overfitting and underfitting are solved in that order.
Further reading
Google’s Machine Learning Crash Course on generalisation covers the same ground with interactive examples, and scikit-learn’s underfitting vs overfitting example reproduces the three-panel figure above in runnable code.
Where to go next
- Training data vs test data — how to get numbers you can trust
- Feature scaling — a common cause of models that will not converge
- What is a machine learning model? — the foundations
- Start Here — the full learning path