Separating training and test data is the first rule of machine learning that actually has teeth. Break it and every number you report will be flattering and wrong. This page explains why, and how to split data properly.
The exam analogy
🧪 The intuition
Do they understand maths? You cannot tell. They might have grasped the methods — or memorised twenty specific answers. The only way to find out is to hand them a different paper.
That is exactly the situation with a trained model. Its accuracy on training data tells you almost nothing, because a high score there can be achieved two ways: by learning the real pattern, or by memorising the examples. Those look identical from the inside and behave completely differently on new data.
The basic training and test data split
The essential property is that the test set must be genuinely unseen. The moment you train on it, it stops measuring anything useful — you have handed the student the exam paper in advance.
⚠️ Shuffle before you cut
Where the validation set comes in
There is a subtler trap. Suppose you train a model, test it, get 82%, adjust a setting, retrain, test again, get 84% — and repeat this twenty times, keeping whichever scores best.
You have now leaked the test set. Not by training on it directly, but by using it repeatedly to make decisions. Your final 91% is optimistic, because you selected the model that happened to suit those particular rows.
| Set | Size | Used for | How often |
|---|---|---|---|
| Training | 60–70% | Adjusting the model’s parameters | Constantly |
| Validation | 15–20% | Comparing settings and models | Many times |
| Test | 15–20% | The final honest estimate | Once |
The validation set absorbs the wear of repeated decisions. The test set stays clean because you touch it exactly once, when everything is finalised.
Cross-validation, when training and test data is scarce
If you only have 300 rows, carving off 20% leaves 60 test rows — too few to mean much, since the result might swing wildly depending on which 60 you happened to pick.
Report their average and their spread. Scores of 84, 83, 85, 84, 83 mean the model is stable. Scores of 91, 68, 88, 72, 85 mean something is wrong — your dataset is probably too small or too varied, and any single number you quote would be an accident of the split.
Four ways a training and test data split goes wrong
⚠️ Leakage checklist
- Scaling before splitting. Computing a mean across the whole dataset leaks test information into training. Split first — see feature scaling.
- Duplicate rows. The same record in both sets means you tested on training data. De-duplicate first.
- Random splits on time series. A random split lets the model train on Thursday and Saturday to predict Friday — it has seen the future. Split chronologically instead.
- Grouped data. Forty photos of each of fifty patients, split randomly, puts the same patient on both sides. The model recognises the patient, not the condition. Split by group.
Reading your training and test data scores together
The gap between training and test scores is diagnostic:
| Training | Test | What it means |
|---|---|---|
| High | High | Working. Ship it. |
| High | Low | Overfitting — memorised rather than learned |
| Low | Low | Underfitting — model too simple, or features too weak |
| Low | High | Almost always a bug in your split or evaluation code |
That last row surprises people, but a test score meaningfully above the training score is a red flag rather than good luck. Check for leakage or a mistake in how you compute the metrics.
A note on accuracy
Splitting correctly gives you an honest number. It does not guarantee the number is meaningful. On imbalanced data — fraud, rare diseases — a model that always predicts “no” can be 99% accurate and useless. Once your split is right, learn to read the confusion matrix, then precision and recall.
🔑 Key Takeaways
- Split your training and test data randomly, before you do anything else to it.
- The test set is used once. Use a validation set for every decision before that.
- With little data, use k-fold cross-validation and report the spread, not just the average.
- Time series split chronologically; grouped data splits by group.
- A test score above your training score means a bug, not luck.
Further reading
In practice you will split training and test data with a library rather than by hand: see scikit-learn’s train_test_split and its cross-validation guide, which covers grouped and time-series splitting properly.
Where to go next
- Overfitting and underfitting — what the train/test gap is telling you
- What is a machine learning model? — if “parameters” is still fuzzy
- Classification metrics — what to measure once the split is right
- Start Here — the full learning path