A single train/test split gives you one number. That number depends on which rows happened to land in the test set — and on a small dataset, that luck can be worth several percentage points. Cross-validation removes the luck.
How cross-validation works
Split the data into k equal parts, called folds. Then run k separate training rounds: each round holds out one fold for testing and trains on the other k−1. Average the k scores at the end.
Read the spread, not just the average
This is the part most tutorials skip, and it is the main reason to bother with cross-validation at all.
| Fold scores | Mean | What it tells you |
|---|---|---|
| 84, 83, 85, 84, 84 | 84% | Stable. You can quote 84% with confidence. |
| 91, 68, 88, 72, 85 | 80.8% | Unstable. The 80.8% is nearly meaningless — a single split could have told you 91% or 68%. |
Both rows average to roughly the same place. Only one of them is a result you should act on. Always report the standard deviation alongside the mean.
Wide spread usually means your dataset is too small, or the folds are not comparable — which brings us to the variants.
The variants you actually need
Leave-one-out takes k all the way to n — one row held out at a time. It squeezes the most from tiny datasets, but it costs n training runs and its estimates are noisy. Reach for it only when you have very few rows.
Choosing k
k = 5 or k = 10 are the standard choices, and the difference between them rarely matters. Higher k means more training data per round (so less pessimistic scores) but more compute. Start at 5, move to 10 if your dataset is small.
The mistake that invalidates cross-validation
⚠️ Preprocess inside the loop, not before it
The same applies to feature selection, imputing missing values, and anything else that learns from the data. All of it must happen inside each fold, fitted only on that round’s training portion. In scikit-learn, wrap the steps in a
Pipeline and pass the pipeline to cross_val_score — it handles this correctly for you.What cross-validation is for
Two jobs, and it is worth keeping them separate:
- Estimating performance — how well will this model do on data it has not seen?
- Comparing options — which model, or which hyperparameter setting, is best? This is what grid search does internally: cross-validate every candidate, keep the winner.
🧪 You still need a final test set
The clean approach is nested: hold out a true test set at the very start, cross-validate within the remaining data to make all your choices, then touch the held-out set once at the end.
🔑 Key Takeaways
- Cross-validation trains k times, holding out a different fold each round.
- Report the spread as well as the mean — a wide spread means the average cannot be trusted.
- Use stratified folds for classification, group folds for repeated subjects, time-based folds for time series.
- k = 5 or 10 is almost always fine.
- All preprocessing must happen inside each fold, or the scores are inflated.
Further reading
scikit-learn’s cross-validation guide documents every splitter mentioned here, and its Pipeline documentation shows the pattern that prevents preprocessing leakage.
Where to go next
- Regularisation: L1 and L2 — the next post in this sequence
- Training data vs test data — the simpler split this builds on
- Overfitting and underfitting — what cross-validation helps you detect
- Start Here — the full learning path