Data leakage is the most expensive bug in machine learning because it does not look like a bug. Nothing crashes. The model trains, the score is excellent, everyone is pleased, and then it meets real data and falls apart. This post covers the five common ways leakage creeps in, how to spot it, and the one habit that prevents most of it.
What data leakage is
A model is only useful if it predicts things it has not seen. That is why you hold back a test set, as covered in training and test data. Leakage is anything that lets information about the answers reach the model during training or evaluation when it would not be available in real use. The model then learns the leak instead of the pattern, and the test score measures the leak.
The experiment behind the right-hand chart is worth understanding. It used 100 rows with 5,000 columns of pure random numbers and random 0/1 labels. There is nothing to learn. But among 5,000 random columns, a few will happen to line up with the labels by chance. Pick the 20 best-matching columns using all 100 rows and those chance matches include the test rows, so every fold is marked on questions the model has already peeked at. Pick them inside each fold, from training rows only, and the fake skill vanishes.
1. Target leakage
Target leakage means a feature contains the answer, or something only known after the answer. It is the most common kind and the hardest to see, because the feature looks perfectly reasonable.
🧪 The antibiotic column
took_antibiotics. Say that in the historical data 92% of pneumonia patients have it set to yes, against 8% of the rest. The model leans on it heavily and scores 0.97 AUC.But patients receive antibiotics because they were diagnosed. At the moment you want a prediction, before diagnosis, that column is always no. In real use the model has lost its best feature and its accuracy collapses.
The test is simple: for each feature, ask “would I know this value at the exact moment I need the prediction?” If not, drop it.
Other classic examples: a days_overdue column in a loan default model, an account_closed_date in a churn model, or an ID number that was assigned in a different sequence to positive and negative cases.
2. Train-test contamination
Here the test rows influence training directly. The obvious version is training on data that includes the test set. The subtle version is tuning. If you try 200 hyperparameter settings and keep whichever scores best on the test set, you have fitted the test set by selection. Use a separate validation set, or cross-validation, for tuning, and touch the test set once at the very end.
3. Preprocessing fitted before the split
This is the leak in the figure, and it is everywhere in tutorials. Any step that learns something from the data counts: scaling learns a mean and standard deviation, imputation for missing data learns a fill value, target encoding learns category averages, and feature selection learns which columns matter.
Fit any of these on the full dataset and the test rows have helped shape the transformation. For scaling alone the effect is usually small. For target encoding and feature selection it can be enormous, as the 91% versus 50% result shows.
The rule: split first, then call fit on the training rows only, and transform on both. Never fit_transform the whole dataset.
4. Time leakage
With time-ordered data, a random split lets the model train on Thursday and be tested on Wednesday. It learns from the future. A stock model trained on a shuffled 2020 to 2025 dataset may score well because it has seen prices from the days either side of every test day.
Split by time instead: train on everything before a cut-off date and test on everything after. Features need the same care. A “30-day average sales” column must use only the 30 days before each row, not a window centred on it.
5. Duplicates and near-duplicates
If the same row, or the same person, appears in both training and test, the model can simply remember it. This happens with repeated customer records, several photos of the same patient, augmented copies of an image, or oversampled rows in imbalanced datasets. Remove exact duplicates before splitting, and when one entity has many rows, use a grouped split so all of that entity’s rows land on the same side.
| Leak | Fix |
|---|---|
| Target leakage | Drop any feature not known at prediction time |
| Train-test contamination | Tune on validation or cross-validation; test once |
| Preprocessing before split | Use a pipeline so every fit sees training rows only |
| Time leakage | Split by date; build features from the past only |
| Duplicates | De-duplicate first; group rows by entity when splitting |
How to spot it
The strongest signal is a score that is too good to be true. If published work on a problem reaches 75% and you get 99% on your first attempt, assume a leak until you have proved otherwise.
⚠️ Warning signs
- One feature dominates. If a single column carries most of the feature importance, ask when its value becomes known.
- Near-perfect scores on a hard problem. Real-world data is noisy; 0.99 AUC on messy human behaviour is suspicious.
- A big gap between validation and production. A model that scored 94% offline and manages 68% after deployment almost always had a leak.
- Random splits beat time-based splits by a wide margin. That gap is the model reading the future.
The fix: pipelines
Most preprocessing leaks come from doing steps by hand in the wrong order. A pipeline bundles every step with the model, so that whenever the pipeline is fitted, every step is fitted on the same training rows, and whenever it predicts, every step only transforms.
In scikit-learn that is pipe = make_pipeline(StandardScaler(), SelectKBest(k=20), LogisticRegression()), followed by cross_val_score(pipe, X, y, cv=5). Inside each fold the scaler and the feature selector see only that fold’s training rows. That is the right-hand bar of the figure: selection redone inside every fold, giving the honest 50% on noise.
Pipelines do not catch target leakage or time leakage, though. Those need you to think about what is known when. No library can check that for you.
🔑 Key Takeaways
- Data leakage lets answers reach the model during training or evaluation, inflating the score.
- Ask of every feature: would I know this value at the moment I need the prediction?
- Split first, then fit scalers, imputers, encoders and feature selectors on training rows only.
- Split time-ordered data by date and keep duplicates or related rows on one side of the split.
- A score that is too good to be true usually is; pipelines prevent the preprocessing leaks.
Further reading
scikit-learn’s guide to common pitfalls and recommended practices walks through preprocessing leakage with runnable pipeline examples, and the Wikipedia article on leakage lists further forms and real cases.
Where to go next
- Training and test data — the split that leakage breaks
- Cross-validation — honest scores, as long as preprocessing stays inside the folds
- Your first machine learning project — where to put the split in a real workflow
- Start Here — the full learning path