Data Leakage
When information that should stay separate crosses into training
- Data leakage is when information that should have stayed separate crosses over into training by accident.
- It has one telltale sign: the score comes out suspiciously good. Then it vanishes the moment the model actually gets used.
- It's a different problem from overfitting. Overfitting is memorizing too much; leakage is seeing something it shouldn't have.
- The most common route is order: clean the whole dataset before splitting it, and information crosses over right then.
- Leakage doesn't lower performance. It inflates it, which is exactly why it's so much harder to notice.
Contents
1The analogy
You want to know how warm a new space heater really gets a room. So you close the bedroom door and leave a thermometer inside. An hour later, the number has climbed nicely. Looks like a great heater.
Except the door had a finger-width gap. Warm air from the living room's furnace kept drifting through that gap the whole time. The thermometer's reading was real, but it wasn't the bedroom heater that produced it. Data leakage is that gap under the door.
Seal the gap properly and measure again, and the number drops sharply. Disappointing, but that's this heater's actual performance. Believe it was a great heater while the gap stayed unnoticed, and the mistake would only show up on the coldest day.
2In detail
Crossed-over information inflates the score
When building a model, data gets split into a training set and a test set. The model learns from the training set, and the test set stays untouched until the very end, when it's pulled out to measure real skill. It only means something if the test set is data the model has genuinely never seen.
But if even a sliver of the test set crosses into training, the model gets graded while already knowing the answers. The score comes out great, and everyone believes they built a good model. The trouble shows up the moment it actually gets deployed: there's no crossed-over information waiting for it there, and performance drops sharply.
What makes this failure especially dangerous is its direction. Most problems announce themselves by dragging the score down. Leakage hides itself by pushing the score up. When a result looks unusually good, checking for a gap under the door should come before celebrating.
Where the gaps show up
The first gap is order. Fill in blanks with an average, or rescale values, using the whole dataset, then split into training and test afterward, and the test data's values have already been folded into that average. Any adjustment based on the data should happen after the split, using only what the training set says.
The second gap is the same thing appearing on both sides. One person's records spread across several rows, split row by row, and the same person can end up in both training and test. Several photos of the same product taken from different angles cause the identical problem. Splitting needs to group by person, or by item, not by row.
The third gap is time. For a problem about predicting what happens next, shuffling the data randomly before splitting means the model learns from something that happened later to predict something that happened earlier. Any problem with a time dimension needs the split to fall at a fixed point, with everything after it held out.
The column that carries the answer in disguise
The hardest gap to spot is a column that carries the answer under a different name. A problem about predicting an illness that includes a "prescribed medication" column lets the model read off the medication name instead of recognizing the illness. A problem about predicting who's about to cancel that includes a "cancellation processed on" date does the same thing.
Columns like this often look perfectly natural to a person. So every column needs checking for when its value actually gets created. If a value only exists after the moment you're trying to predict, it doesn't exist yet at the time the model would actually be used, and it shouldn't go into training.
How to find where it's crossing over
The best signal is a score that's too good. If it's nailing a hard problem almost perfectly, get suspicious before celebrating. If dropping a single column tanks the score, that column is a strong suspect.
Locking the sequence of steps into one fixed pipeline helps too. Split, clean, train, grade, done by hand each time, the order tangles easily. Fix the order in place, and there's simply less room left for a gap to form.
3More precisely
Leakage is often confused with overfitting. Overfitting is memorizing the training data too closely and falling apart on new data, and the test score there comes out low. Leakage is distinctive because the test score comes out high too, the test set has simply stopped being new data.
The analogy breaks in one place. A gap under a door can be felt as a draft, but a gap in data is mostly invisible. Run repeated splits for cross-validation and a gap can show up consistently, making performance look uniformly good across every split. So leakage tends to get found by retracing how the data was assembled, more than by staring at the score alone.
Worth knowing too: the word covers two related situations. One is the case just described, information from the test set crossing into training. The other is when information that would never actually be available at prediction time gets included in training at all. The first only makes the score untrustworthy; the second makes the model itself unusable, since there's no way to reproduce that missing information once it's deployed.
4Try it yourself
5Common misconceptions
It's easy to think leakage makes performance worse, but actually it makes performance look better than it is, which is exactly why it takes so long to notice.
It's easy to assume shuffling the data thoroughly before splitting prevents leakage, but actually thorough shuffling is itself what causes leakage in a problem that has a time order.
It's easy to think leakage only happens when test data gets mistakenly put into training, but actually it happens far more often through cleaning done before the split, and through columns whose values only exist later.
7One-line summary
In shortData leakage is when information that should have stayed separate crosses into training, and because it inflates the score instead of hurting it, an unusually good result is the first thing to be suspicious of.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02