Data Leakage

Leakage is any situation where information reaches the model at training time that would not be available at prediction time. Read that definition carefully: it says nothing about bugs. Every pipeline below runs without error and reports a real number.

The reason it is the most expensive mistake in applied machine learning is the direction of the error. Overfitting makes your test score look bad, so you notice. Leakage makes your test score look good, so you ship. The correction arrives later, from production, in public.

Below are four pipelines over the same 360 loan applications, predicting whether a loan defaults. Three of them are lying. Read the code first.

The first pipeline reports 95.3%. The honest number is 79.4%, so it is overstating by nearly sixteen points, and it does it by including a column called collectionsCalls.

Collections calls are what the lender does after a borrower stops paying. The column is not a cause of default, it is a record of default, written down months after the decision the model is supposed to make. Feed it in and the model learns the world's least useful rule: people who got chased for money tended to default.

This is target leakage, and it is the most common serious version. It hides well because the column is real, populated, and obviously relevant. Nothing about the data says when it was written. The only defence is asking, for every column, one question: would this value be sitting in front of me at the moment I need the prediction? For collections calls the answer is no, and no amount of accuracy changes that.

The other two are subtler, and neither involves a suspicious column.

Splitting rows instead of people. Forty-five of these applicants reapplied twice, on almost identical terms, with the same eventual outcome. Split the table row by row and the same person sits on both sides of the wall. The model recognises them rather than generalising to them.

Here it is worth 0.9 of a point, which is the honest and rather deflating answer. That smallness is the danger: nobody reviews a pipeline over a one-point discrepancy. Change the setting to a hospital, where one patient contributes fifty rows across a year of visits, and the identical mistake is worth twenty points.

Splitting at random when time matters. The default rate on these loans drifts upward across the two years. A random split hands the model applications from month 20 and asks it about month 4 — it has read the ending. This pipeline reports 85.8% against an honest 79.4%.

Notice that this single careless line commits two leaks at once. It ignores time and puts the reapplications back on both sides. That is the normal case. Real pipelines rarely have one clean flaw; they have a default train_test_split call and a table nobody interrogated.

There is no library for this. Leakage is a property of what your columns mean and where your rows came from, and no function can read a data dictionary. What there is instead is a short list of questions, asked before the first model is fitted.

The last one is worth a note of proportion. Fitting a scaler on everything is the textbook example of leakage, and on this data it moves the score by so little that the pipeline is not worth showing. It is still wrong, and it still belongs in a pipeline object rather than in a line above the split — but a beginner who is worrying about their scaler while their table has three rows per customer is worrying about the wrong end.

Not every whole-dataset step is that harmless. Choosing columns is the one that bites. Below are 5,000 columns of pure noise and labels decided by a coin flip, so there is nothing to learn and the honest score is about 50%.

import numpy as npfrom sklearn.feature_selection import SelectKBest, f_classiffrom sklearn.linear_model import LogisticRegressionfrom sklearn.model_selection import cross_val_scorefrom sklearn.pipeline import make_pipeline rng = np.random.default_rng(0)X = rng.normal(size=(100, 5000))   # 5,000 columns of pure noisey = rng.integers(0, 2, 100)        # coin-flip labels: nothing to learn # Leaky: pick the 20 columns that best match y using ALL rows, then cross-validate.best = SelectKBest(f_classif, k=20).fit(X, y)leaky = cross_val_score(LogisticRegression(), best.transform(X), y, cv=5).mean() # Honest: the selection step is refitted inside each fold, on training rows only.honest = cross_val_score(    make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression()), X, y, cv=5).mean() print(f"select, then split: {leaky:.0%}")print(f"split, then select: {honest:.0%}")

Pick the twenty best-looking columns using every row and the cross-validated score comes out at 85%. Do the picking inside each fold, on training rows only, and it drops to 48%. The leaky version did not find a pattern: it chose the columns that happened to match the test rows’ coin flips.

One last habit, and it is the cheapest insurance in this track. Keep a slice of data that is later than everything you trained on, do not look at it while you are building, and score it once at the end. It cannot catch a leaked column — that one leaks into every slice equally — but it catches every mistake involving time or duplication, which is most of them.