Data leakage: why your model is brilliant until it ships

The most dangerous model is the one with a great validation score. In one of the examples below, a churn model scores 0.934 accuracy on held-out data, and 0.535 once it is given only what it would actually know on the day it makes a prediction. Nothing is wrong with the algorithm. The test was leaking the answers.

What leakage actually is

A validation score is a promise: "this is roughly how the model will do on data it hasn't seen." Leakage breaks that promise quietly. Somewhere between the raw table and the score, information that the model won't have in production gets into training, or into the choices made before training. The model learns to use it, the score goes up, and nothing errors.

The mistake I see most isn't exotic. It's a sensible-looking line of pandas or scikit-learn in the wrong place. That's why it survives code review: every line is correct on its own, and the bug is in the order.

The numbers from the examples in this post, all on synthetic data:

CV accuracy on pure noise

0.830

CV accuracy on pure noise

R² forecasting a random walk from the date alone

0.990

R² forecasting a random walk from the date alone

MAE on a noisy sales series

0.00

MAE on a noisy sales series

None of those are real skill. Every example below runs as is on Python 3.14 with scikit-learn 1.9.1, pandas 3.1.0 and NumPy 2.5.4. Seeds are fixed, so with those versions you should get the same output.

1. Preprocessing leakage: fitting on rows you're about to test on

Scalers, imputers, encoders and feature selectors all have a fit method, which means they learn something from the data. If they learn it from the whole dataset before you split, the test rows have already shaped the model before it's tested on them.

With scaling or mean imputation the inflation is usually small, which is exactly why it goes unnoticed. Feature selection makes the same mistake impossible to miss. Here are 10,000 columns of pure noise and labels assigned at random. There is nothing to learn.

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
 
rng = np.random.default_rng(0)
X = rng.normal(size=(200, 10_000))  # pure noise
y = rng.permutation(np.repeat([0, 1], 100))  # random labels, 50/50
 
# Leaky: pick the 20 "best" features using ALL rows, then cross-validate
X_top = SelectKBest(f_classif, k=20).fit_transform(X, y)
leaky = cross_val_score(LogisticRegression(), X_top, y, cv=5)
 
# Correct: selection is a pipeline step, so it is refit inside each fold
pipe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression())
honest = cross_val_score(pipe, X, y, cv=5)
 
print(f"leaky  CV accuracy: {leaky.mean():.3f}")
print(f"honest CV accuracy: {honest.mean():.3f}")

leaky  CV accuracy: 0.830
honest CV accuracy: 0.480

83% accuracy from noise. With 10,000 random columns, some will line up with the labels by chance, and SelectKBest found the 20 that lined up best across all 200 rows, including the rows that later became test folds. The test labels helped choose the features they were then tested on.

Inside the pipeline, selection is refitted on the 160 training rows of each fold, so the chosen columns know nothing about the 40 test rows. The score drops to 0.480, a coin flip, which is the correct answer for random labels.

The same applies to anything with a fit: StandardScaler, SimpleImputer, PCA, and especially target encoding of high-cardinality categories, which can leak almost as badly as selection. scikit-learn's TargetEncoder cross-fits inside fit_transform for exactly this reason. Oversampling before the split is a cousin of this bug: copies of minority rows land on both sides, which is the duplicates problem in section 5.

2. Target leakage: features that only exist after the answer

Target leakage is a feature that is recorded after, or because of, the outcome you're predicting. It looks like a strong predictor because it partly is the outcome.

An illustrative case: predicting which customers will cancel, with a refund_issued column in the table. Refunds get issued when people cancel, so the column is nearly a copy of the label. At prediction time, though, the customer hasn't cancelled yet, so no refund has been issued.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
 
# Weak, honest signal: 5 features that only partly explain churn
X, churned = make_classification(n_samples=5000, n_features=5, n_informative=3,
                                 flip_y=0.3, random_state=0)
 
# refund_issued is only filled in AFTER a customer cancels
rng = np.random.default_rng(0)
refund_issued = ((churned == 1) & (rng.random(5000) < 0.9)).astype(int)
 
X_tr, X_te, y_tr, y_te = train_test_split(np.column_stack([X, refund_issued]), churned,
                                          random_state=0)
honest = RandomForestClassifier(random_state=0).fit(X_tr[:, :5], y_tr)
leaky = RandomForestClassifier(random_state=0).fit(X_tr, y_tr)
 
print(f"honest model, test accuracy:      {honest.score(X_te[:, :5], y_te):.3f}")
print(f"leaky model, test accuracy:       {leaky.score(X_te, y_te):.3f}")
print(f"importance of refund_issued:      {leaky.feature_importances_[-1]:.3f}")
 
# At prediction time nobody has churned yet, so refund_issued is always 0
X_live = X_te.copy()
X_live[:, -1] = 0
print(f"leaky model, as-of-today inputs:  {leaky.score(X_live, y_te):.3f}")

honest model, test accuracy:      0.765
leaky model, test accuracy:       0.934
importance of refund_issued:      0.681
leaky model, as-of-today inputs:  0.535

The test split is perfectly clean here. Rows were split at random and no preprocessing touched the test set, and the leaky model still scores 0.934. The leak is in the column itself, so no splitting strategy will catch it.

The last line is what production looks like. Fed the inputs it will really get, the leaky model falls to 0.535, well below the honest model's 0.765. It spent its capacity learning "refund means churn" and "no refund means stay", and in production every customer has no refund.

Typical suspects, as illustrations: a status field that gets overwritten when the outcome happens, "days since last login" computed on the day the data was extracted rather than the day of the prediction, a count of collection calls when predicting loan default, or a diagnosis code entered after the test you're trying to predict.

3. Temporal leakage: shuffling time and peeking ahead

When rows have dates, a random split answers the wrong question. It asks "can you fill in a gap between days you've seen?" when production asks "can you predict a day you haven't reached yet?"

Random K-fold on time-ordered data

Here's a deliberately extreme case: a metric that drifts like a random walk, and a model whose only feature is the day number.

import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import KFold, TimeSeriesSplit, cross_val_score
 
# Two years of a daily metric that drifts like a random walk
rng = np.random.default_rng(0)
days = np.arange(730).reshape(-1, 1)
level = np.cumsum(rng.normal(size=730))
 
model = RandomForestRegressor(random_state=0)
shuffled = KFold(n_splits=5, shuffle=True, random_state=0)
forward = TimeSeriesSplit(n_splits=5)
 
for name, cv in [("shuffled KFold", shuffled), ("TimeSeriesSplit", forward)]:
    r2 = cross_val_score(model, days, level, cv=cv, scoring="r2")
    print(f"{name:16s} R^2: {r2.mean():.3f}")

shuffled KFold   R^2: 0.990
TimeSeriesSplit  R^2: -1.050

With shuffled folds, the model is tested on day 100 after training on days 99 and 101, so it just interpolates and scores 0.990. TimeSeriesSplit only ever trains on the past and tests on the next block, so the forest has to extrapolate, and the best it can do is repeat the last level it saw. An R² below zero means it did worse than predicting the test period's average. That's the honest result: the date alone can't forecast a random walk.

Real tables rarely have a bare day number as a feature, but they often have something that behaves like one: an auto-incrementing ID, an account number, a "version" column. If one of those ranks high in feature importance, check whether it's standing in for time.

If labels arrive late (a loan defaults 90 days after it's issued, say), TimeSeriesSplit takes a gap argument that drops rows between the training and test blocks so the model can't train on outcomes that wouldn't have been known yet.

Features computed with future information

The split can be right and the features can still look ahead. Rolling windows are the usual way in. The goal below is to predict today's sales from what's known at the start of today.

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import TimeSeriesSplit, cross_val_score
 
# Daily sales with a weekly pattern and plenty of noise
rng = np.random.default_rng(0)
day = np.arange(730)
df = pd.DataFrame({"sales": 100 + 10 * np.sin(2 * np.pi * day / 7) + rng.normal(0, 8, 730)})
 
# Goal: predict today's sales from what we know at the start of today
df["lag_1"] = df.sales.shift(1)
df["lag_2"] = df.sales.shift(2)
df["avg_3_leaky"] = df.sales.rolling(3).mean()  # window includes today
df["avg_3"] = df.sales.shift(1).rolling(3).mean()  # yesterday and before
df = df.dropna()
 
cv = TimeSeriesSplit(n_splits=5)
for avg in ["avg_3_leaky", "avg_3"]:
    mae = -cross_val_score(LinearRegression(), df[["lag_1", "lag_2", avg]], df.sales,
                           cv=cv, scoring="neg_mean_absolute_error").mean()
    print(f"features lag_1, lag_2, {avg:12s} MAE: {mae:.2f}")

features lag_1, lag_2, avg_3_leaky  MAE: 0.00
features lag_1, lag_2, avg_3        MAE: 7.70

A perfect forecast of a noisy series, using time-aware cross-validation. rolling(3).mean() averages today with the previous two days, and the previous two days are also features, so a linear model just solves for today: 3 * avg_3_leaky - lag_1 - lag_2. The honest version shifts first, then rolls, and lands at 7.70, which is what this much noise allows.

The same bug shows up as rolling(..., center=True), a backfill (bfill) that copies tomorrow's value into a gap today, normalising by the mean of the whole series, or joining a lookup table as it is today rather than as it was on the prediction date.

4. Group leakage: the same customer on both sides of the split

Many datasets have several rows per real-world thing: visits per patient, orders per customer, photos per product, readings per device. Rows from the same entity look alike. If they're split at random, the model can score well by recognising the entity instead of learning the pattern.

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GroupKFold, KFold, cross_val_score
 
# 100 patients, 10 scans each. Every patient has their own baseline,
# and the diagnosis is only weakly related to it.
rng = np.random.default_rng(0)
n_patients, scans = 100, 10
baseline = rng.normal(size=(n_patients, 8))
diagnosis = (baseline[:, 0] + rng.normal(0, 1.5, n_patients) > 0).astype(int)
 
patient_id = np.repeat(np.arange(n_patients), scans)
X = baseline[patient_id] + rng.normal(0, 0.3, size=(n_patients * scans, 8))
y = diagnosis[patient_id]
 
model = RandomForestClassifier(random_state=0)
by_row = cross_val_score(model, X, y, cv=KFold(5, shuffle=True, random_state=0))
by_patient = cross_val_score(model, X, y, cv=GroupKFold(5), groups=patient_id)
 
print(f"KFold (rows mixed)       accuracy: {by_row.mean():.3f}")
print(f"GroupKFold (by patient)  accuracy: {by_patient.mean():.3f}")

KFold (rows mixed)       accuracy: 0.935
GroupKFold (by patient)  accuracy: 0.714

With a row-level split, about 8 of each patient's 10 scans sit in training when the other 2 are tested. The forest has seen this patient's diagnosis and finds the near-identical scan, which gives 0.935. GroupKFold keeps each patient entirely on one side, and the score drops to 0.714. That's the number to quote, because a deployed model only ever meets new patients.

The question to ask is "what is the unit I'll be predicting for?" If it's a new customer, split by customer. scikit-learn has GroupKFold, StratifiedGroupKFold and GroupShuffleSplit for this. If the data is both grouped and dated, hold out the latest period and make sure its groups weren't seen in training.

5. Duplicates and near-duplicates across splits

The last one is the least glamorous: the same record appears more than once, and copies end up in both train and test. Exact duplicates are easy to drop. Near-duplicates are the problem, because drop_duplicates() doesn't see them.

import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import GroupShuffleSplit, train_test_split
from sklearn.neighbors import KNeighborsClassifier
 
X, y = make_classification(n_samples=1000, n_features=10, flip_y=0.3, random_state=0)
 
# Each record gets scraped 3 times, with tiny differences (rounding, re-encoding)
rng = np.random.default_rng(0)
source = np.repeat(np.arange(1000), 3)
X3 = X[source] + rng.normal(0, 0.001, size=(3000, 10))
y3 = y[source]
print("exact duplicate rows found:", pd.DataFrame(X3).duplicated().sum())
 
knn = KNeighborsClassifier(n_neighbors=1)
X_tr, X_te, y_tr, y_te = train_test_split(X3, y3, random_state=0)
print(f"random split accuracy:     {knn.fit(X_tr, y_tr).score(X_te, y_te):.3f}")
 
tr, te = next(GroupShuffleSplit(test_size=0.25, random_state=0).split(X3, y3, source))
print(f"split by source accuracy:  {knn.fit(X3[tr], y3[tr]).score(X3[te], y3[te]):.3f}")

exact duplicate rows found: 0
random split accuracy:     0.992
split by source accuracy:  0.704

Zero exact duplicates, yet a random split scores 0.992, because almost every test row has a near-twin in training and a one-nearest-neighbour model simply copies its label. Splitting by source record gives 0.704. A 1-NN model is the extreme case, but any flexible model memorises to some degree, and the more it memorises, the more duplicates flatter it.

Near-duplicates come from the same item ingested twice, resized or re-compressed images, reposted text, re-submitted forms, and data augmentation run before the split. Deduplicate on a normalised key (rounded numbers, lower-cased and whitespace-stripped text, a perceptual hash for images) before any split. If you can't deduplicate, assign a source ID and split by it, exactly as with groups.

How to catch leakage before your users do

Leakage doesn't throw errors, so you have to go looking for it. These are the checks I'd run on any model before trusting its score.

A quick way to find a dominant column is to score every column on its own with a tiny model. Honest features are usually weak alone. A feature that nearly matches the full model by itself is either a great discovery or a leak, and it's usually a leak.

import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier
 
# Same churn data as before, with the refund column still in it
X, churned = make_classification(n_samples=5000, n_features=5, n_informative=3,
                                 flip_y=0.3, random_state=0)
rng = np.random.default_rng(0)
df = pd.DataFrame(X, columns=[f"f{i}" for i in range(5)])
df["refund_issued"] = ((churned == 1) & (rng.random(5000) < 0.9)).astype(int)
 
# Score every column on its own with a tiny model. Anything close to the
# full model's score deserves a hard look at when that value becomes known.
stump = DecisionTreeClassifier(max_depth=2, random_state=0)
for col in df.columns:
    auc = cross_val_score(stump, df[[col]], churned, cv=5, scoring="roc_auc").mean()
    print(f"{col:14s} AUC on its own: {auc:.3f}")

f0             AUC on its own: 0.609
f1             AUC on its own: 0.497
f2             AUC on its own: 0.588
f3             AUC on its own: 0.574
f4             AUC on its own: 0.679
refund_issued  AUC on its own: 0.951

The honest features sit between 0.497 and 0.679. refund_issued scores 0.951 by itself. That doesn't prove it's a leak, but it tells you which column to ask about first.

How to structure code so it can't happen

Checks catch leaks after they happen. Structure stops most of them from happening. The order below is the one I'd use by default:

In code, with a little missing data so the imputer has work to do:

import numpy as np
from sklearn.datasets import make_classification
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
 
X, y = make_classification(n_samples=3000, n_features=40, n_informative=8, random_state=0)
X[np.random.default_rng(0).random(X.shape) < 0.05] = np.nan  # 5% missing
 
# 1. Split first. The holdout is not touched until the very end.
X_dev, X_hold, y_dev, y_hold = train_test_split(X, y, test_size=0.2, stratify=y,
                                                random_state=0)
 
# 2. Every step that learns from data lives inside the pipeline.
pipe = make_pipeline(SimpleImputer(strategy="median"), StandardScaler(),
                     SelectKBest(f_classif, k=10), LogisticRegression())
 
# 3. Cross-validate on the development set only.
cv = StratifiedKFold(5, shuffle=True, random_state=0)
print(f"CV accuracy:      {cross_val_score(pipe, X_dev, y_dev, cv=cv).mean():.3f}")
 
# 4. Fit once on all development data, score the holdout once.
print(f"holdout accuracy: {pipe.fit(X_dev, y_dev).score(X_hold, y_hold):.3f}")

CV accuracy:      0.915
holdout accuracy: 0.925

Cross-validation and the holdout agree to within a point. That agreement is what a leak-free setup looks like. This data has no time order or groups, so a shuffled stratified split is fine here. With dated rows, the holdout becomes the latest period and the splitter becomes TimeSeriesSplit; with repeated entities, it becomes GroupKFold.

For mixed column types, a ColumnTransformer inside the pipeline keeps the same guarantee. Hyperparameter search belongs inside it too: GridSearchCV over the whole pipeline, never tuning against the holdout. Every time you look at the holdout and change something, it leaks a little more and becomes part of training.

The habits side by side:

What I'd take away

All five leaks are the same question answered wrongly: at the moment this prediction is made, what do I actually know, and about whom? Preprocessing leakage knows about the test rows. Target leakage knows the outcome. Temporal leakage knows the future. Group leakage and duplicates know the answer for this exact entity already.

If I were starting a new model tomorrow, I'd write two things down before any feature code: the prediction moment (and the as-of time it implies for every column), and the unit the model will meet in production. Those two decide the split, the splitter and the feature logic, and once they're written down most leaks become visible in review.

And I'd treat a surprisingly good score the way I'd treat a surprisingly fast benchmark: as something to explain before celebrating. The honest numbers in this post (0.480, 0.714, 0.704, 7.70) are less exciting than the leaky ones. They're also the only ones that would survive contact with production.