Leaks in Preparing the Data

A preparation step fit on the whole table before the split, a scaler, an imputer or a step that chooses columns, learns from the rows it will be tested on, which barely moved StreamBox's score for a scaler but turned a worse model into one that looked better when columns were chosen against the label, and putting every step that learns in a pipeline that cross-validation fits inside each fold closes the leak.

Freelancer · AI Basics · Python · Updated

By

After this lesson you can

  • Explain how a scaler, an imputer or a column-choosing step fit before the split leaks the test rows into training
  • Keep every learned step inside a pipeline that cross_val_score fits inside each fold, and compare the leaky score with the honest one

Key takeaways

  • Any step that learns from data, a scaler, an imputer or a step that chooses columns, leaks when it's fit on the whole table before the split or before cross-validation.
  • A scaler fit on all of StreamBox's rows scored the same as an honest one, 0.8735, as its averages hardly moved. A small leak is still a leak, and you can't tell ahead which are small.
  • Choosing 50 columns out of 1,004 on the whole table scored 0.883, better than the 0.874 of four real columns. Chosen inside each fold, the same plan scored 0.852, worse.
  • Put every step that learns in a pipeline, and give the pipeline to cross-validation, so each fold fits every step on its training rows only.

Almost every model gets its data prepared first: numbers rescaled, blanks filled in, the most useful columns picked out. It's tempting to do that once, to the whole table, and then split or cross-validate. That order is a leak, and it's one of the easiest to make by accident, because each step looks like tidying up rather than learning. You'll use what this lesson shows every time a model has a preparation step, which is almost every time.

The example is StreamBox's subscribers.csv, from Exercise: Plug the Leak. Each code block is one notebook cell, run in order: the first one loads the data. Every score here comes from five folds that keep each subscriber's rows together (GroupKFold, from Splits for Time and Groups), because some subscribers are listed twice; Leaky Features and Duplicate Rows shows why that matters.

Every step that learns

A scaler learns something: the average and the spread of each column, which it then uses to rescale every row (Scaling Features). An imputer learns the value it fills blanks with. A step that chooses columns learns which ones line up best with the label. Fit any of them on the whole table, and the test rows help decide what it learns. The rule from scikit-learn is short: never call fit on the test data, for the model or for any step before it.

Here is the leaky order, scaling all 2,000 rows and then cross-validating:

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GroupKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

subscribers = pd.read_csv("subscribers.csv")
features = ["months_subscribed", "hours_last_month", "profiles", "support_tickets"]
X = subscribers[features]
y = subscribers["canceled"]
groups = subscribers["subscriber_id"]
folds = GroupKFold(n_splits=5, shuffle=True, random_state=0)

X_scaled = StandardScaler().fit_transform(X)
scaled_first = cross_val_score(LogisticRegression(), X_scaled, y, cv=folds, groups=groups)
print("Scaled before the split:", round(scaled_first.mean(), 4))
Scaled before the split: 0.8735

shuffle=True with random_state=0 mixes the subscribers before they're dealt into folds, the same way on every run. And the honest order, with the scaler inside a pipeline, so each fold fits it on that fold's training rows:

model = make_pipeline(StandardScaler(), LogisticRegression())
scaled_inside = cross_val_score(model, X, y, cv=folds, groups=groups)
print("Scaled inside each fold:", round(scaled_inside.mean(), 4))
Scaled inside each fold: 0.8735

A small leak is still a leak

The same score, to four places. That's honest to report: here, the leak changed nothing you can see, because the averages hardly move when a fifth of the rows is added:

fit_rows, check_rows = next(folds.split(X, y, groups))
print("Averages from all 2,000 rows:", StandardScaler().fit(X).mean_.round(2))
print("Averages from one fold's training rows:", StandardScaler().fit(X.iloc[fit_rows]).mean_.round(2))
Averages from all 2,000 rows: [24.34 21.44  2.36  0.88]
Averages from one fold's training rows: [24.34 21.47  2.37  0.88]

So why bother? Because a scaler is the mildest step there is. It only looks at the features. A step that looks at the label leaks far more, and from the outside you can't tell a mild leak from a big one.

Choosing columns on the whole table

Picking the most useful columns is called feature selection. scikit-learn's SelectKBest(k=50) scores each column by how strongly it lines up with the label and keeps the best 50. To see what it does with a leak, give StreamBox's four real columns 1,000 extra yes-or-no columns that are pure coin flips: we made them up with a seeded random generator, so we know they carry nothing about canceling.

import numpy as np
from sklearn.feature_selection import SelectKBest

rng = np.random.default_rng(0)
flags = pd.DataFrame(rng.integers(0, 2, size=(len(subscribers), 1000)),
                     columns=[f"flag_{n}" for n in range(1, 1001)])
X_wide = pd.concat([X, flags], axis=1)

chosen_first = SelectKBest(k=50).fit_transform(X_wide, y)
leaky = cross_val_score(model, chosen_first, y, cv=folds, groups=groups)
print("Columns to choose from:", X_wide.shape[1])
print("Four real columns:", round(scaled_inside.mean(), 3))
print("50 columns chosen before the split:", round(leaky.mean(), 3))
Columns to choose from: 1004
Four real columns: 0.874
50 columns chosen before the split: 0.883

The leaky run says the coin flips helped. They can't have: among 1,000 random columns, some line up with canceling by luck, and choosing them with every row's label, the test rows' included, picks exactly the lucky ones for those test rows.

The fix: a pipeline inside cross-validation

Put the choosing step in the pipeline too:

honest = make_pipeline(SelectKBest(k=50), StandardScaler(), LogisticRegression())
chosen_inside = cross_val_score(honest, X_wide, y, cv=folds, groups=groups)
print("50 columns chosen inside each fold:", round(chosen_inside.mean(), 3))
50 columns chosen inside each fold: 0.852

0.852: worse than the four real columns alone. That's the truth the leak hid. As Pipelines: Every Step, Every Time showed, cross_val_score fits a fresh copy of the whole pipeline in every fold, on that fold's training rows only, and then scores it on the rows it held out. No step ever learns from the rows it's scored on.

Five folds of StreamBox's subscribers shown as five rows: in each row, four blocks of training subscribers feed one pipeline of three boxes, SelectKBest, then StandardScaler, then LogisticRegression, which is fit on those blocks only, and the fifth block, held out behind a fence, goes through the fitted pipeline to be scored; a different block is held out in each row

The same goes for filling blanks: a SimpleImputer fit on the whole table learns its median from the test rows. Every step that learns goes in the pipeline, and the pipeline goes into cross_val_score (or is fit on the training part only, then scored once on the test part).

Check yourself

2 questions to try on the lesson page.

So what

A preparation step learns from data like a model does, so it belongs inside the pipeline that cross-validation refits in every fold, where it can't peek at the rows it's scored on.

Sources

  • scikit-learn, Common pitfalls and recommended practices, Data leakage, How to avoid data leakage and Data leakage during pre-processing: split first, never include test data in fit or fit_transform, transformations are learned from the training data only, the risk applies to almost all transformations (StandardScaler, SimpleImputer and feature selection among them), and a pipeline, alone or inside cross_val_score, makes sure the right rows are used.
  • scikit-learn, Cross-validation: evaluating estimator performance, User Guide, Computing cross-validated metrics: preprocessing such as standardization and feature selection should be learned from a training set and applied to held-out data, which a Pipeline does under cross-validation.
  • scikit-learn, Pipelines and composite estimators, User Guide: a pipeline avoids leaking statistics from the test data into the trained model in cross-validation by making sure the same samples train the transformers and the predictor.
  • scikit-learn 1.9.1, SelectKBest, API reference: keeps the k features with the highest scores, scored by default with f_classif, which works for classification.
  • scikit-learn 1.9.1, GroupKFold, API reference: each group appears in the test set of exactly one fold; shuffle shuffles the groups before they're split, and random_state makes that repeatable (both added in version 1.6).

Key terms

Related