What Data Leakage Is
Data leakage is information the model won't have when it's really used, such as a column only known after the outcome or facts about the test rows, slipping into what it learns from, so its score comes out better than it will ever do on new data, and a score that looks too good to be true is the warning sign.
Journeyman · AI Basics · Python · Updated
After this lesson you can
- Explain what data leakage is, and why a score that looks too good to be true is its warning sign
- Sort a leak into a leak from the label or a leak from the test data, and name the common forms of each
Key takeaways
- Data leakage is information the model won't have when it's really used getting into what it learns from. Its score then looks better than the model will ever do on new data.
- A score that looks too good to be true on a hard question is the warning sign. Find out what the model leans on before you trust it.
- On StreamBox's subscribers, a tree scored 0.993 with exit_survey_sent, a column filled in only after a cancellation, and 0.87 without it, against 0.795 for always guessing "stays".
- Leaks come from the label (a column known only after the outcome) or from the test data (preparing the whole table first, one subscriber on both sides, or shuffled time order).
StreamBox, a streaming service, wants to offer a discount to the subscribers who are about to cancel, before they go. An analyst trains a model, tests it on subscribers it never saw, and it guesses right 99.3% of the time. Before anyone builds a campaign on that number, someone has to ask why it's so good. A model that looks great in testing and then fails in real use is one of the most common mistakes in machine learning, in class projects and in jobs alike, and this chapter teaches you to catch it.
The example for the whole chapter is StreamBox's subscribers.csv: 2,000 rows for 1,700 subscribers, our own made-up data. It comes with Exercise: Plug the Leak, beside the seed script that made it. Each code block is one notebook cell, run in order: the first one loads the data, and the others use what it made.
A score too good to be true
The analyst's model is a small decision tree, three questions deep, trained on every number column and tested on a fifth of the rows held back:
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
subscribers = pd.read_csv("subscribers.csv")
X = subscribers.drop(columns=["subscriber_id", "plan", "canceled"])
y = subscribers["canceled"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
model = DecisionTreeClassifier(max_depth=3, random_state=0)
model.fit(X_train, y_train)
print("Features:", list(X.columns))
print("Test accuracy:", round(model.score(X_test, y_test), 3))
Features: ['months_subscribed', 'hours_last_month', 'profiles', 'support_tickets', 'exit_survey_sent']
Test accuracy: 0.993
The split is honest: the test rows were held back before the tree learned anything. Still, 0.993 is suspicious. People cancel for reasons a table can't see (a move, a tighter budget, a show that ended), and guessing who will cancel from viewing hours and support tickets is hard. When a score on a hard question looks too good to be true, treat it as a clue, not a result.
What the model leans on
A tree can show its questions. export_text prints them:
from sklearn.tree import export_text
print(export_text(model, feature_names=list(X.columns)))
|--- exit_survey_sent <= 0.50
| |--- support_tickets <= 2.50
| | |--- hours_last_month <= 9.45
| | | |--- class: 0
| | |--- hours_last_month > 9.45
| | | |--- class: 0
| |--- support_tickets > 2.50
| | |--- hours_last_month <= 22.75
| | | |--- class: 0
| | |--- hours_last_month > 22.75
| | | |--- class: 0
|--- exit_survey_sent > 0.50
| |--- class: 1
The first question decides it: every subscriber who got the exit survey is guessed "canceled" (class 1), and everyone else "stays". A count of the two columns side by side shows why:
pd.crosstab(subscribers["exit_survey_sent"], subscribers["canceled"])
canceled 0 1
exit_survey_sent
0 1590 30
1 0 380
All 380 subscribers who got the survey canceled; nobody who stayed got one. StreamBox emails its exit survey after a subscriber cancels. So the column is the answer written another way, and at the moment StreamBox wants to use the model, before anyone has canceled, it's 0 for everyone.
What data leakage is
A student who saw the answer key before the exam scores 100%, and the score says nothing about what they know; next time there's no key. A model can see an answer key too. Data leakage is information the model won't have when it's really used getting into what it learns from. scikit-learn's own definition: information that would not be available at prediction time is used when building the model, which gives overly optimistic scores and poorer results on new data.
Here is the honest version, with only the columns StreamBox knows before a cancellation:
known_now = ["months_subscribed", "hours_last_month", "profiles", "support_tickets"]
model.fit(X_train[known_now], y_train)
print("Test accuracy:", round(model.score(X_test[known_now], y_test), 3))
print("Always guessing 'stays':", round((y_test == 0).mean(), 3))
Test accuracy: 0.87
Always guessing 'stays': 0.795
0.87 against a baseline of 0.795, the score of always guessing that nobody cancels. Lower than 0.993, and worth far more, because it's a number the model can earn in real use. (This table hides two more problems, which the next two lessons find.)
Two families of leaks
Every leak gets in one of two ways:
- From the label. A column carries the answer: it's filled in only after the outcome, like
exit_survey_sent, or it stands in for the outcome, like an account number that changes when someone cancels. Many books call this target leakage, because the target (scikit-learn's word for the label) leaks into the features. - From the test data. Facts about the rows you test on reach training, so the test is no longer a stranger. You may see it called train–test contamination. Three common forms: preparing the data (scaling, filling blanks, choosing columns) on the whole table before the split, which Leaks in Preparing the Data shows; the same subscriber on both sides of the split, in Leaky Features and Duplicate Rows; and shuffling rows that are in time order, so the model learns from the future, which Splits for Time and Groups showed with a tree that scored 0.72 on shuffled folds and 0.59 on past-then-future ones.
The warning signs are the same for both: a score far above the baseline on a hard question, one column doing nearly all the work, or a model that scores well in testing and disappoints in use.
Check yourself
2 questions to try on the lesson page.
So what
A score is only worth what the model will know when it's used, so when a score looks too good to be true, find the leak before you trust it.
Sources
- scikit-learn, Common pitfalls and recommended practices, Data leakage: leakage is information that would not be available at prediction time being used when building the model, which gives overly optimistic performance estimates and poorer performance on new data; a common cause is not keeping the training and test data separate.
- scikit-learn, Glossary of Common Terms and API Elements, data leakage: generalization performance over-estimated because knowledge of the test data was included in training, for instance by fitting a transformer on the whole dataset.
- scikit-learn, Cross-validation: evaluating estimator performance, User Guide, Cross validation of time series data and Cross-validation iterators for grouped data: shuffled splits assume independent rows, which time-ordered data and several rows per person break.
- Shachar Kaufman, Saharon Rosset, Claudia Perlich and Ori Stitelman, Leakage in Data Mining: Formulation, Detection, and Avoidance, ACM Transactions on Knowledge Discovery from Data 6(4), 2012 (doi:10.1145/2382577.2382579): leakage as information about the target that the model shouldn't legitimately have, in competitions and real projects alike.
- scikit-learn 1.9.1, export_text, API reference: a text report of a decision tree's rules.