Data Leakage
A score that's too good to be true usually is. Data leakage is information from the answer, or from the test data, sneaking into training: a column only known after the outcome, a scaler fit on all the data before the split, the same customer on both sides. Learn to spot each kind and to close it with the habits this module built. The running example is a streaming service predicting which subscribers will cancel.
Updated
What Leakage Is
- What Data Leakage Is (Journeyman): Data leakage is information the model won't have when it's really used, such as a column only known after the outcome or facts about the test rows, slipping into what it learns from, so its score comes out better than it will ever do on new data, and a score that looks too good to be true is the warning sign.
Where Leaks Hide
- Leaks in Preparing the Data (Freelancer): A preparation step fit on the whole table before the split, a scaler, an imputer or a step that chooses columns, learns from the rows it will be tested on, which barely moved StreamBox's score for a scaler but turned a worse model into one that looked better when columns were chosen against the label, and putting every step that learns in a pipeline that cross-validation fits inside each fold closes the leak.
- Leaky Features and Duplicate Rows (Freelancer): Asking "when is this known?" of every column finds the ones filled in only after the outcome, and a table that lists the same subscriber twice puts them on both sides of a row-by-row split, where a tree recognized them and scored 0.843 instead of an honest 0.814, so leaky columns are dropped and the rows are split by subscriber with GroupKFold or GroupShuffleSplit.
Exercise: Plug the Leak
- Exercise: Plug the Leak (Freelancer): Write three functions that close the leaks in StreamBox's 2,000 subscriber rows, dropping the column known only after a cancellation, splitting so no subscriber is on both sides, and cross-validating a scaled model with the scaler inside a pipeline, and let pytest check that nothing leaks.