Where Leaks Hide
Where Leaks Hide: Leaks in Preparing the Data, Leaky Features and Duplicate Rows.
Updated
- Leaks in Preparing the Data (Freelancer): A preparation step fit on the whole table before the split, a scaler, an imputer or a step that chooses columns, learns from the rows it will be tested on, which barely moved StreamBox's score for a scaler but turned a worse model into one that looked better when columns were chosen against the label, and putting every step that learns in a pipeline that cross-validation fits inside each fold closes the leak.
- Leaky Features and Duplicate Rows (Freelancer): Asking "when is this known?" of every column finds the ones filled in only after the outcome, and a table that lists the same subscriber twice puts them on both sides of a row-by-row split, where a tree recognized them and scored 0.843 instead of an honest 0.814, so leaky columns are dropped and the rows are split by subscriber with GroupKFold or GroupShuffleSplit.