Overfitting and Cross-Validation
A model can memorize its examples instead of learning from them, or be too simple to learn anything. See both on a curve, then get a score you can trust: cross-validation tests a model on every part of the data in turn, a separate validation set keeps the test set for one last check, and data in time order or in groups needs splits of its own. The running example is a cinema's ticket sales.
Updated
Memorizing vs Learning
- Overfitting and Underfitting (Journeyman): A model that memorizes its training rows scores almost perfectly on them and much worse on rows it never saw (overfitting), a model too simple for the pattern scores low on both (underfitting), and putting the training score beside the test score is how you tell them apart.
- Seeing It: Validation and Learning Curves (Freelancer): A validation curve draws a model's training score and validation score for each value of one setting, such as a tree's max_depth, so you can see where underfitting ends and overfitting begins, and a learning curve draws the same two scores for more and more training rows, to show whether more data would help.
Cross-Validation
- Cross-Validation (Journeyman): Cross-validation splits the rows into k folds and tests the model on each fold in turn, learning from the others, so every row is tested exactly once; cross_val_score gives the k scores, you report their mean and their spread, and KFold with shuffle=True and a random_state makes the folds random and repeatable.
- Validation Sets: Keep the Test Set for Last (Journeyman): Every time you look at the test score while choosing a model or a setting, the choice learns a little about the test rows, so choose with a validation set or with cross-validation on the training part, and score the test set once, at the end, for the model you chose.
- Splits for Time and Groups (Freelancer): Shuffled folds assume every row is independent of the others, which isn't true for data in time order, where shuffling lets a model learn from the future, or for data with repeats, where the same customer lands on both sides; TimeSeriesSplit always tests on rows that come after the ones it learned from, and GroupKFold keeps each group on one side.
Exercise: Honest Scores
- Exercise: Honest Scores (Journeyman): Write three functions that score a model with shuffled, seeded cross-validation, put its training score beside its validation score, and choose a tree's depth by cross-validation without ever touching the test showings, on Lakeside Cinema's 900 showings, and let pytest check them.