Overfitting and Underfitting
A model that memorizes its training rows scores almost perfectly on them and much worse on rows it never saw (overfitting), a model too simple for the pattern scores low on both (underfitting), and putting the training score beside the test score is how you tell them apart.
Journeyman · AI Basics · Python · Updated
After this lesson you can
- Tell overfitting from underfitting by putting a model's training score beside its test score
- Explain bias and variance in plain words, and choose a fix for overfitting such as more data, a simpler model or a limit like max_depth
Key takeaways
- Overfitting is a model learning its training rows' random wobble along with the pattern. Its training score is high and its test score much lower.
- Underfitting is a model too simple for the pattern. Both scores are low, and close together.
- A tree with no depth limit scored a training R squared of 1.0 on Lakeside Cinema's showings and 0.54 on showings it never saw.
- Your textbook calls the two problems bias (a model too simple to follow the pattern) and variance (a model that changes a lot with the rows it learned from).
- The usual fixes for overfitting are more data, a simpler model, or a limit such as max_depth or min_samples_leaf.
Lakeside Cinema's new model scored perfectly on the showings it learned from. A week later its guesses for real showings are off by twenty tickets or more. That's one of the most common ways a machine learning project goes wrong: a model that looks great while you build it and lets you down in use. You'll check for it every time you train a model, in a class project or at work, and it takes two numbers side by side.
The example for the whole chapter is Lakeside Cinema: 900 showings in showings.csv, our own made-up data, in date order, with the tickets each one sold. It comes with Exercise: Honest Scores, beside the seed script that made it. Each code block is one notebook cell, run in order: the first one loads the data, and the others use what it made.
The memorizer and the student who never studied
Picture two students before a test. One memorized last year's answer key, word for word. On last year's questions they score 100%; on this year's, the memorized answers don't fit and they do badly. The other never studied at all, and does badly on both tests.
Models fail the same two ways. A model that learns its training rows so closely that it learns their random wobble too has overfit: a high score on the rows it learned from and a much lower one on new rows. A model too simple to follow the pattern has underfit: low scores on both. The one you want is in between, like the student who learned the ideas: a little worse on the practice questions than the memorizer, and much better on the real test.
Three trees, two scores each
The cinema guesses tickets_sold from five number columns. (genre is text, which these models can't use yet; Turning Categories into Numbers comes in the next chapter.) The split holds back 180 showings as test data:
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeRegressor
showings = pd.read_csv("showings.csv")
features = ["day_number", "weekday", "start_hour", "is_new_release", "screen_seats"]
X = showings[features]
y = showings["tickets_sold"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
print("Training showings:", len(X_train))
print("Test showings:", len(X_test))
Training showings: 720
Test showings: 180
Now three regression trees: one allowed a single question, one five questions deep, and one with no limit. A model's score method gives its R squared, read "R squared" and written R², from R Squared: Better Than Guessing the Average?: 1 is perfect, 0 is no better than guessing the average.
for depth in [1, 5, None]:
tree = DecisionTreeRegressor(max_depth=depth, random_state=0)
tree.fit(X_train, y_train)
print("max_depth", depth, "| training R²", round(tree.score(X_train, y_train), 2),
"| test R²", round(tree.score(X_test, y_test), 2), "| leaves", tree.get_n_leaves())
max_depth 1 | training R² 0.34 | test R² 0.29 | leaves 2
max_depth 5 | training R² 0.77 | test R² 0.69 | leaves 32
max_depth None | training R² 1.0 | test R² 0.54 | leaves 651

Reading the two scores
- One question deep: 0.34 and 0.29. Both low, close together: underfitting. Two leaves can't hold what a new release, an evening start and a Friday each add.
- No limit: a perfect 1.0 in training, from 651 leaves for 720 showings, nearly one leaf per showing. On new showings, 0.54. It memorized: overfitting.
- Five deep: 0.77 and 0.69. A small gap is normal, since new rows are a little harder than ones a model learned from, and its test score is the best of the three.
The same story in tickets, with the mean absolute error from Module 2:
from sklearn.metrics import mean_absolute_error
for depth in [1, 5, None]:
tree = DecisionTreeRegressor(max_depth=depth, random_state=0).fit(X_train, y_train)
print("max_depth", depth, "| training MAE", round(mean_absolute_error(y_train, tree.predict(X_train)), 1),
"| test MAE", round(mean_absolute_error(y_test, tree.predict(X_test)), 1))
max_depth 1 | training MAE 25.6 | test MAE 29.4
max_depth 5 | training MAE 14.2 | test MAE 18.1
max_depth None | training MAE 0.0 | test MAE 22.6
The memorizer misses its own showings by 0.0 tickets and new ones by 22.6. A training score alone would have picked it.
Bias and variance, in your textbook
Your textbook names the two problems. Bias is a model's error from being too simple: whatever rows it learns from, it misses in the same way, like the one-question tree. Variance is how much a model changes with the rows it happens to learn from: a tree with no limit can grow a very different set of leaves from another 720 showings. Underfitting is high bias; overfitting is high variance. A more flexible model usually lowers bias and raises variance, so you look for the settings where both are as low as you can get them: the bias-variance tradeoff. The rest of a model's error, the showings' own random wobble, is noise, which no model can remove.
Fixing overfitting
Three fixes, from the least work to the most:
- A limit:
max_depth, ormin_samples_leaf, the fewest training rows a leaf may hold. - A simpler model: fewer features, or a less flexible kind of model.
- More data: more showings make each leaf's average less of a coin toss.
for leaf in [1, 20]:
tree = DecisionTreeRegressor(min_samples_leaf=leaf, random_state=0).fit(X_train, y_train)
print("min_samples_leaf", leaf, "| training R²", round(tree.score(X_train, y_train), 2),
"| test R²", round(tree.score(X_test, y_test), 2), "| leaves", tree.get_n_leaves())
min_samples_leaf 1 | training R² 1.0 | test R² 0.54 | leaves 651
min_samples_leaf 20 | training R² 0.77 | test R² 0.7 | leaves 26
Twenty showings in every leaf, and the gap nearly closes. Underfitting is fixed the other way: a more flexible model, or features that carry the pattern. One more thing to keep in mind: these showings are in date order, and the split shuffled them. Splits for Time and Groups comes back to what that hides.
Check yourself
2 questions to try on the lesson page.
So what
Put the training score beside the test score every time: two low scores say underfitting, a high one beside a much lower one says overfitting, and a limit, a simpler model or more data closes the gap.
Sources
- scikit-learn, Cross-validation: evaluating estimator performance, User Guide, its opening: a model that repeats the labels of the samples it has seen would score perfectly and fail on unseen data, a situation called overfitting.
- scikit-learn, Validation curves: plotting scores to evaluate models, User Guide: an estimator's generalization error can be split into bias (its average error for different training sets), variance (how sensitive it is to varying training sets) and noise (a property of the data); both scores low means underfitting, a high training score and a low validation score means overfitting.
- scikit-learn, Validation curves: plotting scores to evaluate models, its polynomial example: a model too simple to fit has high bias, one that fits its training data perfectly but not the true function has high variance, bias and variance are chosen to be as low as possible, and more training data is another way to reduce variance.
- scikit-learn, Decision Trees, User Guide: trees can grow over-complex and overfit, and setting the maximum depth or the minimum number of samples at a leaf helps avoid it; a very small
min_samples_leafusually means the tree will overfit; small changes in the data can give a completely different tree. - scikit-learn 1.9.1, DecisionTreeRegressor, API reference:
max_depth,min_samples_leaf,get_n_leaves, andscore, which returns R².