How to Tell When a Model Is Wrong

Hold some examples back with train_test_split and a fixed random_state, measure the model's guesses on them with the mean absolute error, and see why a score on the training data misleads and when a model's guesses shouldn't be trusted at all.

Apprentice · Python · AI Basics · Updated

By

After this lesson you can

  • Hold back test data with train_test_split and a fixed random_state, and explain why a score on the training data misleads
  • Measure a model's guesses with the mean absolute error, and say when a model's guesses shouldn't be trusted

Key takeaways

  • Scoring a model on the examples it learned from is grading your own homework. Score it on examples it never saw.
  • train_test_split holds back a share of the examples (test_size=0.2 holds back a fifth), and random_state fixes the shuffle, so every run splits the same way.
  • The mean absolute error is how far off a guess is on average, in the label's own units, such as 8.1 cups.
  • A model that memorizes its examples (overfitting) scores well on them and worse on new ones. Only held-back examples show it.
  • Don't trust a model's guesses far outside the examples it learned from, or after the world it learned from has changed.

Before you buy 100 lemons on a model's say-so, you want to know how far off its guesses tend to be. Every model you'll meet, from a store's sales forecast to a spam filter, should come with a number like that. This lesson shows where it comes from, how it can fool you, and when no number makes a model safe to use.

The example is the Sunny Days Lemonade Stand's summer of 92 days, and the straight-line model from A Model That Guesses, which guesses the cups sold from the temperature.

Grading your own homework

The obvious check is to have the model guess the 92 days it learned from and compare. A student who memorized last year's answer key scores 100% on last year's test and might know nothing about this year's. A model scored on its own examples is the same: it has already seen the answers.

The fix is to hold some days back before the model learns anything. The model learns from the training data, and it's scored on the test data: days it has never seen, standing in for the days to come.

Holding days back with train_test_split

scikit-learn's train_test_split shuffles the examples and splits them in two:

import pandas as pd
from sklearn.model_selection import train_test_split

days = pd.read_csv("lemonade_days.csv")
X = days[["temp_f"]]
y = days["cups_sold"]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
print(len(X_train), "days to learn from")
print(len(X_test), "days held back")
print(X_test.head(3))
73 days to learn from
19 days held back
    temp_f
40    87.7
22    86.5
55    87.0

test_size=0.2 holds back a fifth of the days: a fifth of 92 is 18.4, and scikit-learn rounds the test part up to 19, leaving 73. The four parts come back in the order written: training features, test features, training labels, test labels. The days are shuffled first, so the held-back ones (rows 40, 22 and 55 first) come from all over the summer.

random_state=42 is the shuffle's seed, the number it starts from. The same seed gives the same shuffle every run, so your split matches this lesson's and a test can check it; leave it out and each run splits differently. Any whole number works; 42 is only a habit.

How wrong, in cups: the mean absolute error

How far off is one guess? Take the distance between the guess and what really sold, always counted as a positive number. For the summer's first three days, the model from A Model That Guesses guessed 62, 91 and 80 cups, and the stand sold 52, 87 and 78. Python's abs() (read "absolute value") drops the minus sign:

misses = [abs(52 - 62), abs(87 - 91), abs(78 - 80)]
print(misses)
print(round(sum(misses) / len(misses), 1))
[10, 4, 2]
5.3

The average of the misses is the mean absolute error. Why not average the plain differences? A guess 10 too high and one 10 too low would cancel out to 0, a perfect score for two bad guesses. scikit-learn's mean_absolute_error(actual, guesses) does the same arithmetic (0.0 is perfect). Here it is on both halves of the split:

import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split

days = pd.read_csv("lemonade_days.csv")
X_train, X_test, y_train, y_test = train_test_split(
    days[["temp_f"]], days["cups_sold"], test_size=0.2, random_state=42
)
model = LinearRegression()
model.fit(X_train, y_train)

print("Off on the days it learned from:", round(mean_absolute_error(y_train, model.predict(X_train)), 1))
print("Off on the held-back days:", round(mean_absolute_error(y_test, model.predict(X_test)), 1))
Off on the days it learned from: 6.6
Off on the held-back days: 8.1

In plain words: on days it never saw, a guess is off by about 8 cups on average, some days more and some less. The score on its own days looked better than that.

A model that memorizes

A different kind of model, a decision tree with no limits, can carve the training days into groups so small that it nearly memorizes each one: call it the memorizer. Swap it in (import DecisionTreeRegressor from sklearn.tree, and use DecisionTreeRegressor(random_state=0) in place of LinearRegression()) and the same two lines print:

Off on the days it learned from: 1.3
Off on the held-back days: 9.6

Nearly perfect on its own days, and worse than the straight line on new ones. It learned each day's random wobble instead of the pattern. That's overfitting, and only the held-back days reveal it.

Two report cards side by side. The straight line: off by 6.6 cups on the days it learned from and 8.1 on the held-back days. The memorizer: off by only 1.3 cups on the days it learned from, but 9.6 on the held-back days, the worst score of the four

When not to trust a model

A good score on held-back days still isn't a promise:

  • Far outside its examples. The coldest day in the data was 64.1°F. Ask the model from A Model That Guesses about a 30°F day and it guesses about −14 cups. The line keeps going where no example was; a model only knows the range it learned from.
  • When the world changes. A new spot by the pool or a higher price makes last summer a poor guide. A model learns what was, not what will be. When things change over time, hold back the latest days instead of a shuffled few, so the test looks like the future.
  • When the reason matters. The model found that hot days sell more; it can't say why. Maybe hot days are also weekends. It learned a pattern in the data, not how the world works.

That's the honest core of machine learning. Machine Learning with Python goes further: models that look at many features, models that guess a category such as spam or not spam, more careful ways to score them, and explaining a model's limits to someone who isn't technical.

Check yourself

2 questions to try on the lesson page.

So what

A model's score means something only when it comes from examples the model never saw, in units you can plan with, and even then it holds only for days like the ones it learned from.

Sources

  • scikit-learn 1.9.1, train_test_split, API reference: test_size as a share of the data (0.25 when no size is given), shuffle on by default, and random_state for the same split on every call.
  • scikit-learn, Cross-validation: evaluating estimator performance, User Guide, its opening: testing on the data a model learned from is a mistake, a model that repeats the labels it has seen would score perfectly (overfitting), and the fix is to hold out a test set.
  • scikit-learn 1.9.1, mean_absolute_error, API reference: the mean absolute error regression loss, never negative, best value 0.0.
  • scikit-learn, Glossary of Common Terms and API Elements, random_state: an integer gives the same results across calls.
  • Python documentation, abs().

Key terms

Related