Errors and the Mean Absolute Error

A guess's error, or residual, is the actual value minus the guess, and because errors above and below cancel out when they're averaged, the mean absolute error averages their sizes instead, giving how far off the guesses usually are in the label's own units, by hand or with scikit-learn's mean_absolute_error.

Apprentice · AI Basics · Python · Updated

By

After this lesson you can

  • Compute a residual as the actual value minus the guess, and explain why plain errors can cancel out when they're averaged
  • Compute the mean absolute error by hand and with mean_absolute_error, and read it in the label's units

Key takeaways

  • An error, or residual, is the actual value minus the guess. Positive means the guess was too low, negative too high.
  • Averaging the plain errors lets a guess too high and a guess too low cancel out, so two bad guesses can average to 0.
  • The mean absolute error (MAE) averages each error's size, its absolute value, so nothing cancels.
  • MAE is in the label's own units. An MAE of 3.63 means the guesses are off by about 3.6 minutes on average.
  • mean_absolute_error(y_true, y_pred) takes the actual values first and the guesses second.

Splitting Data with train_test_split ended with three honest guesses for Maple Grove homes the model had never seen: $313,464 for a home that sold for $335,000, $283,153 for one that sold for $317,500, and $217,019 for one that sold for $228,000. Three misses can be read one by one. Eighty test homes, or a hundred test orders, aren't: you need one number that says how far off a model usually is. Every model you build, for a class, a project or a job, gets judged with a number like that, and you'll be the one asked to explain it.

This chapter's example is Pine Ridge Delivery, a food delivery service whose app tells each customer "your order arrives in about 31 minutes." Its 500 past orders are in deliveries.csv, our own made-up data, which comes with Exercise: Delivery Time Errors, beside the seed script that made it.

How far off was each guess?

A model trained on 400 of the orders guessed the minutes for the other 100, the test orders. Here are the first five, with each guess rounded to a whole minute:

actual = [40, 28, 37, 52, 30]
guessed = [41, 29, 31, 31, 33]

for a, g in zip(actual, guessed):
    print("took", a, "guessed", g, "error", a - g)
took 40 guessed 41 error -1
took 28 guessed 29 error -1
took 37 guessed 31 error 6
took 52 guessed 31 error 21
took 30 guessed 33 error -3

One guess's error is the actual value minus the guess. Its formal name is a residual, what's left over after the guess; machine learning writing, scikit-learn's documentation included, uses both words. A positive residual means the order took longer than the app said (the guess was too low); a negative one means it arrived early. The fourth order took 52 minutes when the app said 31: a customer who waited 21 minutes longer than promised.

Why plain errors cancel out

The first idea is to average the errors. Look what happens with four orders, two of them 10 and 8 minutes late, two of them 10 and 8 minutes early:

errors = [10, -10, 8, -8]
print("Average error:", sum(errors) / len(errors))
Average error: 0.0

An average of 0, a perfect score, for four guesses that were all badly off. A late order and an early one cancel each other out. The average of the plain errors says whether a model guesses too high or too low overall, which is worth knowing, but it can't say how big the misses are.

The mean absolute error

The fix is to ignore the direction and keep the size. A number's absolute value is its distance from 0: the absolute value of −3 is 3, and of 6 is 6. Python's abs() gives it. Average the sizes, and nothing cancels:

actual = [40, 28, 37, 52, 30]
guessed = [41, 29, 31, 31, 33]

total = 0
for a, g in zip(actual, guessed):
    total += abs(a - g)
print("Mean absolute error:", total / len(actual))
Mean absolute error: 6.4

The sizes are 1, 1, 6, 21 and 3, which add up to 32, and 32 ÷ 5 = 6.4. That's the mean absolute error, MAE for short: on these five orders, the guesses were off by 6.4 minutes on average. It's in the label's own units, minutes here, dollars for Maple Grove's prices, which makes it the easiest error number to explain to anyone.

In a textbook you'll see it written MAE = (1/n) Σ |yᵢ − ŷᵢ|, read "one over n, times the sum of the absolute value of y minus y hat": add up the size of each actual value minus its guess (ŷ, "y hat", is the guess), then divide by how many there are.

mean_absolute_error in scikit-learn

scikit-learn has the same calculation built in. Here's the whole model: split the orders, train on 400, guess for the other 100, and score the guesses:

import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split

deliveries = pd.read_csv("deliveries.csv")
features = ["distance_miles", "stops_before", "is_raining"]
train, test = train_test_split(deliveries, test_size=0.2, random_state=42)

model = LinearRegression()
model.fit(train[features], train["minutes"])
guesses = model.predict(test[features])

print("Test orders:", len(test))
print("Mean absolute error:", round(mean_absolute_error(test["minutes"], guesses), 2))
Test orders: 100
Mean absolute error: 3.63

mean_absolute_error(y_true, y_pred) takes the actual values first and the guesses second. On 100 orders it never saw, the model is off by about 3.6 minutes on average. The plain errors tell a different story:

errors = test["minutes"] - guesses
print("Average plain error:", round(errors.mean(), 2))
print("Average size of error:", round(errors.abs().mean(), 2))
Average plain error: 0.83
Average size of error: 3.63

The plain average, 0.83 minutes, makes the model look nearly perfect: early and late orders cancel. The average size, 3.63, is the MAE, the same number scikit-learn gave. A test set's MAE is the honest one to report: on the training orders, the model has already seen the answers.

Do it: pick the closer delivery estimate

Pine Ridge Delivery has an old rule for its estimates and a new model, and you want to know which one's guesses come closer on the same orders.

The inputs:

  • actual: how many minutes each order really took.
  • predicted: one estimate's guesses for the same orders, in the same order.
  • old and new: the old rule's guesses and the new model's guesses for the same orders.

For example, with four orders:

actual = [30, 25, 40, 20]
old = [24, 25, 30, 28]
new = [28, 27, 41, 22]

The old rule's errors, actual minus guess:

Took Old guess Error Size of the error
30 24 6 6
25 25 0 0
40 30 10 10
20 28 −8 8
Total 24

So mae(actual, old) returns 6.0 (24 ÷ 4). The new model's sizes are 2, 2, 1 and 2, so mae(actual, new) returns 1.75, and closer_estimate(actual, old, new) returns "new".

Your job: write two functions.

  • mae(actual, predicted) returns the mean absolute error: the size of each actual value minus its guess, added up, divided by how many orders there are.
  • closer_estimate(actual, old, new) returns "new" when the new guesses' MAE is smaller, and "old" when the old guesses' MAE is smaller.
  • If the two MAEs are equal, closer_estimate returns "old": the new model has to do better to replace the old rule.

Practiced live on the lesson page: write the Python right there and check it.

Check yourself

2 questions to try on the lesson page.

So what

An error is the actual value minus the guess, and the mean absolute error is the average size of those errors: one honest number, in the label's own units, that says how far off a model usually is on examples it never saw.

Sources

  • scikit-learn, Metrics and scoring: Mean absolute error, User Guide: mean_absolute_error computes the mean absolute error, the average over the samples of the absolute value of each true value minus its predicted value.
  • scikit-learn 1.9.1, mean_absolute_error, API reference: mean_absolute_error(y_true, y_pred), with the correct target values first and the estimated ones second; the result is never negative, and its best value is 0.0.
  • scikit-learn, Metrics and scoring: Visual evaluation of regression models, User Guide: a residual is the difference between the actual and the predicted value.
  • Python documentation, abs(): the absolute value of a number.

Key terms

Related