Squared Errors and RMSE

Squaring each error before averaging makes big misses count much more, giving the mean squared error in squared units, and its square root, the root mean squared error (RMSE), puts it back in the label's units; RMSE is never smaller than the MAE, and a big gap between them says a few guesses are very wrong.

Journeyman · AI Basics · Python · Updated

By

After this lesson you can

  • Compute the mean squared error and its square root, the RMSE, by hand and with root_mean_squared_error
  • Explain why RMSE is never smaller than the MAE, and what a big gap between them says about the misses

Key takeaways

  • The mean squared error (MSE) averages the squared errors, so a big miss counts far more than a small one. It's in squared units, such as squared minutes.
  • The root mean squared error (RMSE) is the square root of the MSE, back in the label's own units.
  • RMSE is never smaller than the MAE. They're equal only when every miss is the same size.
  • A big gap between RMSE and MAE says a few guesses are very wrong, while most are close.
  • root_mean_squared_error(y_true, y_pred) gives the RMSE in scikit-learn.

Two estimates for Pine Ridge Delivery's app are each off by 4 minutes on average. One is 4 minutes off on every order. The other is right on four orders and 20 minutes off on the fifth: a customer whose dinner arrived cold, who leaves a one-star review. The mean absolute error calls them equally good, and most customers wouldn't. When a big miss hurts more than several small ones (a late delivery, a budget that runs out, a train that's missed), you need an error number that notices big misses, and it's the one many reports and textbooks lead with.

The example is Pine Ridge Delivery's deliveries.csv, from Exercise: Delivery Time Errors. The scikit-learn code blocks are notebook cells, run in order: the first loads the data, and the next one uses what it made.

The same MAE, two different estimates

Here are the two estimates' errors (actual minus guess) on the same five orders:

steady = [4, -4, 4, -4, 4]
one_big_miss = [0, 0, 0, 0, 20]

for name, errors in [("steady", steady), ("one big miss", one_big_miss)]:
    total = 0
    for e in errors:
        total += abs(e)
    print(name, "MAE:", total / len(errors))
steady MAE: 4.0
one big miss MAE: 4.0

The same 4.0 minutes. The MAE treats a 20-minute miss as exactly five 4-minute misses.

Square the errors first

Fitting a Line squared each error before adding them up, so that nothing cancels and a big miss counts for more. Do the same here, then average: that's the mean squared error, MSE for short.

import math

for name, errors in [("steady", steady), ("one big miss", one_big_miss)]:
    total = 0
    for e in errors:
        total += e ** 2
    mse = total / len(errors)
    print(name, "MSE:", mse, "RMSE:", round(math.sqrt(mse), 2))
steady MSE: 16.0 RMSE: 4.0
one big miss MSE: 80.0 RMSE: 8.94

The steady estimate's squares are 16 each, so its MSE is 16. The big miss squares to 400, and 400 ÷ 5 = 80. Now the two estimates look different.

The MSE is in squared minutes, a unit nobody can picture, like the squared dollars in Fitting a Line. Its square root (math.sqrt) puts it back in minutes: that's the root mean squared error, RMSE. The name lists the steps from last to first: the root of the mean of the squared errors. The steady estimate's RMSE is 4.0 minutes; the one with a big miss is 8.94.

RMSE in scikit-learn

scikit-learn has both: mean_squared_error and root_mean_squared_error, each with the actual values first and the guesses second. On the same model and test orders as Errors and the Mean Absolute Error:

import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, root_mean_squared_error
from sklearn.model_selection import train_test_split

deliveries = pd.read_csv("deliveries.csv")
features = ["distance_miles", "stops_before", "is_raining"]
train, test = train_test_split(deliveries, test_size=0.2, random_state=42)

model = LinearRegression()
model.fit(train[features], train["minutes"])
guesses = model.predict(test[features])

print("MAE:", round(mean_absolute_error(test["minutes"], guesses), 2))
print("MSE:", round(mean_squared_error(test["minutes"], guesses), 2))
print("RMSE:", round(root_mean_squared_error(test["minutes"], guesses), 2))
MAE: 3.63
MSE: 32.74
RMSE: 5.72

The model's RMSE is 5.72 minutes, well above its MAE of 3.63. (Older code gets the RMSE with mean_squared_error(..., squared=False). scikit-learn 1.4 added root_mean_squared_error in its place, and the old option is gone from today's versions.)

What the gap between RMSE and MAE says

RMSE is never smaller than the MAE. Squaring makes the big errors weigh more, so they pull the RMSE up more than they pull the MAE; the two are equal only when every miss is the same size, like the steady estimate's. So the gap between them says something: a small gap means the misses are about the same size, and a big gap means a few misses are much bigger than the rest. Here are this model's biggest misses:

errors = test["minutes"] - guesses
print("Biggest misses:", errors.abs().sort_values().tail(6).round(1).tolist())

small = errors.abs() <= 10
print("Orders off by 10 minutes or less:", small.sum())
print("Their MAE:", round(mean_absolute_error(test["minutes"][small], guesses[small]), 2))
print("Their RMSE:", round(root_mean_squared_error(test["minutes"][small], guesses[small]), 2))
Biggest misses: [16.6, 18.0, 18.2, 19.3, 21.0, 22.5]
Orders off by 10 minutes or less: 94
Their MAE: 2.63
Their RMSE: 3.3

Six orders took 16 to 23 minutes longer than the app said, and all of them were late, never early. Leave them out and the gap nearly closes: 2.63 against 3.3. Those six are what seed_data.py planted: 3% of orders hit a traffic jam, which no column in the file shows, so no model could see them coming. A value far from the rest like this is called an outlier.

Which number to report depends on what a miss costs. MAE says how far off a typical guess is. RMSE is the one to watch when one big miss hurts more than several small ones, and reporting both, with their gap, tells the fullest story. Removing the six orders to get a better score would hide exactly the customers who waited longest: the gap is a reason to look at those orders, not to drop them.

Do it: measure the gap between RMSE and MAE

Pine Ridge Delivery's team wants to know, for any set of orders, whether the misses are all about the same size or whether a few are much bigger. You'll write the RMSE and the gap between it and the MAE.

The inputs:

  • actual: how many minutes each order really took.
  • predicted: the app's guesses for the same orders, in the same order.

mae(actual, predicted) is given in the starter, ready to use.

For example, with five orders:

actual = [28, 32, 35, 20, 48]
predicted = [30, 30, 33, 18, 40]
Took Guessed Error Squared
28 30 −2 4
32 30 2 4
35 33 2 4
20 18 2 4
48 40 8 64
Mean 16.0

The mean squared error is 80 ÷ 5 = 16.0, so rmse(actual, predicted) returns 4.0, its square root. The MAE is (2 + 2 + 2 + 2 + 8) ÷ 5 = 3.2, so rmse_gap(actual, predicted) returns 0.8, which is 4.0 − 3.2 rounded to 2 decimal places.

Your job: write two functions.

  • rmse(actual, predicted) returns the root mean squared error: each actual value minus its guess, squared, averaged over the orders, then the square root of that average.
  • rmse_gap(actual, predicted) returns the RMSE minus the MAE, rounded to 2 decimal places with round(..., 2).

Practiced live on the lesson page: write the Python right there and check it.

Check yourself

2 questions to try on the lesson page.

So what

RMSE squares the misses before it averages them, so it notices the few guesses that are badly wrong; read it beside the MAE, and the gap between them tells you whether the misses are all small or whether a few customers waited far longer than promised.

Sources

  • scikit-learn, Metrics and scoring: Mean squared error, User Guide: mean_squared_error computes the average of the squared differences between the true and predicted values; its square root, the root mean squared error (RMSE), is in the same units as the target and is available through root_mean_squared_error.
  • scikit-learn 1.9.1, root_mean_squared_error, API reference: root_mean_squared_error(y_true, y_pred), the root mean squared error regression loss, added in version 1.4.
  • scikit-learn 1.9.1, mean_squared_error, API reference: mean_squared_error(y_true, y_pred), the mean squared error regression loss, with no squared parameter in this version.
  • Python documentation, math.sqrt: the square root of a number.

Key terms

Related