Exercise: Delivery Time Errors

Write the mean absolute error, the root mean squared error, R squared and a check against the always-the-average baseline in plain Python for Pine Ridge Delivery's time estimates, and let pytest compare each one with scikit-learn's own function.

Journeyman · AI Basics · Python · Updated

By

Pine Ridge Delivery's app tells every customer how long their order will take, and the team wants a report card for the model behind it: how far off it usually is, whether a few orders are badly off, and whether it beats saying the average to everyone. You'll write the four functions behind that report card in plain Python, the way Errors and the Mean Absolute Error, Squared Errors and RMSE and R Squared: Better Than Guessing the Average? worked them out by hand. Tests compare each one with scikit-learn's own function on the same values, so you'll know your arithmetic matches the library's to the ninth decimal place.

Get the exercise code

Open this lesson's GitHub repository, github.com/ask-codey/exercise-ml-delivery-time-errors, also linked from the Exercise box beside the lesson. Clone it into your python-course folder. In VS Code's terminal:

cd ~/python-course
git clone https://github.com/ask-codey/exercise-ml-delivery-time-errors.git

Git makes a folder, exercise-ml-delivery-time-errors, with the exercise code inside. Open that folder in VS Code (File > Open Folder), open its terminal with Terminal > New Terminal, and make the exercise's virtual environment. On Windows (PowerShell):

python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -r requirements.txt

On a Mac (zsh):

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

The prompt starts with (.venv) once the virtual environment is on. The exercise's README says what each command does, and what to do if PowerShell won't run the second one.

The folder holds:

  • deliveries.csv: the 500 orders, our own made-up data, the same file the chapter's lessons use.
  • seed_data.py: the script that made the CSV, with the pattern it planted, traffic jams included. You don't need to run it: the committed CSV is what keeps everyone's numbers the same.
  • delivery_errors.py: the four functions, for you to finish.
  • explore.ipynb: a notebook for looking at the orders, drawing the model's residuals, and trying your functions beside scikit-learn's. Choose the .venv kernel with Select Kernel (top right) before you run it. The tests don't read it.
  • report.py: trains a model on 400 orders, guesses the other 100, and scores the guesses with your four functions. Ready to use.
  • tests/: the tests, which call each function and check what comes back, and conftest.py, which gives a test a fresh copy of the orders. You don't need to change them.
  • requirements.txt and pytest.ini: the packages the exercise needs, at exact versions, and where pytest finds the tests and your code.

What to do

Every function in delivery_errors.py is already declared with its name and parameters, and for now it stops with a NotImplementedError that says Not built yet. The docstring under each one says what it should do, with examples:

def rmse(actual, predicted):
    """Return the root mean squared error: the square root of the average squared error.

    Square each error (actual value minus guess), average the squares (that's
    the mean squared error, in squared minutes), then take the square root to
    get back to minutes. math.sqrt(x), or x ** 0.5, is the square root of x.

    rmse([28, 32, 35, 20, 48], [30, 30, 33, 18, 40]) -> 4.0   (squares 4, 4, 4, 4, 64; 80 / 5 = 16)
    rmse([30, 30], [20, 40])                         -> 10.0  (not 100.0, the squared minutes)
    """
    raise NotImplementedError("Not built yet.")

In each function, replace the raise NotImplementedError("Not built yet.") line with your own code, in plain Python: no scikit-learn, NumPy or pandas inside the functions (Python's own math is fine), and no rounding.

  1. mae(actual, predicted) returns the mean absolute error: the size of each actual value minus its guess, averaged. mae([30, 25, 40, 20], [24, 25, 30, 28]) returns 6.0, and one order off by 6, mae([30], [36]), returns 6.0.
  2. rmse(actual, predicted) returns the root mean squared error: each error squared, averaged, then the square root. rmse([28, 32, 35, 20, 48], [30, 30, 33, 18, 40]) returns 4.0, and rmse([30, 30], [20, 40]) returns 10.0, minutes, not squared minutes.
  3. r_squared(actual, predicted) returns 1 minus the model's squared errors added up, divided by the squared errors of always guessing the average of actual. r_squared([10, 20, 30, 40], [12, 18, 33, 37]) returns 0.948; perfect guesses give 1.0, always guessing 25 gives 0.0, and the guesses backwards, [40, 30, 20, 10], give -3.0. The tests always give actual at least two different values.
  4. beats_baseline(y_train, y_test, y_pred) returns True when the model's MAE on the test values is smaller than the MAE of guessing the average of y_train for every test value, as DummyRegressor(strategy="mean") does, and False otherwise, a tie included. beats_baseline([20, 30, 40], [25, 35], [26, 34]) returns True: the model is off by 1.0 on average and the baseline, guessing 30, by 5.0.

Every function also works on a pandas column and a NumPy array, as report.py gives them: a for loop over zip, len and sum work the same on all three.

Check your work

Run the tests from the exercise folder, with the virtual environment on:

python -m pytest

At first all 20 tests fail. That's expected: each function you finish turns its tests green. A failing test's message says what it called, what it expected and what came back, for example rmse([30, 30], [20, 40]) should return 10.0, but it returned 100.0. Several tests compare your answer with scikit-learn's mean_absolute_error, root_mean_squared_error or r2_score on the same eight orders, to 9 decimal places, and one compares beats_baseline with a real DummyRegressor on the 100 test orders. No test checks how good the model is: they check your arithmetic.

To see your functions at work on the whole file, run python report.py.

Hints

  • One error at a time. zip(actual, predicted) gives each actual value and its guess together in a for loop, as the Do Its in Errors and the Mean Absolute Error and Squared Errors and RMSE did.
  • Name the steps. RMSE is the steps of its name from last to first: the errors, squared, their mean, then the root.
  • Two totals for R squared. R Squared: Better Than Guessing the Average? adds up the model's squared errors and the average guess's squared errors in the same loop; the average is sum(actual) / len(actual).
  • Reuse your own work. beats_baseline can build a list of the training average, one per test value, and call your mae twice.
  • Which average? The baseline guesses the training labels' average, the only one a model could know before the test.

When you're done

Every test is green: pytest's last line says 20 passed. And python report.py prints:

1) orders: 500
2) training orders: 400
3) test orders: 100
4) MAE: 3.63
5) RMSE: 5.72
6) R squared: 0.69
7) baseline MAE: 8.26
8) beats the baseline: True

The same numbers the chapter's lessons got from scikit-learn, now from your own code. Look at lines 4 and 5: what does the gap between them say about the orders? In the notebook, find the orders whose residuals are over 10 minutes and look at their rows: does any column (the distance, the stops, the rain, the hour) set them apart, or are they the traffic jams seed_data.py describes, which no column shows?

Submit your exercise

The exercise is how you finish this lesson. When your work is done, put it on GitHub in a repo of your own, then submit that repo's address here. Hand In Your Work with Git walks through each step.

  1. On GitHub, click +, then New repository. Name it (for example exercise-ml-delivery-time-errors), choose Public, and click Create repository. Leave out the README: your folder already has one.

  2. In VS Code's terminal, in your exercise folder, send your work to your new repo. Point your copy at your own repo, then commit and push:

    git remote set-url origin https://github.com/your-name/exercise-ml-delivery-time-errors.git
    git add .
    git commit -m "My solution"
    git push -u origin main
    
  3. Paste your repo's address, such as https://github.com/your-name/exercise-ml-delivery-time-errors, into Submit your exercise on this page. The site checks that the repo is public and holds your commits, and then the lesson is complete.

Changed something after you submitted? Push it, and submit again: your latest submission is the one that counts.

So what

Every error number a library prints is a few lines of arithmetic you can now write and check yourself, which is what lets you explain a model's report card, and trust it.

Key terms

Related