Trusting a Model in the Real World

A model is trained once, but the world keeps changing. Check whether new data still looks like the data the model learned from, save a trained model safely and load it again, and write a model card: a one-page description of what the model is for, how well it does for whom, and where it must not be used. Then look back at the course and ahead to what comes next. The running example is the delivery service, a year later.

Updated

When the World Changes

  1. When the World Changes (Journeyman): A model learns the world as it was in its training data, so when new examples stop looking like those rows (data drift), or the same inputs start leading to different answers (concept drift), its guesses slip; checking each column's range and mean on the new rows catches the first, and the error on new real answers catches both, which is when you retrain.

Saving and Describing a Model

  1. Saving and Loading a Model (Journeyman): joblib.dump writes a trained model to a file and joblib.load brings it back with the same guesses, as long as you save the whole pipeline, load it with the same scikit-learn version it was saved with, and only ever load files you or someone you trust made, because loading one can run any code inside it.
  2. Model Cards (Journeyman): A model card is a short document that travels with a trained model and says what it's for, what it must not be used for, what data it learned from, how well it does overall and for each group, its limits, and when it was trained, as Mitchell and colleagues proposed in Model Cards for Model Reporting (2019).

Exercise: Model Check-Up

  1. Exercise: Model Check-Up (Journeyman): Give Pine Ridge Delivery's year-old model its check-up in Python, saving the whole pipeline with joblib and loading it back only when the scikit-learn version matches, reporting each column's share of new orders outside the training range and its change in mean, and building a model card that refuses an empty section, with pytest checking each step.

Where to Go Next

  1. Where to Go Next (Journeyman): Look back at the whole course as the steps of one machine learning project, from deciding whether a model fits the question to watching it after it's in use, see plainly what the course left out, such as neural networks, text, images and forecasting, and find the ask-codey course that builds on it.