Up Next: Making Models Better, Honestly

Look back at what Module 3 taught, classifiers that guess a category and the confusion matrix, precision, recall and thresholds that measure them, try a short fraud check memo with all of it, and see what Module 4 adds, scores you can trust and the mistakes that make a model look better than it is.

Journeyman · AI Basics · Python · Updated

By

After this lesson you can

  • Describe how a model's score can look better than it really is, and what Module 4 adds to make a score worth trusting

Key takeaways

  • After Module 3 you can train classifiers, count each kind of mistake they make, and choose a metric and a threshold by what those mistakes cost.
  • A score is only as honest as the data it comes from. A deep tree catches every fraud it trained on and fewer than half of the fraud it never saw.
  • Module 4 spots overfitting, scores models with cross-validation, keeps every preparation step in a pipeline, finds data leaks, and tunes and compares models fairly.

When this module started, every model you'd built guessed a number. Now you can build models that guess a category, spam or not, apple, pear or plum, fraud or honest, and, harder and more valuable, you can say exactly what kind of wrong they are. That second part is what people who use a model care about, whether it's a fraud team, a spam filter's users or a library deciding who gets a reminder.

This lesson looks back at what you can do, gives you something to try with all of it, and shows what Module 4 adds: scores you can trust.

What you can do now

After Module 3, you can:

  • train a classifier with logistic regression, a decision tree or nearest neighbors, read its classes_ and its predict_proba, and split with stratify=y so both parts keep each class's share;
  • count right and wrong with accuracy, always beside a baseline's;
  • sort every guess into true positives, false positives, false negatives and true negatives with confusion_matrix, rows actual and columns guessed;
  • ask the two questions that matter: precision (of the flagged, how many were right) and recall (of the ones to find, how many were found), and choose between them by what each mistake costs;
  • summarize with the F1 score and read a classification_report class by class;
  • see through the accuracy trap with a DummyClassifier baseline and balanced accuracy when one class is rare;
  • move the threshold to trade precision for recall, choosing it on validation data from what the mistakes cost.

Try it: a fraud check memo

A score helps no one until someone can act on it. Try writing a short memo for Northside Credit Union's fraud team, with the files from Exercise: Fraud Check Scorecard and only what you know now:

  • The scorecard, from python report.py: the confusion counts, precision and recall at the default 0.5 line and at 0.05, each in a plain sentence ("Of 20 frauds, the check caught 8").
  • What the 0.05 line costs and saves. Using the chapter's estimates, $200 for a missed fraud and $20 for a false alarm, what does each line cost on the test payments?
  • A different team's costs. If a false alarm cost $100 instead (a member who leaves the credit union), which of the thresholds in Moving the Threshold would you choose on the validation payments? Work it out in the notebook.
  • One thing the scorecard can't tell the team, such as whether next year's fraud will look like this year's.

Write it as a Markdown cell at the end of the notebook. There's no test for this one: when the fraud team could read it and decide where to set the line, it works.

What Module 4 unlocks

Everything in this module rested on one promise: the test score says how the model will do on data it has never seen. Module 4 is about keeping that promise, because it can break without anyone noticing. Here's a sneak peek, a decision tree with no depth limit on the chapter's payments:

import pandas as pd
from sklearn.metrics import recall_score
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

payments = pd.read_csv("transactions.csv")
features = ["amount", "hour", "miles_from_home", "new_merchant", "online"]
train, test = train_test_split(
    payments, test_size=0.2, random_state=42, stratify=payments["is_fraud"]
)

tree = DecisionTreeClassifier(random_state=0)
tree.fit(train[features], train["is_fraud"])

print("Recall on training payments:", recall_score(train["is_fraud"], tree.predict(train[features])))
print("Recall on test payments:", recall_score(test["is_fraud"], tree.predict(test[features])))
Recall on training payments: 1.0
Recall on test payments: 0.45

Every one of the 80 training frauds caught, and 9 of the 20 it never saw. Scored on its own training data, this tree would look perfect. Module 4 shows how to catch that, and the quieter versions of it:

  • Overfitting and cross-validation: spotting a model that memorized its examples, and scoring on several splits instead of one lucky (or unlucky) one.
  • Preparing data in a pipeline: scaling numbers, turning categories into numbers and filling in missing values, every step learned from the training rows only.
  • Data leakage: columns and steps that sneak the answer into training, so a model scores well in testing and fails in real use.
  • Tuning and comparing models fairly: choosing settings on validation data, random forests, and comparing models on the same folds with the same metric.

Check yourself

2 questions to try on the lesson page.

So what

Module 3 taught you to say exactly how a classifier is wrong; Module 4 makes sure the numbers you say it with are honest, so the model works as well in real use as it did in testing.

Sources

  • scikit-learn, Common pitfalls and recommended practices: data leakage happens when information that wouldn't be available when the model is used finds its way into building it, which makes its scores look better than they will be; preparation steps should be learned from the training data only, which a pipeline makes easier.
  • scikit-learn, Cross-validation: evaluating estimator performance, User Guide: testing a model on the same data it learned from is a methodological mistake, because a model that repeats the labels it has seen would score perfectly and still fail on new data; this situation is called overfitting.
  • scikit-learn, Decision Trees: Tips on practical use, User Guide: decision trees tend to overfit data with a large number of features, and settings such as max_depth help prevent it.

Key terms

Related