Up Next: Groups, Fairness and Explaining

Look back at what Module 4 taught, catching overfitting, cross-validation, pipelines, data leakage, tuning and fair comparisons, try a short honest write-up of the inn's model with all of it, and see what Module 5 adds, finding groups in data that has no labels, and explaining a model's results, limits and fairness to the people it affects.

Journeyman · AI Basics · Python · Updated

By

After this lesson you can

  • Describe what finding groups without labels and explaining a model to the people it affects will let you do

Key takeaways

  • After Module 4 you can score a model honestly, keep every learned step inside a pipeline, find leaks, tune settings and compare models fairly.
  • Module 5 finds groups in data with no label to learn from, with k-means, and says what each group means and doesn't.
  • Module 5 also explains a model to someone who isn't technical, measures whether it treats groups of people differently, and says when it shouldn't be trusted.

When this module started, a model that scored well was a model that worked. Now you know better. You can tell a model that learned from one that memorized, score it on folds instead of one lucky split, keep every step that learns from data inside a pipeline, spot the leaks that make a score too good to be true, tune its settings without touching the test set, and choose between models on equal terms. That's what separates a model that works in a notebook from one that works on new data, and it's what a reviewer, a manager or an interviewer will check first.

This lesson looks back at what you can do, gives you something to try with all of it, and shows what Module 5 adds.

What you can do now

After Module 4, you can:

  • spot overfitting, a training score far above the test score, and fix it with limits, a simpler model or more data;
  • cross-validate, with a seeded splitter, and report the mean and the spread, with splits that respect time order and repeated customers;
  • keep the test set for last, choosing with validation folds and looking at the test rows once;
  • prepare data in a pipeline: scale numbers, one-hot encode categories, fill missing values, each learned from the training rows only, and different steps for different columns;
  • find leaks: a column known only after the outcome, preparation done before the split, the same customer on both sides;
  • tune settings with a grid search over a pipeline's step__setting names, refit on the training rows;
  • use a random forest, and know what it costs in time and readability;
  • compare models fairly: one splitter, one metric, a baseline, the mean and spread side by side, ties to the simpler model, and one test score, reported as it is.

Try it: an honest write-up

A score is only useful when someone can act on it. With the files from Exercise: Fair Model Showdown and only what you know now, write a short note for the Harbor View Inn's manager:

  • The model you'd use, its test accuracy from python report.py, and the baseline's accuracy beside it.
  • How sure that number is: the spread across folds, in plain words.
  • Which mistakes it makes. In the notebook, count the canceled test bookings the model missed and the kept bookings it flagged (The Confusion Matrix shows how), and say which one costs the inn more.
  • One thing it can't tell the manager, such as why a guest canceled, and why.

Write it as a Markdown cell at the end of the notebook. There's no test for this one: when the manager could read it and know how far to trust the model, it works.

What Module 5 unlocks

So far, every dataset came with the answer to learn: a price, a cancellation, a spam label. Plenty of data doesn't. A café knows how often each loyalty member visits and what they spend, but nobody ever labeled the members "morning commuter" or "weekend brunch". Finding groups like those without labels is clustering, a kind of unsupervised learning, and Module 5 starts there:

  • Finding groups with k-means: how the k-means method finds groups, how many groups to look for, and how to describe each group honestly, including what the grouping can't tell you.
  • Explaining a model: what a model leans on, how a guess changes with one feature, and how to explain a result to someone who isn't technical.
  • Bias and fairness: where bias in data comes from, how to measure a model's results group by group, and why some kinds of fairness can't all hold at once.
  • Trusting a model in the real world: noticing when new data stops looking like the training data, saving a model safely, and writing a model card that says where it must not be used.

Here's a sneak peek, on eight made-up café members: visits a month and average spend in dollars, with no label at all:

import pandas as pd
from sklearn.cluster import KMeans

members = pd.DataFrame({
    "visits_per_month": [22, 25, 20, 24, 3, 2, 4, 3],
    "avg_spend": [4.5, 5.0, 4.0, 4.75, 18.0, 21.5, 19.0, 22.0],
})
groups = KMeans(n_clusters=2, n_init=10, random_state=0).fit_predict(members)
print(groups)
[1 1 1 1 0 0 0 0]

There's no y this time: the model is given the members and a number of groups, and it puts the frequent small spenders in one group and the rare big spenders in the other. The numbers 0 and 1 are only names; what each group means is for you to work out and explain. You don't need to understand every part of it yet: that's what Module 5 is for.

Check yourself

2 questions to try on the lesson page.

So what

Module 4 taught you to measure a model honestly and choose one fairly. Module 5 finds patterns nobody labeled, and asks the questions that come after a model works: what it leans on, who it treats differently, and when not to trust it.

Sources

  • scikit-learn, Clustering, User Guide: clustering of unlabeled data with sklearn.cluster; KMeans tries to separate samples in n groups of equal variance, and needs the number of clusters to be specified.
  • scikit-learn 1.9.1, KMeans, API reference: n_clusters, n_init for the number of times k-means runs with different starting centers (the best run is kept), random_state for the centers' starting points (an integer makes the randomness deterministic), and fit_predict, which computes the cluster centers and predicts the cluster index for each sample.

Key terms

Related