Probabilities and the 0.5 Line

predict_proba gives each example one probability per class, in the order of classes_, predict picks the class with the highest one, which for two classes is a line at 0.5, and the probability is the model's confidence, not a guarantee, so you can draw the line somewhere else when the job needs it.

Journeyman · AI Basics · Python · Updated

By

After this lesson you can

  • Read predict_proba's columns in the order of classes_, and pick out the positive class's probability
  • Turn probabilities into labels at a threshold, and explain why a probability is confidence, not a guarantee

Key takeaways

  • predict_proba gives one column per class, in the order of classes_, and each row adds up to 1. With classes [0, 1], column 1 is the chance of spam.
  • For two classes, predict picks the class whose probability is higher, which is the same as calling spam above 0.5.
  • A probability is the model's confidence, learned from past examples, not a guarantee. Some emails at 0.88 aren't spam, and some at 0.1 are.
  • You can draw the line somewhere else. A lower line flags more emails as spam, and a higher one flags fewer.

Inbox Guard doesn't have to treat every email the model calls spam the same way. An email it's nearly sure about can go straight to the spam folder; one it's unsure about could stay in the inbox with a warning. That choice needs more than a yes or a no: it needs the model's probability. Whenever a model's guess sets off an action (a reminder email, a fraud hold, a doctor's second look), you'll read its probabilities and decide where to draw the line.

The example is Inbox Guard's emails.csv, from Exercise: Spam or Not. Each code block is one notebook cell, run in order: the first one loads the data, splits it and trains the model from Logistic Regression.

One column per class

predict_proba (read "predict proba", short for probability) gives each email a row of probabilities, one for each class:

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

emails = pd.read_csv("emails.csv")
FEATURES = ["num_links", "num_exclamations", "mentions_free", "sender_in_contacts", "subject_all_caps"]
X_train, X_test, y_train, y_test = train_test_split(
    emails[FEATURES], emails["is_spam"], test_size=0.2, random_state=42
)
model = LogisticRegression()
model.fit(X_train, y_train)
probabilities = model.predict_proba(X_test)
print("Classes:", model.classes_)
print(probabilities[:4].round(3))
Classes: [0 1]
[[0.118 0.882]
 [0.971 0.029]
 [0.641 0.359]
 [0.781 0.219]]

The columns come in the order of classes_: first the chance of class 0 (not spam), then of class 1 (spam). The first test email is 0.118 not spam and 0.882 spam. Each row adds up to 1, because the email has to be one or the other. To get the chance of spam for every email, take column 1:

spam_chance = probabilities[:, 1]
print("Chances of spam:", spam_chance[:4].round(3))
print("Guesses:        ", model.predict(X_test.head(4)))
Chances of spam: [0.882 0.029 0.359 0.219]
Guesses:         [1 0 0 0]

probabilities[:, 1] reads "every row, column 1". Take it by position from classes_, never by habit: with the classes ['not spam', 'spam'], spam is column 1 too, but with ['spam', 'trusted'] it would be column 0.

predict draws the line at 0.5

predict picks the class with the highest probability. With two classes, that's the same as calling an email spam when its chance is above 0.5:

at_half = (spam_chance > 0.5).astype(int)
print("Same as predict for all 200:", (at_half == model.predict(X_test)).all())
Same as predict for all 200: True

(spam_chance > 0.5) gives True or False for each email, and .astype(int) turns those into 1 and 0. An email at exactly 0.5 is called not spam by predict, which needs more than 0.5; real probabilities almost never land exactly on it. The line's name is the threshold (your textbook may say decision threshold).

Confidence, not a guarantee

A probability is how sure the model is, from the patterns in the emails it learned from. It isn't a promise about one email:

print("Chance of spam:", spam_chance[0].round(3))
print("Guess:", model.predict(X_test.head(1))[0])
print("What it really was:", y_test.iloc[0])
Chance of spam: 0.882
Guess: 1
What it really was: 0

88.2% sure, and wrong. Across many emails, the probabilities do mean something:

sure = spam_chance >= 0.8
unlikely = spam_chance < 0.2
print("Given 0.8 or more:", sure.sum(), "emails, and", y_test[sure].sum(), "were spam")
print("Given under 0.2:", unlikely.sum(), "emails, and", y_test[unlikely].sum(), "were spam")
Given 0.8 or more: 7 emails, and 6 were spam
Given under 0.2: 107 emails, and 10 were spam

y_test[sure] keeps only the emails where sure is True, and adding up their 1s counts the spam. "Nearly sure" was right 6 times out of 7, and "unlikely" still let 10 spam emails through. A model that's right most of the time is never right every time, so a filter that acts on a probability needs a way back for the emails it gets wrong, such as a spam folder people can check.

Drawing the line somewhere else

0.5 is the default, not a rule. You can count an email as spam when its chance is at or above any line you choose:

for threshold in [0.3, 0.5, 0.7]:
    flagged = (spam_chance >= threshold).sum()
    print("Line at", threshold, "flags", flagged, "of 200 emails")
Line at 0.3 flags 63 of 200 emails
Line at 0.5 flags 32 of 200 emails
Line at 0.7 flags 14 of 200 emails

A lower line catches more spam and moves more good emails to the spam folder; a higher line does the opposite. Which line is right depends on which mistake costs more, and choosing it fairly (from validation data, never the test emails) is what Moving the Threshold teaches.

Do it: flag the likely spam

Inbox Guard's model has scored four new emails, and you want the ones to send to the spam folder.

The inputs:

  • rows: what predict_proba gave, one pair per email: the chance it's not spam, then the chance it's spam (the classes are [0, 1]).
  • email_ids: each email's ID, in the same order as rows.
  • chances: each email's chance of spam, in the same order.
  • threshold: the line. An email whose chance is at or above it is flagged.

For example:

email_ids = ["IG-2001", "IG-2002", "IG-2003", "IG-2004"]
rows = [[0.9, 0.1], [0.25, 0.75], [0.5, 0.5], [0.6, 0.4]]
Email Row Chance of spam At or above 0.5?
IG-2001 [0.9, 0.1] 0.1 no
IG-2002 [0.25, 0.75] 0.75 yes
IG-2003 [0.5, 0.5] 0.5 yes
IG-2004 [0.6, 0.4] 0.4 no

So spam_chances(rows) returns [0.1, 0.75, 0.5, 0.4], and flag_spam(email_ids, [0.1, 0.75, 0.5, 0.4], 0.5) returns ["IG-2002", "IG-2003"].

Your job: write two functions.

  • spam_chances(rows) returns a list of each row's chance of spam: the second number of each pair, in order.
  • flag_spam(email_ids, chances, threshold) returns a list of the IDs whose chance is at or above threshold, in the order they're given.
  • A chance exactly on the line is flagged.
  • If no chance reaches the line, or the lists are empty, flag_spam returns [].

Practiced live on the lesson page: write the Python right there and check it.

Check yourself

2 questions to try on the lesson page.

So what

A classifier's probabilities are more useful than its yes or no: read the positive class's column by classes_, treat each one as confidence rather than a promise, and draw the line where the job needs it.

Sources

  • scikit-learn 1.9.1, LogisticRegression, API reference: predict_proba returns an array of shape (n_samples, n_classes) with the probability of each class, ordered as in classes_; in the binary case, decision_function gives the score for classes_[1], where a score above 0 means that class is predicted.
  • scikit-learn, Glossary of Common Terms and API Elements: predict_proba outputs one column per class in binary classification, and each row adds to 1; classes_ gives the order of predict_proba's columns.
  • scikit-learn, Linear Models: Logistic regression, User Guide: the predicted probability is used as a classifier by applying a threshold, by default 0.5.
  • scikit-learn, Tuning the decision threshold for class prediction, User Guide: for binary classification, predict uses a cut-off of 0.5 on the probability from predict_proba, which isn't the best cut-off for most uses.

Key terms

Related