The Confusion Matrix
A confusion matrix sorts a classifier's guesses into four counts, true positives, false positives, false negatives and true negatives, so you can see which mistakes it makes, and scikit-learn's confusion_matrix puts the actual class in rows and the guessed class in columns, in sorted label order.
Journeyman · AI Basics · Python · Updated
After this lesson you can
- Sort a classifier's guesses into true positives, false positives, false negatives and true negatives, and say what each one means
- Read scikit-learn's confusion_matrix, with the actual classes in rows and the guessed classes in columns, in sorted label order
Key takeaways
- Every guess lands in one of four cells. A true positive is fraud caught, a false positive is a false alarm, a false negative is fraud missed, and a true negative is an honest payment let through.
- confusion_matrix(y_true, y_pred) puts the actual class in rows and the guessed class in columns, both in sorted order. With 0 and 1 the first row is the honest payments.
- For two classes, confusion_matrix(...).ravel() gives the four counts in the order tn, fp, fn, tp.
- The fraud check's 98.4% accuracy is 8 frauds caught, 12 missed and 4 honest payments flagged. The matrix shows what one score hides.
Accuracy ended with two fraud checks that both scored 80% on the same ten payments: one paused two honest payments, the other let both frauds through. A manager reading "80%" can't tell which check they're getting. Whenever someone asks what a model gets wrong (a security team, a doctor reading a screening result, a teacher grading an essay checker), the answer is a small table of four counts. You'll make that table for every classifier you build.
The example is still Northside Credit Union's fraud check, on the payments in transactions.csv, which comes with Exercise: Fraud Check Scorecard.
Four kinds of guess
A fraud check can be right in two ways and wrong in two ways. Each has a name, built from two words: positive or negative is what the check said (fraud, the positive class, or honest), and true or false is whether it was right.
- A true positive (TP): the check said fraud, and it was fraud. Fraud caught.
- A false positive (FP): the check said fraud, and it was honest. A member's card paused for nothing: a false alarm.
- A false negative (FN): the check said honest, and it was fraud. Fraud missed, and the credit union pays for it.
- A true negative (TN): the check said honest, and it was honest. Nothing happens, which is exactly right.
Read the second word first: a false negative is a "not fraud" guess that was wrong. Sorting the ten payments from Accuracy by hand:
actual = [0, 0, 1, 0, 0, 0, 1, 0, 0, 0]
flagged = [0, 0, 1, 0, 1, 0, 0, 0, 0, 0]
tp = fp = fn = tn = 0
for a, f in zip(actual, flagged):
if a == 1 and f == 1:
tp += 1
elif a == 0 and f == 1:
fp += 1
elif a == 1 and f == 0:
fn += 1
else:
tn += 1
print("True positives:", tp)
print("False positives:", fp)
print("False negatives:", fn)
print("True negatives:", tn)
True positives: 1
False positives: 1
False negatives: 1
True negatives: 7
The check's two wrong guesses were one of each: a false alarm (payment 5) and a missed fraud (payment 7). The four counts always add up to the number of payments, 10, and the right ones, TP + TN = 8, give the accuracy, 8 ÷ 10.
confusion_matrix in scikit-learn
Laid out as a grid, the four counts are a confusion matrix: it shows where the classifier confuses one class for the other. This cell trains the same fraud check as Accuracy and counts its 1,000 test guesses:
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
payments = pd.read_csv("transactions.csv")
features = ["amount", "hour", "miles_from_home", "new_merchant", "online"]
train, test = train_test_split(
payments, test_size=0.2, random_state=42, stratify=payments["is_fraud"]
)
model = LogisticRegression()
model.fit(train[features], train["is_fraud"])
guesses = model.predict(test[features])
print(confusion_matrix(test["is_fraud"], guesses))
[[976 4]
[ 12 8]]
confusion_matrix(y_true, y_pred) puts the actual class in the rows and the guessed class in the columns, both in sorted order: 0 (honest) first, then 1 (fraud). So the grid reads:
| Guessed honest (0) | Guessed fraud (1) | |
|---|---|---|
| Actually honest (0) | 976 true negatives | 4 false positives |
| Actually fraud (1) | 12 false negatives | 8 true positives |
The right guesses run down the diagonal, from top left to bottom right: 976 + 8 = 984, the 98.4% accuracy. The mistakes sit off it. And now the 98.4% has a story: of the 20 frauds, the check caught 8 and missed 12, and it paused 4 honest payments on the way.
The four counts by name
For two classes, .ravel() flattens the grid row by row, so the four counts come out in the order tn, fp, fn, tp:
tn, fp, fn, tp = confusion_matrix(test["is_fraud"], guesses).ravel()
print("Fraud caught:", tp)
print("Honest payments flagged:", fp)
print("Fraud missed:", fn)
print("Honest payments let through:", tn)
Fraud caught: 8
Honest payments flagged: 4
Fraud missed: 12
Honest payments let through: 976
Sorted order matters when the classes are words. With the labels "fraud" and "honest", "fraud" sorts first, so the fraud row is on top and the grid is flipped from the one above:
print(confusion_matrix(["fraud", "honest", "honest"], ["honest", "honest", "fraud"]))
[[0 1]
[1 1]]
The first row is the one fraud: 0 guessed fraud, 1 guessed honest. Before you read a matrix, check which class is first; confusion_matrix's labels= setting lets you choose the order.
Seeing it
ConfusionMatrixDisplay draws the same grid, with names on the rows and columns:
import matplotlib.pyplot as plt
from sklearn.metrics import ConfusionMatrixDisplay
ConfusionMatrixDisplay.from_predictions(
test["is_fraud"], guesses, display_labels=["honest", "fraud"]
)
plt.show()

display_labels names the classes in the same sorted order as the grid: 0 is honest, 1 is fraud. The colors only show how big each count is; the numbers carry the meaning.
Do it: build the fraud check's grid
Northside Credit Union wants the confusion matrix for a batch of payments, laid out the way scikit-learn lays it out, so it can be printed beside scikit-learn's.
The inputs:
actual: what each payment really was, 1 for fraud and 0 for honest.predicted: what the check guessed for the same payments, in the same order, 1 or 0.
For example, with eight payments:
actual = [1, 0, 0, 1, 0, 1, 0, 0]
predicted = [1, 0, 1, 0, 0, 1, 1, 0]
| Payment | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|---|---|---|---|---|---|---|---|---|
| Actual | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 |
| Predicted | 1 | 0 | 1 | 0 | 0 | 1 | 1 | 0 |
| Cell | TP | TN | FP | FN | TN | TP | FP | TN |
That's 3 true negatives, 2 false positives, 1 false negative and 2 true positives, so matrix(actual, predicted) returns [[3, 2], [1, 2]].
Your job: write one function.
matrix(actual, predicted)returns a list of two rows, as scikit-learn'sconfusion_matrixdoes with the labels 0 and 1:[[tn, fp], [fn, tp]].- The first row is the honest payments: how many were guessed honest (tn), then how many were guessed fraud (fp).
- The second row is the frauds: how many were guessed honest (fn), then how many were guessed fraud (tp).
- A cell with no payments in it is
0.
Practiced live on the lesson page: write the Python right there and check it.
Check yourself
2 questions to try on the lesson page.
So what
A confusion matrix turns one score into the four things that actually happened, fraud caught, fraud missed, false alarms and honest payments let through, so you can see which mistakes a classifier makes before you decide whether you can live with them.
Sources
- scikit-learn, Metrics and scoring: Confusion matrix, User Guide: each row of the matrix is a true class and each column a predicted class; for a binary problem,
tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()gives the counts of true negatives, false positives, false negatives and true positives. - scikit-learn 1.9.1, confusion_matrix, API reference: entry i, j is the number of samples whose true label is the i-th class and predicted label the j-th class; when
labelsisNone, the labels that appear iny_trueory_predare used in sorted order, andlabelscan reorder them. - scikit-learn 1.9.1, ConfusionMatrixDisplay, API reference:
from_predictionsplots a confusion matrix from true and predicted labels, anddisplay_labelsnames the classes on the plot.