Accuracy

Accuracy is the share of a classifier's guesses that are right, worked out by counting matches or with scikit-learn's accuracy_score, and it's a fair first number, but the same accuracy can hide very different mistakes, and a fraud check that never says fraud scores 98% on payments where 2% are fraud.

Apprentice · AI Basics · Python · Updated

By

After this lesson you can

  • Compute accuracy as the share of right guesses, by hand and with accuracy_score
  • Explain why two classifiers with the same accuracy can make very different mistakes, and why a high accuracy can hide a useless model

Key takeaways

  • Accuracy is the number of right guesses divided by the number of guesses. 984 right out of 1,000 is an accuracy of 0.984, or 98.4%.
  • accuracy_score(y_true, y_pred) takes the actual classes first and the guesses second, like every scikit-learn metric.
  • Two checks with the same accuracy can make very different mistakes. One flags honest payments, the other lets every fraud through.
  • When one class is rare, a guess that ignores the data scores high. Never saying fraud is right on 98% of these payments and catches nothing.

Northside Credit Union checks every card payment its members make. A model looks at each one and says "fraud" or "honest", and a payment it calls fraud is paused until someone calls the member. The team built a fraud check, and the first question anyone asks is "how good is it?" You'll answer that question for every classifier you build, for a class, a project or a job, and the first answer most people reach for is the one this lesson teaches. So is the one it warns about.

The example for the whole chapter is Northside Credit Union's 5,000 card payments in transactions.csv, our own made-up data, which comes with Exercise: Fraud Check Scorecard, beside the seed script that made it. Exactly 100 of the payments are fraud: 2%. The column is_fraud is 1 for fraud, the class the check is looking for (the positive class from What Classification Is), and 0 for an honest payment.

Counting the right guesses

Start with ten payments. actual says what each payment really was, and flagged says what the fraud check guessed:

actual = [0, 0, 1, 0, 0, 0, 1, 0, 0, 0]
flagged = [0, 0, 1, 0, 1, 0, 0, 0, 0, 0]

right = 0
for a, f in zip(actual, flagged):
    if a == f:
        right += 1
print("Right:", right, "of", len(actual))
print("Accuracy:", right / len(actual))
Right: 8 of 10
Accuracy: 0.8

A guess is right when it matches the actual class, whichever class that is. The check called eight of the ten payments correctly. Accuracy is that share: the number of right guesses divided by the number of guesses, 8 ÷ 10 = 0.8, or 80%. It's the number a classifier's score gave in What Classification Is, and it's the place every report card starts.

accuracy_score in scikit-learn

scikit-learn has it built in. Each code block from here is one notebook cell, and the cells run in order. This one loads the payments, keeps 1,000 back as test payments with a stratified split (Stratified Splits), trains a logistic regression on the other 4,000, and scores its guesses on the test payments:

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split

payments = pd.read_csv("transactions.csv")
features = ["amount", "hour", "miles_from_home", "new_merchant", "online"]
train, test = train_test_split(
    payments, test_size=0.2, random_state=42, stratify=payments["is_fraud"]
)

model = LogisticRegression()
model.fit(train[features], train["is_fraud"])
guesses = model.predict(test[features])

print("Test payments:", len(test))
print("Fraud among them:", test["is_fraud"].sum())
print("Accuracy:", accuracy_score(test["is_fraud"], guesses))
Test payments: 1000
Fraud among them: 20
Accuracy: 0.984

accuracy_score(y_true, y_pred) takes the actual classes first and the guesses second, the same order as mean_absolute_error. The model was right on 984 of the 1,000 payments it never saw: 98.4%. That sounds excellent. Before you believe it, look at what accuracy can't tell you.

The same score, different mistakes

Here are two fraud checks on the same ten payments, two of which are fraud:

from sklearn.metrics import accuracy_score

actual = [0, 0, 1, 0, 0, 0, 1, 0, 0, 0]
check_a = [0, 1, 1, 0, 0, 0, 1, 1, 0, 0]
check_b = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

print("Check A:", accuracy_score(actual, check_a))
print("Check B:", accuracy_score(actual, check_b))
Check A: 0.8
Check B: 0.8

Both score 80%, and they're nothing alike. Check A caught both frauds and paused two honest payments: two members get a phone call they didn't need. Check B never says fraud at all: it caught neither, and the credit union pays for both. Accuracy counts every wrong guess the same, so it can't tell an annoying mistake from an expensive one. The Confusion Matrix counts the two kinds separately.

A high score for doing nothing

Check B's trick works even better on the real test payments. A "check" that never says fraud, whatever the payment, is right about every honest one:

never_fraud = [0] * len(test)
print("Never says fraud:", accuracy_score(test["is_fraud"], never_fraud))
Never says fraud: 0.98

98%, for a rule that looks at nothing and catches no fraud at all. It's the "most common answer" baseline from The Simplest Guess: Baselines, and when one class is rare it scores almost as well as a real model: the trained model's 98.4% is only 0.4 points better. That's why a classifier's accuracy always goes next to the baseline's, and why accuracy alone is the wrong number for a rare class. When the Answer Is Rare shows what to use instead.

Do it: score two fraud checks

Northside Credit Union has a fraud check's guesses for some payments, and it wants two numbers: how often the check is right, and how many frauds it let through.

The inputs:

  • actual: what each payment really was, 1 for fraud and 0 for honest.
  • predicted: what the check guessed for the same payments, in the same order, 1 or 0.

For example, with eight payments:

actual = [0, 1, 0, 0, 1, 0, 0, 1]
predicted = [0, 1, 1, 0, 0, 0, 0, 1]
Payment 1 2 3 4 5 6 7 8
Actual 0 1 0 0 1 0 0 1
Predicted 0 1 1 0 0 0 0 1
Right? yes yes no yes no yes yes yes

Six of the eight guesses are right, so accuracy(actual, predicted) returns 0.75 (6 ÷ 8). Payment 5 was fraud and the check said 0, so missed_fraud(actual, predicted) returns 1. Payment 3 is a mistake too, but it's an honest payment flagged as fraud, not a missed fraud.

Your job: write two functions.

  • accuracy(actual, predicted) returns the number of guesses that match the actual class, divided by how many payments there are.
  • missed_fraud(actual, predicted) returns how many payments were fraud (actual 1) that the check guessed were honest (predicted 0).
  • The lists always hold at least one payment.

Practiced live on the lesson page: write the Python right there and check it.

Check yourself

2 questions to try on the lesson page.

So what

Accuracy is the honest first count of how often a classifier is right, and it's only worth reading beside the baseline: when the answer you're looking for is rare, a check that does nothing can score 98%.

Sources

  • scikit-learn, Metrics and scoring: Accuracy score, User Guide: accuracy_score computes the fraction (by default) or the count of correct predictions, the share of samples whose predicted class equals the true class.
  • scikit-learn 1.9.1, accuracy_score, API reference: accuracy_score(y_true, y_pred), with the ground truth labels first and the predicted labels second.
  • scikit-learn, Metrics and scoring: Dummy estimators, User Guide: comparing a model against simple rules of thumb is a sanity check, and an accuracy close to such a rule's can mean the classifier is suffering from class imbalance.

Key terms

Related