Precision and Recall
Precision asks how many of the payments a fraud check flagged were really fraud, TP divided by TP plus FP, and recall asks how much of the fraud it caught, TP divided by TP plus FN, and which one matters more depends on what each kind of mistake costs, worked out by hand or with precision_score and recall_score.
Journeyman · AI Basics · Python · Updated
After this lesson you can
- Compute precision and recall from the confusion matrix's counts, by hand and with precision_score and recall_score
- Choose whether precision or recall matters more for a problem from what each kind of mistake costs
Key takeaways
- Precision is TP / (TP + FP). Of the payments the check flagged, the share that were really fraud. Low precision means many false alarms.
- Recall is TP / (TP + FN). Of the real fraud, the share the check caught. Low recall means fraud slips through. It's also called sensitivity or the true positive rate.
- Which one matters more depends on what each mistake costs. A missed fraud costs money, and a false alarm costs a member's time and trust.
- precision_score and recall_score take the actual classes first. With nothing flagged, precision is 0 divided by 0, which scikit-learn reports as 0.0 with a warning unless you set zero_division.
Northside Credit Union's fraud team has two different complaints about its fraud check, from two different people. The member services manager says, "Too many of the payments you flag turn out to be honest, and my staff spend the day apologizing." The finance manager says, "Too much fraud gets through, and we refund every dollar of it." Both are right, and one number can't answer both. Any time a model flags things for people to act on (fraud, spam, a part that might fail, a student who might need help), you'll be asked these two questions, and they have names.
The example is still the fraud check on Northside Credit Union's transactions.csv, from Exercise: Fraud Check Scorecard. The Confusion Matrix counted its 1,000 test guesses: 8 true positives, 4 false positives, 12 false negatives and 976 true negatives.
Two questions, two numbers
The member services manager is asking about the payments the check flagged: of those, how many were really fraud? That's precision:
precision = TP / (TP + FP), read "true positives over true positives plus false positives".
The finance manager is asking about the payments that were fraud: of those, how many did the check catch? That's recall:
recall = TP / (TP + FN), read "true positives over true positives plus false negatives".
Both have the true positives on top. The difference is the bottom: precision divides by everything the check said was fraud, and recall by everything that was fraud. In plain Python, from the four counts:
tp, fp, fn = 8, 4, 12
precision = tp / (tp + fp)
recall = tp / (tp + fn)
print("Precision:", round(precision, 3))
print("Recall:", recall)
Precision: 0.667
Recall: 0.4
Of the 12 payments the check flagged, 8 were fraud: a precision of 0.667, so two flags in three are right. Of the 20 frauds, it caught 8: a recall of 0.4, so it misses six frauds in ten. Recall has other names you'll meet: sensitivity in medicine, and the true positive rate in many textbooks. The true negatives don't appear in either number, which is why they hold up when honest payments outnumber fraud 49 to 1.
precision_score and recall_score
scikit-learn's functions take the actual classes first and the guesses second, like accuracy_score, and they treat the class 1 as the positive class:
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score
from sklearn.model_selection import train_test_split
payments = pd.read_csv("transactions.csv")
features = ["amount", "hour", "miles_from_home", "new_merchant", "online"]
train, test = train_test_split(
payments, test_size=0.2, random_state=42, stratify=payments["is_fraud"]
)
model = LogisticRegression()
model.fit(train[features], train["is_fraud"])
guesses = model.predict(test[features])
print("Precision:", round(precision_score(test["is_fraud"], guesses), 3))
print("Recall:", recall_score(test["is_fraud"], guesses))
Precision: 0.667
Recall: 0.4
The same two numbers. Next to the 98.4% accuracy, they tell the real story: a check that's right two times in three when it speaks up, and silent about most of the fraud.
Which mistake costs more?
Neither number is "the" right one. The choice comes from what each mistake costs, and that's a question about the problem, not about machine learning. Here's what the 12 missed frauds cost:
missed = test[(test["is_fraud"] == 1) & (guesses == 0)]
print("Fraud missed:", len(missed))
print("Their total amount:", round(missed["amount"].sum(), 2))
Fraud missed: 12
Their total amount: 1661.51
$1,661.51 that the credit union refunds to its members, from these 1,000 test payments alone. A false alarm costs something too, but less: a staff member's phone call, and a member stuck at a checkout. When a miss costs more than a false alarm, recall matters more, and you'll accept more false alarms to catch more fraud. Other problems weigh the two mistakes differently:
- A spam filter: a false positive hides a real email, perhaps a job offer, in the spam folder, while a missed spam email costs a second of deleting. Precision matters more.
- A reminder email to library members who may not renew: a reminder to someone who'd renew anyway costs almost nothing, and a member lost costs a lot. Recall matters more.
- A screening test for an illness, followed by a second test: missing a sick patient is worse than an extra test. Recall matters more.
Usually you watch both, and decide how much of one to give up for the other. Moving the Threshold shows how.
When nothing is flagged
A check that never says fraud flags nothing, so its precision is 0 ÷ 0, which has no answer. scikit-learn reports it as 0.0 and prints an UndefinedMetricWarning that says Precision is ill-defined and being set to 0.0 due to no predicted samples. Setting zero_division=0 says you know, and gives 0.0 quietly:
never_flags = [0] * len(test)
print("Precision:", precision_score(test["is_fraud"], never_flags, zero_division=0))
print("Recall:", recall_score(test["is_fraud"], never_flags))
Precision: 0.0
Recall: 0.0
Recall here is 0 ÷ 20, which is a real 0: twenty frauds, none caught. Recall has the same problem only when there's no fraud at all to find.
Do it: report precision and recall
The fraud team counts each week's confusion matrix by hand and wants precision and recall from the counts.
The inputs:
tp: frauds the check flagged (true positives).fp: honest payments the check flagged (false positives).fn: frauds the check let through (false negatives).
For example, a week with tp = 6, fp = 2 and fn = 4:
- precision = 6 / (6 + 2) = 6 / 8 =
0.75 - recall = 6 / (6 + 4) = 6 / 10 =
0.6
So precision_and_recall(6, 2, 4) returns (0.75, 0.6).
Your job: write one function.
precision_and_recall(tp, fp, fn)returns a pair: the precision, TP / (TP + FP), then the recall, TP / (TP + FN).- If nothing was flagged (
tp + fpis 0), the precision is0.0. - If there was no fraud to find (
tp + fnis 0), the recall is0.0.
Practiced live on the lesson page: write the Python right there and check it.
Check yourself
2 questions to try on the lesson page.
So what
Precision says how much you can trust a flag, and recall says how much of what you're looking for gets caught; which one to protect is a decision about what each mistake costs, and it belongs to the people who pay for those mistakes.
Sources
- scikit-learn, Metrics and scoring: Precision, recall and F-measures, User Guide: precision is tp / (tp + fp), recall is tp / (tp + fn), and recall is also called sensitivity.
- scikit-learn 1.9.1, precision_score, API reference: precision is intuitively the ability of the classifier not to label as positive a sample that is negative;
pos_labelis 1 by default; when true positives plus false positives is 0, precision returns 0 and raises anUndefinedMetricWarning, whichzero_divisionchanges. - scikit-learn 1.9.1, recall_score, API reference: recall is tp / (tp + fn), intuitively the ability of the classifier to find all the positive samples.
- scikit-learn, Tuning the decision threshold for class prediction, User Guide: when a missed positive costs much more than a false positive (its example is a missed cancer against further tests), it can be better to call a case positive at a probability well below 0.5, accepting more false positives.