The F1 Score and the Classification Report
The F1 score joins precision and recall into one number with their harmonic mean, which stays low when either one is low, scikit-learn gives it with f1_score, and classification_report prints precision, recall and F1 for every class with macro and weighted averages, which you read class by class, because one number can hide the class you care about.
Journeyman · AI Basics · Python · Updated
After this lesson you can
- Compute the F1 score as the harmonic mean of precision and recall, and explain why it's low when either one is low
- Read a classification_report's rows for each class, its support, and its macro and weighted averages
Key takeaways
- The F1 score is the harmonic mean of precision and recall, 2 × P × R / (P + R). It's high only when both are high.
- A plain average lets a big number hide a small one. Precision 0.9 and recall 0.1 average 0.5, but their F1 is 0.18.
- classification_report prints precision, recall, F1 and support (how many examples) for each class, then accuracy and two averages.
- The macro average treats every class the same, and the weighted average counts each class by its support, so a common class can make it look good.
- F1 counts a false alarm and a missed fraud the same. When their costs differ, read precision and recall themselves.
Northside Credit Union's fraud team is comparing two new fraud checks, and the director wants one number per check for the slide: "Which is better?" Precision and recall are two numbers, and there's a classic way to make them one. Comparing models in a project, reading a paper's results table or entering a competition, you'll meet it constantly, along with the report scikit-learn prints for every classifier. Both are useful, and both can mislead if you don't know what they leave out.
The example is still the fraud check on transactions.csv from Exercise: Fraud Check Scorecard: precision 0.667 and recall 0.4 on its 1,000 test payments (Precision and Recall).
Why not the plain average?
The first idea is to average the two. Here's the trouble. A check that flags only the single most suspicious payment of the day could have a precision of 0.9 and a recall of 0.1: nearly always right when it speaks, and nearly always silent. Its plain average is 0.5, as good-sounding as a check with 0.5 for both.
The fix is the harmonic mean, an average that a small number drags down hard. For two numbers it's twice their product divided by their sum, and for precision and recall it has its own name, the F1 score:
F1 = 2 × precision × recall / (precision + recall)
def f1(precision, recall):
return 2 * precision * recall / (precision + recall)
print("Balanced:", round(f1(0.5, 0.5), 3))
print("Lopsided:", round(f1(0.9, 0.1), 3))
print("Plain average of 0.9 and 0.1:", (0.9 + 0.1) / 2)
Balanced: 0.5
Lopsided: 0.18
Plain average of 0.9 and 0.1: 0.5
The lopsided check scores 0.18, not 0.5. F1 runs from 0 to 1, and it's high only when precision and recall are both high. (Your textbook may call it the F-measure or F-score; the 1 means precision and recall count equally.)
f1_score in scikit-learn
For the chapter's fraud check, F1 = 2 × 0.667 × 0.4 / (0.667 + 0.4), which is 0.5 (exactly 0.5 with the precision unrounded, 8 ÷ 12). f1_score takes the actual classes first, like the other metrics:
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score
from sklearn.model_selection import train_test_split
payments = pd.read_csv("transactions.csv")
features = ["amount", "hour", "miles_from_home", "new_merchant", "online"]
train, test = train_test_split(
payments, test_size=0.2, random_state=42, stratify=payments["is_fraud"]
)
model = LogisticRegression()
model.fit(train[features], train["is_fraud"])
guesses = model.predict(test[features])
print("F1:", f1_score(test["is_fraud"], guesses))
F1: 0.5
The same 0.5. Straight from the counts, F1 is also 2 × TP / (2 × TP + FP + FN): 16 ÷ (16 + 4 + 12) = 16 ÷ 32. A check that flags nothing has no precision at all (0 ÷ 0), but its F1 still has an answer: with frauds to find, 2 × 0 ÷ (0 + 0 + FN) is 0, and f1_score gives 0.0 with no warning. Only when there is nothing to find and nothing flagged is F1 itself 0 ÷ 0; scikit-learn then reports 0.0 with a warning, and zero_division sets that number, as it does for precision_score.
The classification report
classification_report prints the whole picture at once:
from sklearn.metrics import classification_report
print(classification_report(test["is_fraud"], guesses, target_names=["honest", "fraud"]))
precision recall f1-score support
honest 0.99 1.00 0.99 980
fraud 0.67 0.40 0.50 20
accuracy 0.98 1000
macro avg 0.83 0.70 0.75 1000
weighted avg 0.98 0.98 0.98 1000
Read it a row at a time:
- Each class gets its own row, as if that class were the positive one. The
fraudrow is the precision, recall and F1 you know. Thehonestrow asks the same questions about honest payments: of those the check called honest, 0.99 were, and it called 1.00 of the honest ones honest (976 of 980, rounded). supportis how many test payments really belong to each class: 980 honest and 20 fraud.target_namesnames the classes in sorted order, 0 then 1.accuracyis the 98.4% from Accuracy, rounded to 0.98.macro avgis the plain average of the two class rows: (0.40 + 1.00) ÷ 2 = 0.70 for recall. Every class counts the same, however small.weighted avgweights each class by its support, so the 980 honest payments count 49 times as much as the 20 frauds, and every number comes out 0.98.
When one number is the wrong summary
The weighted average says 0.98 across the board for a check that misses 12 of 20 frauds. Whenever one class is much bigger, any number that weights by size mostly describes the big class. Look at the row for the class you care about.
F1 has a blind spot of its own: it treats a false alarm and a missed fraud as equally bad. Precision and Recall found the 12 missed frauds cost $1,661.51, far more than 4 phone calls. When mistakes cost different amounts, F1 is still a fair way to compare models in general, but the decision comes from precision and recall themselves, and from where you put the threshold (Moving the Threshold).
Do it: work out a fraud check's F1 score
The fraud team wants one number for each check it compares, the F1 score, worked out from the counts in each check's confusion matrix.
The inputs:
precisionandrecall: a check's precision and recall, each between 0 and 1.tp,fp,fn: a check's true positives, false positives and false negatives.
For example, a check with tp = 6, fp = 2 and fn = 4 has a precision of 6 / 8 = 0.75 and a recall of 6 / 10 = 0.6:
- F1 = 2 × 0.75 × 0.6 / (0.75 + 0.6) = 0.9 / 1.35 =
0.667(rounded to 3 places) - from the counts, 2 × 6 / (2 × 6 + 2 + 4) = 12 / 18, the same
0.667
Your job: write two functions.
f1(precision, recall)returns 2 × precision × recall / (precision + recall).- If precision and recall are both 0,
f1returns0.0. f1_from_counts(tp, fp, fn)returns the F1 score from the three counts: 2 × tp / (2 × tp + fp + fn).- If
tp,fpandfnare all 0,f1_from_countsreturns0.0.
Practiced live on the lesson page: write the Python right there and check it.
Check yourself
2 questions to try on the lesson page.
So what
The F1 score is a fair one-number summary of precision and recall because it can't be fooled by one of them alone, and the classification report lays every class side by side, so you can see when a high average is only the big class talking.
Sources
- scikit-learn, Metrics and scoring: Precision, recall and F-measures, User Guide: the F-measure is the weighted harmonic mean of precision and recall, and F1 is the case where precision and recall count equally; F1 for a set of only true negatives is 0 by default, which
zero_divisionchanges. - scikit-learn 1.9.1, f1_score, API reference: the F1 score is the harmonic mean of precision and recall, 2 × (precision × recall) / (precision + recall), with a best value of 1 and a worst of 0.
- scikit-learn 1.9.1, classification_report, API reference: a text report of the main classification metrics; the reported averages include the macro average (the unweighted mean per label) and the weighted average (the support-weighted mean per label);
target_namesare display names matching the labels, in the same order.