Fairness Trade-offs
When two groups have different base rates, a useful screen can't give them equal selection rates and equal error rates at once, and one whose invitations mean the same in both groups can't have equal error rates either unless it makes no mistakes, as Barocas, Hardt and Narayanan, Chouldechova, and Kleinberg, Mullainathan and Raghavan show, so which goal matters is a choice for people, and sometimes the answer is no model.
Freelancer · AI Basics · Python · Updated
After this lesson you can
- Show with small numbers why different base rates keep a useful screen from giving equal selection rates and equal error rates at once
- Explain the trade-off between equal precision and equal error rates, why choosing a fairness goal is for people, and when not to use a model
Key takeaways
- A group's base rate is the share of it that's truly positive, such as the share of a district's applicants who meet the standard.
- With different base rates, equal selection rates and equal error rates pull apart. Only a screen that ignores who's qualified can have both.
- With different base rates, a screen whose invitations are right equally often in each group can't also have equal true and false positive rates, short of perfect.
- No metric decides which goal matters. That's a choice for the people responsible and the people affected, and sometimes the honest answer is not to use a model.
The Summit Scholarship Fund's board has the gaps from Measuring Results Group by Group, and its instruction is short: "Make the screen fair." It sounds like one job. It's several, and some of them can't all be done at once. Whenever a model sorts people, someone has to choose which kind of fair matters most, and this lesson shows why that choice can't be handed to the model.
Three things fair can mean
Each is reasonable, and the textbook Fairness and Machine Learning (Barocas, Hardt and Narayanan) names each one:
- The same share invited from each district: demographic parity, from the last lesson. The book's word is independence.
- The same chances for the qualified, and for the unqualified, in each district: equal true positive and false positive rates, equalized odds. The book's word is separation.
- An invitation means the same in each district: of those invited, the same share qualified, which is equal precision (from Precision and Recall). For a screen that gives each applicant a chance, it's calibration by group: among applicants given a 70% chance, about 70% in each district are qualified. The book's word is sufficiency.
A group's base rate is the share of it that's truly positive: here, the share of a district's applicants who meet the standard. The examples below use two made-up districts of 100 applicants each: North's base rate is 0.5, South's 0.3.
Equal invitations or equal error rates
Three screens, each one judged on both goals. The first knows exactly who's qualified and invites them all. The second must invite 40 from each district, and picks qualified applicants first. The third ignores qualifications completely: it invites 40% of each district's qualified applicants and 40% of the rest, as a lottery would on average:
| Screen | District | Invited | Selection rate | True positive rate | False positive rate |
|---|---|---|---|---|---|
| Invite the qualified | North | 50 | 0.5 | 50 ÷ 50 = 1.0 | 0 ÷ 50 = 0.0 |
| Invite the qualified | South | 30 | 0.3 | 30 ÷ 30 = 1.0 | 0 ÷ 70 = 0.0 |
| 40 from each | North | 40 | 0.4 | 40 ÷ 50 = 0.8 | 0 ÷ 50 = 0.0 |
| 40 from each | South | 40 | 0.4 | 30 ÷ 30 = 1.0 | 10 ÷ 70 = 0.143 |
| 40% at random | North | 40 | 0.4 | 20 ÷ 50 = 0.4 | 20 ÷ 50 = 0.4 |
| 40% at random | South | 40 | 0.4 | 12 ÷ 30 = 0.4 | 28 ÷ 70 = 0.4 |
The first screen meets equalized odds and misses demographic parity. The second meets demographic parity, and even at its best turns away 10 qualified North applicants while inviting 10 unqualified South ones. Only the random screen meets both, because it ignores who's qualified, which makes it useless.
The book proves the pattern in general (its Proposition 3): when the outcome has two values, the base rates differ, and the screen carries any information about the outcome, independence and separation can't both hold.
Equal precision or equal error rates
Now the board asks for invitations that mean the same in both districts, and equal chances for the qualified. This screen invites 8 in 10 of each district's qualified applicants, and 8 in 10 of its invitations go to qualified applicants:
for district, applicants, qualified in [("North", 100, 50), ("South", 100, 30)]:
tp = qualified * 8 // 10 # 8 in 10 of the qualified are invited
invited = tp * 10 // 8 # and 8 in 10 of those invited are qualified
fp = invited - tp
print(district, "invites", invited)
print(" true positive rate:", tp / qualified)
print(" precision:", tp / invited)
print(" false positive rate:", round(fp / (applicants - qualified), 3))
North invites 50
true positive rate: 0.8
precision: 0.8
false positive rate: 0.2
South invites 30
true positive rate: 0.8
precision: 0.8
false positive rate: 0.086
The same true positive rate (so the same false negative rate, 0.2) and the same precision, and still the false positive rates differ: 0.2 against 0.086. The base rates force it. Alexandra Chouldechova (2017) showed that when base rates differ between two groups, a calibrated score (her word is test-fair) that makes mistakes can't have equal false positive and false negative rates across them. Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan (2016) proved a matching result for risk scores: calibrated within each group, plus the same average score for each group's positives and for each group's negatives, can't all hold unless prediction is perfect or the base rates are equal.
Choosing is for people
So the board has to choose, and each goal protects someone. Equal selection rates give each district the same share of interviews. Equal error rates mean a qualified student's chance doesn't depend on their district. Equal precision means an invitation tells the interviewers the same thing about everyone. And the mistakes cost different people: a qualified student turned away loses a shot at a scholarship; an unqualified one invited takes an interview slot. Weighing those is a judgment about values, for the people responsible together with the people affected. No metric makes it.
Fairlearn, the Python library behind the last lesson's definitions, does this measuring in real projects: MetricFrame computes any metric group by group, and demographic_parity_difference and equalized_odds_difference give the gaps.
Sometimes the honest answer is not to use a model. If the only labels are the old reviewers' decisions and nobody can say who was truly qualified, a model can only repeat the past faster, and a clear written standard that every application is read against may serve the fund better. Many places also have laws about automated decisions; what they require is outside this course, and a real project asks someone who knows them.
Do it: find the best rates when the invitations are fixed
The board wants to see what a fixed number of invitations does to one district's error rates, even for a screen that knew exactly who was qualified and picked them first.
The inputs:
applicants: how many applicants the district has.qualified: how many of them meet the fund's standard. It's at least 1 and fewer thanapplicants.invited: how many the screen must invite, from 0 up toapplicants.
The screen invites qualified applicants first, then fills any places left with unqualified ones. For example, a district of 200 applicants, 80 of them qualified, with 60 invitations:
| Step | Count |
|---|---|
| Qualified applicants invited (true positives) | the smaller of 60 and 80: 60 |
| Unqualified applicants invited (false positives) | 60 − 60 = 0 |
| True positive rate | 60 ÷ 80 = 0.75 |
| False positive rate | 0 ÷ (200 − 80) = 0.0 |
So best_rates(200, 80, 60) returns (0.75, 0.0).
Your job: write best_rates(applicants, qualified, invited).
- It returns a pair: the true positive rate, then the false positive rate.
- Round each rate to 3 decimal places with
round(value, 3).
Practiced live on the lesson page: write the Python right there and check it.
Check yourself
2 questions to try on the lesson page.
So what
Different base rates mean some fairness goals can't all be met, so the work is to measure every rate, say plainly which goal you chose and who pays for the gaps that remain, and be willing to answer "no model".
Sources
- Solon Barocas, Moritz Hardt and Arvind Narayanan, Fairness and Machine Learning: Limitations and Opportunities (MIT Press, 2023), Chapter 3, Classification: independence (equal acceptance rates, demographic parity for a binary classifier), separation (equal true positive and false positive rates) and sufficiency (calibration by group); Proposition 3, "Assume Y is binary, A is not independent of Y, and R is not independent of Y. Then, independence and separation cannot both hold"; Propositions 4 and 5, that separation and sufficiency can't both hold when the outcome isn't independent of the group.
- Alexandra Chouldechova, Fair prediction with disparate impact: A study of bias in recidivism prediction instruments, Big Data 5(2), 153–163, 2017: when prevalence differs between two groups, a test-fair (well-calibrated) score cannot have equal false positive and false negative rates across those groups.
- Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan, Inherent Trade-Offs in the Fair Determination of Risk Scores, 2016 (ITCS 2017): calibration within groups, balance for the positive class and balance for the negative class can all hold only with perfect prediction or equal base rates.
- Fairlearn, Common fairness metrics and Fairness in Machine Learning, User Guide: demographic parity and equalized odds; fairness is a contested, sociotechnical question, mathematical metrics capture only some of it, and people decide which metrics fit the situation; the API reference lists
MetricFrame,demographic_parity_differenceandequalized_odds_difference.