Skip to content
AI.info

Responsible AI

Fairness Criteria and Incompatible Goals

Compare demographic parity, equal opportunity, equalized odds, predictive parity, calibration, and individual fairness with their assumptions and conflicts.

By the end you can

Key idea

What the impossibility result does not say

Impossibility results do not prove that fairness is impossible. They show that certain statistical properties cannot generally be achieved together under specific conditions. That forces an explicit normative and operational choice. Metrics cannot settle legal or moral questions about legitimate grouping, historical injustice, resource constraints, or the desired counterfactual. They support deliberation rather than replace it.

The qualifier that does the work is not vague. There are exactly two escapes from the impossibility, and the theorem names them both. Take a risk assignment that satisfies calibration within groups, balance for the negative class and balance for the positive class. It must either allow perfect prediction — p_sigma equal to 0 or 1 for every feature vector — or have equal base rates across the two groups. That is the whole list. Two cases.

The credit team asked for equal approval rates, equal false-negative rates, calibrated scores and one threshold. Its base rates differ and its model makes errors, so it stands in neither escape. The impossibility result is the reason the answer was no.

Dropping one of the four requirements is not a neutral technical concession; someone is on the losing side of whichever one goes.

Comparison

Demographic parity, Equalized odds, or Calibration / predictive parity?

Demographic parity targets equal outcome rates. Equalized odds targets equal error rates. Predictive parity targets a score that means the same thing everywhere. Where base rates differ, the three cannot all hold at once.

Calibration does not fail gradually either; it fails at a known point. It can be reconciled with exactly one of the two error constraints, never both. Five researchers proved that in 2017. Their abstract puts it in one sentence: “We show that calibration is compatible only with a single error constraint (i.e. equal false-negatives rates across groups), and show that any algorithm that satisfies this relaxation is no better than randomizing a percentage of predictions for an existing classifier.”

So calibration survives alongside one equalized-odds constraint. The survival is bought by withholding predictions at random from a classifier you already have. That is a price a comparison table does not show.

FigureComparison · 3 columns

Demographic parity

Targets equal positive outcome rates.

  • Can address exclusion from opportunity
  • May ignore qualification or need labels
  • Can require different errors by group
  • Needs justification of the desired allocation

Equalized odds

Targets equal false-positive and true-positive rates.

  • Focuses on error distribution
  • Requires trustworthy outcome labels
  • May conflict with calibration when base rates differ
  • Can use group-aware post-processing

Calibration / predictive parity

Targets consistent score or decision meaning.

  • Useful for risk communication and thresholds
  • Can coexist with unequal error rates
  • Sensitive to prevalence and label quality
  • Does not guarantee equal access or burden

The mechanism behind fairness criteria and trade-offs

Fairness criteria formalize different moral and operational questions. To select one, a team needs a legitimate target, a valid label, a defined benefit or harm, and an explicit account of whose errors or opportunities matter. Demographic parity concerns outcome rates. Equal opportunity and equalized odds constrain conditional error rates. Predictive parity concerns the meaning of decisions, and calibration the reliability of a score. Individual fairness depends on a defensible similarity relation.

None is universally correct. The rest of this lesson is the evidence for that sentence. A theorem that says which combinations are unavailable. An identity that turns a prevalence gap into an error gap. Two research teams reaching opposite verdicts on one instrument. And a credit portfolio on which each criterion carries a price.

A team that reaches for equal opportunity or predictive parity without first saying whose errors count has let the formula choose the moral question for it.

Case

Three conditions, two escapes

Two results, from 2016 and 2017, turned this from a preference into a proof. The first writes down three fairness conditions precisely: calibration within groups, balance for the negative class, balance for the positive class. It then shows that no method satisfies all three at once, except in highly constrained special cases. Kleinberg and two coauthors published it in 2016.

The characterization is what makes it usable. The paper says so in one line: “Our first main result establishes that these are in fact the only two cases in which a risk assignment can achieve all three fairness guarantees simultaneously.” The two cases are perfect prediction, with p_sigma equal to 0 or 1 for every feature vector, and equal base rates across the two groups. A team that wants all three guarantees is claiming one of those two things about its own data. It can be asked which.

Case

Predictive parity in a courtroom

The same tension turns up in the instruments courts use to predict reoffending. There it comes down to one line of algebra: FPR = [p/(1−p)]·[(1−PPV)/PPV]·(1−FNR). Chouldechova derived that identity in 2017 and read the consequence straight off it: “A direct implication of this simple expression is that when the recidivism prevalence differs between two groups, a test-fair score Sc cannot have equal false positive and negative rates across those groups.”

On the ProPublica Broward County data the identity has somewhere to bite. Recidivism prevalence there is 51% for Black defendants against 39% for White defendants. COMPAS's error rates are forced apart accordingly: Black false-positive rate 45% and false-negative rate 28%, White false-positive rate 23% and false-negative rate 48%. That is what “considerable disparate impact” denotes in this literature. Not a vague unfairness — a specific pair of numbers, roughly double their counterparts in the other direction.

ProPublica had already published the split, in May 2016: “In forecasting who would re-offend, the algorithm correctly predicted recidivism for black and white defendants at roughly the same rate (59 percent for white defendants, and 63 percent for black defendants) but made mistakes in very different ways.” It measured a 44.85% false-positive rate for Black defendants against 23.45% for White. The false-negative rate ran the other way: 47.72% for White defendants against 27.99% for Black. Accuracy nearly equal. Errors nearly double.

Neither result tells you which criterion to choose. Both mean the same thing for a review.

A team claiming to have satisfied all of them has not read the arithmetic.

Position

A metric chosen after the results is a report about the choice

Choosing the fairness metric after the results are in is the most ordinary way a fairness review goes wrong, and no theorem forbids it. Outside the two special cases, at least one of the three conditions is going to fail. Which failure gets reported then becomes a choice rather than a measurement.

The clearest demonstration is a public dispute over one instrument and one dataset. Flores and two colleagues re-analysed the same Broward County data under psychometric bias standards, and published the rejoinder in Federal Probation in September 2016. It opens: “We think ProPublica’s report was based on faulty statistics and data analysis, and that the report failed to show that the COMPAS itself is racially biased, let alone that other risk instruments are biased.” Their measurements found COMPAS equally predictive by race: AUC-ROC of 0.71 overall, 0.69 for White and 0.70 for Black defendants, with no significant difference. The sample's general rearrest base rate was 47% overall, 39% for White and 52% for Black defendants.

Nobody disputed anybody's arithmetic. One team measured predictive equivalence and found it. The other measured error distribution and found it split. So a criterion that passes is not evidence that nobody was harmed. Here that is not an inference from a theorem but a matter of record: the same instrument passes one team's criterion and fails the other's, on the same defendants.

The defensible order is the one this lesson's own procedure sets out. Name the benefit or harm. Commit to the criterion that concern implies. Then record what the criterion leaves untouched. No paper requires that order, and no metric can be made to supply it. It is the difference between a fairness claim that survives someone else's arithmetic and one assembled from whichever column came out green.

The impossibility results tell you a choice is unavoidable; they do not tell you which one is right.

Visual

Five ways of saying fair

Outcome parity, error parity, predictive value, calibration, individual consistency: five ways of saying fair. Each answers a different question about the same decision.

The fifth arrived with its own caveat attached. Individual fairness was introduced in 2012, by Dwork and four coauthors. Their abstract promises “an algorithm for maximizing utility subject to the fairness constraint, that similar individuals are treated similarly”. The similarity metric that constraint turns on is hypothetical and task-specific. The framework assumes it rather than deriving it.

So individual consistency is not the escape from group statistics it looks like. It moves the normative argument into the definition of similar, where it still has to be made and defended.

FigureProcess · 5 steps
  1. 1

    Outcome parity

    Compare positive decision or resource rates across groups.

  2. 2

    Error parity

    Compare false-positive, false-negative, or true-positive rates conditional on labels.

  3. 3

    Predictive value

    Compare the reliability of positive or negative decisions.

  4. 4

    Calibration

    Compare observed outcome frequency at similar score levels.

  5. 5

    Individual consistency

    Treat meaningfully similar people similarly under a justified metric.

Steps

How to test fairness criteria and trade-offs before release

Validating the target comes second for a reason. A criterion computed on a label that means different things across groups measures the label.

A commercial care-management algorithm affecting millions of patients used future cost as a proxy for illness. Four researchers put a size on what that did, in Science in 2019. Their abstract states the mechanism: “The bias arises because the algorithm predicts health care costs rather than illness, but unequal access to care means that we spend less money caring for Black patients than for White patients.” Simulate an algorithm with no predictive health gap between Black and White patients. The share of Black patients among those auto-identified for the programme at the 97th-percentile threshold rises from 17.7% to 46.5%.

The authors then worked with the manufacturer and changed the training label itself, from future cost to an index combining health and cost prediction. Excess active chronic conditions among Black patients at a given risk score fell to 7,758 — an 84% reduction in bias.

No fairness criterion was chosen, tuned or traded off in either result. The outcome variable was changed, and the numbers moved that far. A review that reaches step three without doing step two is choosing between criteria computed on a construct nobody has checked.

FigureProcess · 5 steps

  1. 1. Define the benefit or harm

    State which outcome, error, burden, or opportunity matters.

  2. 2. Validate the target

    Check whether labels represent the construct fairly across groups.

  3. 3. Choose candidate criteria

    Connect each metric to a defensible normative concern.

  4. 4. Examine conflicts

    Test base rates, thresholds, calibration, capacity, and legal constraints.

  5. 5. Document the decision

    Record selected criteria, rejected alternatives, trade-offs, and monitoring.

Example

What each metric answers, and what it leaves out

Translate each candidate metric into the real-world question it answers. Then write the memo that names the harm your chosen criterion leaves untouched.

  • Metric translation: For each candidate metric, write the real-world question it answers — outcome rates, conditional error rates, the meaning of a score at a given level, or consistency between two people a justified similarity metric calls similar.
  • Conflict simulation: Put your group confusion matrices through the identity FPR = [p/(1−p)]·[(1−PPV)/PPV]·(1−FNR). Hold PPV equal across groups whose p differs, and watch what that does to the false-positive rates. In Broward County a 51% against 39% prevalence gap produced 45% against 23%.
  • Label challenge: Ask whether the outcome in the denominator is equally observable and equally meaningful across groups. Changing that one variable moved a care-management algorithm's Black share at the 97th-percentile threshold from 17.7% to 46.5%.
  • Decision memo: Justify the selected criterion and name the harm it does not address. If the criterion is calibration, only one of the two error constraints can come with it — that is already settled, not a matter of effort.

Example

Four fairness properties, one threshold, no solution

A credit team that wants equal approval rates, equal false-negative rates, calibrated risk scores and one threshold for every group is asking a question that has been answered on a real portfolio. The portfolio is 301,536 TransUnion TransRisk (FICO) scores from 2003, across Asian, White non-Hispanic, Hispanic and Black borrowers. The common 620 cutoff there corresponds to an 82% non-default rate. Hardt and two coauthors ran the criteria across it in 2016. Their paper defined equalized odds as the predictor being independent of the protected attribute conditional on the outcome, and equal opportunity as the weaker constraint on Y=1 only.

Each criterion carries a price: “At 82%, we find that a race blind threshold gets 99.3% of the maximal profit, equal opportunity gets 92.8%, equalized odds gets 80.2%, and demographic parity gets 69.8%.” Base rates differ across those groups and the score is imperfect, so the four requested properties cannot all hold at once. The concession is no longer an abstraction. It is a figure the team has to sign for.

  • Allocation goal: Equal approval rates focus on who receives a beneficial outcome; on the TransRisk scores demographic parity gets 69.8% of the maximal profit.
  • Error goal: Equal opportunity and equalized odds focus on conditional error rates, and they are not interchangeable — 92.8% against 80.2% of maximal profit at the same 82% non-default rate.
  • Score interpretation: Calibration asks whether the same score means similar observed risk, which is what the 620 cutoff assumes when it is applied to every group as though it carried one meaning.
  • Policy constraint: A common threshold may be required or preferred for legal and operational reasons; the race blind threshold gets 99.3% of the maximal profit, which is why it is the option teams drift toward.
  • Trade-off: With differing base rates and imperfect prediction, several criteria conflict. The portfolio sits outside both escapes the impossibility theorem names, so one of the four requests has to go.

When evidence about fairness criteria and trade-offs supports restriction or redesign

Choosing a fairness criterion is a normative decision wearing statistical clothing. It belongs in the record alongside the alternatives that were rejected. The record is what lets a reader do what the Federal Probation rejoinder did to ProPublica: take the same data, apply a different criterion, and see exactly where the two accounts part company. A review that names only its passing criterion has removed the possibility of that check.

Say in advance what would make the team redesign, restrict, remedy or retire the system. The care-management algorithm is the case where redesign was available and worked, because the defect sat in the label rather than in the choice between criteria. The credit portfolio is the case where no redesign removes the conflict. Someone must decide which of 99.3%, 92.8%, 80.2% and 69.8% is the one they are prepared to defend.

Key takeaways