Fairness Mitigation Across Data, Models, Decisions, and Institutions

A commercial health-risk algorithm affecting millions of patients had a defect, and the defect was nowhere near the cut-off. The system had been trained to predict health-care cost rather than illness. So the score being thresholded was a prediction of spending. Every adjustment to where the line fell still ranked patients by spending. The dissection ran in Science in 2019, by Obermeyer and colleagues. Their abstract states what a change of label alone would have done: “Remedying this disparity would increase the percentage of Black patients receiving additional help from 17.7 to 46.5%.” That is 28.8 percentage points. The choice of what to predict moved all of it. The threshold moved none.

Diagnose the mechanism before choosing a method. A threshold fix applied to a labeling problem moves the disparity somewhere less visible.

Step four is not a formality. In 2023 the Supreme Court of the United States held, in Students for Fair Admissions v. Harvard, that Harvard's and UNC's race-conscious admissions programmes violate the Equal Protection Clause of the Fourteenth Amendment. The same opinion left the framing layer open. Chief Justice Roberts wrote for the Court: “At the same time, as all parties agree, nothing in this opinion should be construed as prohibiting universities from considering an applicant’s discussion of how race affected his or her life, be it through discrimination, inspiration, or otherwise.” In that jurisdiction, a group-conditioned decision rule and a change to what an applicant is asked to write are not merely differently effective. They are differently lawful. Step four has a citation to reach for rather than a worry to express.

Fairness mitigation should aim at whatever mechanism is actually creating the unjustified harm. An intervention can change how the data are collected, which labels are used, which features are read, which objectives are optimised, how the inputs are represented. It can change thresholds, workflow, how resources are shared out, support, or remedy. Pre-processing improves coverage or reweights examples. In-processing adds constraints or adversarial objectives. Post-processing alters thresholds or decisions. Institutional redesign changes eligibility, support, capacity, and recourse. Each of them moves benefits, errors, and burden differently.

Those interventions run roughly from the cheapest change to the hardest. The reflex to stop at the cheap end is documented rather than merely suspected. NIST Special Publication 1270, from 2022, identifies three categories of AI bias — systemic, statistical and human — and names the habit outright: “Likely due to expectations based on techno-solutionism and a lack of mature AI process governance, organizations often default to overly technical solutions for AI bias issues.” A national standards body is saying that the technical fix gets shipped while the systemic and institutional sources get overlooked.

The health-risk algorithm is that sentence with numbers attached. The technical layer was available, cheap and useless. The label was where the 17.7% to 46.5% was hiding.

Framing, data, training, decision policy, institution: five places to intervene, and the most effective one is often the furthest from the model.

The outermost box is not decorative, and it is reached late. The Dutch Tax Administration used (dual) nationality as an indicator in a system that automatically flagged childcare-benefit applications as risky. In May 2018 some 1.4 million people were still registered as dual nationals in its systems. On 7 December 2021 the Dutch Data Protection Authority fined the Tax Administration EUR 2.75 million. The processing had been unlawful, discriminatory and improper. The authority's chair, Aleid Wolfsen, put the penalty inside a longer process rather than calling it a repair: “Obviously this fine cannot undo any of the harm done. But it is an important step within a broader recovery process.” That is the institution-and-remedy layer working exactly as designed. Years downstream, in money, on harm that the framing and data layers had already done.

Pre-processing changes the evidence. In-processing changes the objective. Post-processing changes the decision after the model has already spoken. Two published results put prices on the second and third columns, so the trade-offs stop being adjectives.

In-processing. Adversarial debiasing was applied to the UCI Adult census dataset in 2018, and the headline barely moved: “We notice that debiasing has only a small effect on overall accuracy (86.0% vs 84.5%), and that the debiased model indeed (nearly) obeys equality of odds”. That is Zhang and colleagues. The aggregate held. The burden shifted. Women's false-positive rate rose from 0.0248 to 0.0647 while men's fell from 0.0917 to 0.0701. Men's false-negative rate rose from 0.3667 to 0.4349. Somebody absorbed the constraint, and only the disaggregated table says who.

Post-processing and workflow. One fixed credit score, four fairness criteria, four different prices in lender profit. Hardt and colleagues post-processed a sample of 301,536 TransUnion TransRisk (FICO) scores from 2003 and reported in 2016: “A race blind threshold gets 99.3% of the maximal profit, equal opportunity gets 92.8%, equalized odds gets 80.2%, and demographic parity only 69.8%.” The adjustment also requires the protected attribute at decision time. The method's own authors state that condition. It is the one the third column has to live with.

Force at least one option at every layer, then say plainly who absorbs the extra errors, delay and review that the chosen one creates. Published tables answer that question. Headline accuracy and headline fairness metrics do not.

A mitigation moves burden as well as error, so the monitoring plan has to watch the group that was doing fine before the intervention.

State the rollback conditions in advance: what would force the team to redesign, restrict, remedy, or retire the system. Ofqual shows what happens when they are not stated. Of 718,276 A level entries in summer 2020, the standardisation model adjusted 35.6% down by one grade, 3.3% down by two and 0.2% down by three or more. That is 39.1% downgraded against 2.2% raised. The same interim report, published on 13 August 2020, carries a reassuring headline: “Overall, 96.4% of A level and 91.5% of AS grades awarded were the same as, or within 1 grade, of that submitted by the centre.” Both numbers are true. The aggregate is the one that reads as safety. On 17 August 2020 the model was abandoned. The Office for Statistics Regulation later recorded that “The grades in all four countries were re-issued based on the grades that schools and colleges had originally submitted”, and found that the models' limitations had not been fully communicated. Four days is what an undefined rollback criterion costs.

The law orders these layers rather than merely listing them. Article 9(5) of the EU AI Act governs how risk management measures are chosen for high-risk systems. First, eliminate or reduce the identified risk through adequate design and development, as far as that is technically feasible. Second, where appropriate, add mitigation and control measures for the risks that design alone cannot eliminate. Third, provide the required information and, where appropriate, training to deployers. Informing the deployer comes last. It is a residual step, not a substitute for the first two.