Bias Is More Than Biased Data

Dataset balancing changes the training data. Model debiasing changes the objective. Process redesign changes the decision that was being made in the first place.

The third repair has a dated public record. On 17 August 2020 the UK Government and Ofqual withdrew the standardisation model used to award GCSE, AS and A level grades, and awarded centre assessment grades instead. Ofqual's own later evaluation put figures on what the withdrawn model had done. For A level entries the calculated grade matched the teacher's grade in 59% of cases. It came out lower in 39%, and higher in just over 2%. The Secretary of State for Education, Gavin Williamson, said that day: “We worked with Ofqual to construct the fairest possible model, but it is clear that the process of allocating grades has resulted in more inconsistency and unfairness than can be reasonably resolved through an appeals process.”

Note what was not done. Nobody rebalanced the entry data. Nobody added a fairness constraint to the objective. The decision rule itself was retired, by the body that had the authority to retire it. That authority is the resource process redesign needs and the other two repairs do not. The Office for Statistics Regulation reviewed the episode in March 2021 and found that all four UK regulators' approaches had failed to command public confidence. Its lesson: “a statistical model might not always be the best approach to meet your need”.

The disadvantage a recruiting model would learn is measurable before the model exists. Matched résumés measured it. An audit study published in the American Journal of Sociology in 2007 sent 1,276 of them to 638 employers over 18 months. Childless women received 2.1 times as many callbacks as equally qualified mothers: 6.6% against 3.1%. A paired laboratory experiment ran alongside the audit. Mothers were recommended for hire 47% of the time, non-mothers 84%. Mothers were rated about 10% lower on competence, and offered starting salaries $11,000 (7.4%) lower. The authors' own reading of the audit arm: “The results suggest that real employers do discriminate against mothers.” No algorithm took part in any of it.

Now train a screening model on decisions of that kind. Balance the dataset by gender. Check that aggregate accuracy by gender comes out level. It can. The model will still favour candidates with uninterrupted career histories. That is the pattern the process it learned from rewarded.

Bias is a systematic tendency. It can arise from history, institutions, measurement, sampling, labeling, modeling, interfaces, human judgment, or deployment. Harmful bias can persist when data are balanced and protected attributes are absent. Naming which of the nine sources is at work is what decides whether the repair belongs to the dataset or to the process.

The cleanest measured case of that sits in a hospital. A 2019 study in Science dissected a commercial health-risk algorithm affecting millions of patients. Its data were not unbalanced. The fault was in the target: the system used health costs as the proxy for health need. Correcting that target variable would have raised the share of Black patients identified for extra care from 17.7% to 46.5%. Same algorithm class, same patients, a factor of 2.6 in who gets care — turning on which quantity the model was asked to predict. The authors close their abstract with the general point: “We suggest that the choice of convenient, seemingly effective proxies for ground truth can be an important source of algorithmic bias in many contexts.”

NIST distinguishes systemic, computational and statistical, and human-cognitive sources of bias. The categories interact. An institutional rule shapes the labels. The model amplifies the pattern. Operators then read the output through expectations they already hold.

Before any mechanism is traced, name the benefit or burden being allocated. “Biased” is a claim about an allocation rather than about a matrix.

Step 1 has a worked example on a federal docket. iTutorGroup's tutor-application software was programmed to reject female applicants aged 55 or older and male applicants aged 60 or older. The EEOC sued the company on 5 May 2022. Every slot in the allocation is filled. The benefit is a tutoring job. The burden is automatic rejection at the screening step. The affected group is a class of more than 200 applicants, and the consequence was decided rather than argued. A consent decree entered 8 September 2023 carried $365,000, a five-year term and a bar on requesting dates of birth before an offer. It was the agency's first settled hiring-discrimination suit over recruitment software. Timothy Riera, Acting Director of the EEOC's New York District Office: “Everyone loses when employers engage in age discrimination. Hundreds of applicants lost out on employment during a difficult time for job seekers, and iTutorGroup's students lost the opportunity to learn English from highly qualified and experienced tutors.”

Step 5 has one too. Run PredPol's published algorithm on Oakland drug-arrest records and it flags neighbourhoods that already carried about 200 times more drug-related arrests than the rest of the city. Lum and Isaac did exactly that in 2016 and reported it in Significance. The model would target Black residents at roughly twice the rate of white residents, and other non-white residents at about 1.5 times the white rate. Survey-estimated drug use was roughly equal across racial groups. Patrols follow the flags. Arrests follow the patrols. Those arrests are the records the next fit is trained on. In their own phrase, the system reinforces what is already in the data “rather than correcting for the apparent biases in the police data”. That is what step 5 is watching for, and it is why steps 1 and 5 belong to the same review.

Systemic, statistical, cognitive, interactional: four sources of bias feed one allocation decision, and the remedy is chosen in a fifth layer of law, alternatives and stakeholder goals.

Two of those layers now meet in a statute. The EU AI Act of 2024 writes them down article by article. Article 10(2)(f) requires that training, validation and testing data sets for high-risk AI systems undergo “examination in view of possible biases”. Article 10(2)(g) requires measures to detect, prevent and mitigate any biases so identified. Article 10(5) permits the processing of special-category personal data strictly for bias detection and correction. That is the statistical node made mandatory.

Article 14(4)(b) reaches the cognitive node by name. Human-oversight measures must enable the person assigned to oversight “to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias), in particular for high-risk AI systems used to provide information or recommendations for decisions to be taken by natural persons”. Automation bias here is not a bullet in a training deck. It is a numbered article of a regulation. That is what the remediation layer looks like once a legislature has written in it.

Take one observed disparity and write down every mechanism that could have produced it, before proposing a single fix. Use a real one: the flagged Oakland neighbourhoods carrying about 200 times more drug-related arrests than the rest of the city.

A bias claim needs a stated baseline. A disparity is always measured against some standard, and leaving that standard unwritten hides the whole argument. Every anchored case in this lesson has one and says so. The Science study measured against health need instead of health cost, and the distance between the two baselines was 17.7% against 46.5%. Lum and Isaac measured against survey-estimated drug use, roughly equal across racial groups, rather than against an arrest record concentrated about 200 times over. The résumé audit measured against equally qualified applicants who differed in one thing, whether they were mothers: 6.6% against 3.1%.

Then define when bias across the AI lifecycle requires the team to redesign, restrict, remedy, or retire the system. All four have been done on the record. Ofqual and the UK Government retired a model on 17 August 2020. The consent decree of 8 September 2023 restricted a screening system and remedied the harm, at $365,000 for more than 200 applicants. Article 10(2)(f) of the AI Act requires the examination that is supposed to find these before deployment. Article 14(4)(b) requires that the person watching the output be kept aware of the pull to defer to it.