Harm Taxonomy and Risk Scenarios

A hazard becomes a harm. But only after exposure, vulnerability and a failed barrier have each played their part.

The Dutch childcare benefits scandal ran the whole chain to its end, and every link in it is on the record. The hazard was a self-learning risk classification model, adopted in 2013, that used nationality as an indicator and automatically flagged applications as risky. The exposure was everyone who applied for childcare benefit. The vulnerability sat in the Tax Administration's own systems: in May 2018 some 1.4 million people were still registered there as dual nationals. The barrier that failed was the one that should have asked whether that input was lawful at all. It did not hold until 7 December 2021, when the Dutch Data Protection Authority fined the Tax Administration EUR 2.75 million. The regulator wrote that day: “The fine was imposed because for many years the Tax Administration processed data on the (dual) nationality of childcare benefit applicants in an unlawful, discriminatory and therefore improper manner.”

The consequence is the part no diagram box can hold. Tens of thousands of parents and caregivers were falsely accused of fraud, as Amnesty International documented in Xenophobic Machines. The scandal brought down the Dutch Cabinet in 2021.

A harm is a negative consequence experienced by people, institutions, society, or the environment. A model error is only one possible initiating event. Risk scenarios connect hazards, exposure, vulnerability, controls and consequences rather than listing abstract values. Exposure does the separating. A model error that reaches nobody stays a defect; the same error reaching someone who acts on it is where the harm starts.

The vocabulary for the consequence end of that chain is not something a team has to improvise. One cut of it was published in 2023, built by reading the literature rather than by brainstorming: “Based on a scoping review of computing research (n=172), we present an applied taxonomy of sociotechnical harms to support a more systematic surfacing of potential harms in algorithmic systems.” Renee Shelby and her co-authors sort what they found into five themes — representational, allocative, quality-of-service, interpersonal, and social system/societal harms.

A second published taxonomy cuts the same territory differently, which is the useful thing about having two. DeepMind's 2022 paper counted instead of grouping: “We identify twenty-one risks, drawing on expertise and literature from computer science, linguistics, and the social sciences.” Those twenty-one sit in six risk areas — Discrimination, Hate speech and Exclusion; Information Hazards; Misinformation Harms; Malicious Uses; Human-Computer Interaction Harms; Environmental and Socioeconomic harms. For each observed risk the paper also sets out the causal mechanism leading to harm. That is the step that turns a category into a scenario.

Either list earns its length when it is run against one system. An automated debt notice begins as an allocative harm. For the household that pays it, it ends as an economic one. A risk score built on nationality is representational before anybody's benefit is suspended. Severity, scale, duration, reversibility, detectability and distribution all affect priority.

Robodebt raised debts against Australian social security recipients by averaging their income. It ran until mid-2020, and it stopped then only because the Federal Court found it unlawful. The court recorded the size of it: at least A$1.763 billion in debts unlawfully asserted against approximately 433,000 Australians, and approximately A$751 million already recovered from about 381,000 of them. The Commonwealth undertook to withdraw and refund all of it.

The Royal Commission into the Robodebt Scheme reported on 7 July 2023, under Commissioner Catherine Holmes. Its verdict is the passage to keep beside any harm taxonomy: “Robodebt was a crude and cruel mechanism, neither fair nor legal, and it made many people feel like criminals. In essence, people were traumatised on the off-chance they might owe money. It was a costly failure of public administration, in both human and economic terms.”

Error taxonomies, harm taxonomies and risk scenarios sort the same incident three different ways. Only one of them explains how it travelled.

Run Robodebt through all three. An error taxonomy asks what the system got wrong, and finds an averaging method applied as designed. On that sorting the scheme has almost nothing to report, and approximately 433,000 people never appear. A harm taxonomy files the same events under allocative and social system/societal harms — the sorting Shelby and her co-authors built from 172 computing papers. It can hold a debt that looked lawful and was unjust. Only the third sorting explains the route: every recipient in the dataset exposed, a barrier that did not exist, and a sequence that ran until a court stopped it. That is why the DeepMind taxonomy attaches a causal mechanism to each of its twenty-one risks rather than leaving them as labels. A label tells you what happened. A mechanism tells you how it reached someone.

This review starts with the people who carry the consequence, not with the model that set it off. Begun that way, a Robodebt review opens on the approximately 433,000 people the scheme would reach. It asks what a wrongly averaged debt does to a household, before anyone opens the error log. Begun from the model instead, it opens on a method that was working exactly as specified.

Step three is the one that takes the labour, and it has a published template. The DeepMind taxonomy states a causal mechanism for each of its twenty-one risks. That is the same object this review asks you to write for your own system: the exposure, the vulnerability, the control that would have to fail, and the downstream action somebody takes on the output.

Write the chain, keep the near miss, count who pays for the recovery, and name one harm the telemetry cannot see.

Aviation built a standing institution for the second of those, and its design is the instructive part. The FAA's Aviation Safety Reporting Program, instituted on 30 April 1975, routes reports to NASA rather than to the FAA. That routing is deliberate, and it is the reason near misses are reported at all. The advisory circular governing the program says why: “The effectiveness of this program in improving safety depends on the free, unrestricted flow of information from the users of the NAS.” NASA's Aviation Safety Reporting System had received 2,068,784 incident reports between April 1976 and December 2023. By late 2024 the total had passed 2.1 million — over 120,000 a year, every one read by an expert analyst within 3-5 days. That is what a near-miss register looks like when the people filing it are not being asked to incriminate themselves.

Harm analysis repays its cost only when the scenarios are written down, reopened after incidents, and allowed to change the design. The test of whether an organisation has that capacity is who ends up making the stop decision. Robodebt was stopped in mid-2020 by the Federal Court, not by its operators. The Dutch risk classification model was stopped by a regulator, with a EUR 2.75 million fine and a finding of unlawful, discriminatory processing. In both cases the scenario was writable years earlier, by people inside. Either it was not written, or it did not bind.

What a binding scenario looks like afterwards is also on the record. The Royal Commission reported on 7 July 2023. The Australian Government's response of 13 November 2023 accepted or accepted in principle all 56 recommendations. That is the cost of learning the lesson from a commission of inquiry rather than from a design review — for a scheme the commission called “a costly failure of public administration, in both human and economic terms”. Define, before release, when harm and risk scenario analysis requires the team to redesign, restrict, remedy, or retire the system. Name who is allowed to invoke it.