Responsible AI as Sociotechnical Governance

Michigan's unemployment agency ran a fraud-detection programme and then reviewed itself. It pulled every fraud-penalty case from October 2013 to August 2015 that was never appealed — 62,784 of them — and sorted them by how the decision had actually been reached.

40,195 had been resolved by computer program alone. 85 percent of those fraud findings were reversed.

22,589 had been initiated by the computer and then referred to a human investigator. 44 percent of those were reversed.

The agency began refunding more than $20.8 million. Its own press release, on 11 August 2017, states the first half of the comparison directly: “Of those cases, 40,195 were originally resolved by way of computer program based on available information. As part of the review, 85 percent of these original fraud findings were reversed.”

No retraining, no new features, no change of threshold separates those two figures. What separates them is whether a person stood between the program and the finding.

Five layers stand between an intended decision and the people who live with its output: purpose and context, technical components, operational workflow, institutional environment, and affected world. Most layer diagrams are an argument. Michigan's review is a measurement.

One detection logic ran through two operational workflows from October 2013 to August 2015. The agency reviewed 62,784 never-appealed cases. Among the 40,195 resolved by computer program alone, 85 percent of the original fraud findings were reversed. Among the 22,589 the computer initiated and then referred to a human investigator, 44 percent were reversed. The technical-components layer accounts for none of the distance between those two numbers.

A layer diagram is a claim about where the variance lives. Here it lived in the workflow layer. It was measured, it was refunded at more than $20.8 million, and it was recited in court as a system that made every fraud determination with no human review.

Responsible AI means managing risks, impacts and accountability with discipline across an AI system's full social and technical context. A model can be accurate while the process around it stays unfair, unsafe, unlawful or ineffective. So the unit of governance is the decision system: objectives, data, model, interface; the users and affected people it reaches; and the policies, incentives, monitoring, recourse and retirement that hold it in place.

A court has drawn the unit exactly there. SyRI was the Dutch welfare-fraud risk-profiling system. On 5 February 2020 the District Court of The Hague struck down the legislation that authorised it. The judgment's own summary heading reads “SyRI legislation in breach of European Convention on Human Rights”. The court holds: “For this reason, the court declares in this judgment that Section 65 SUWI Act and Chapter 5a SUWI Decree have no binding effect, being contrary to Article 8 paragraph 2 ECHR.”

No accuracy figure appears anywhere in that finding. The arrangement was insufficiently transparent and verifiable, and that was enough to void it. Rachovitsa and Johann, in the Human Rights Law Review, read the judgment the same way: the case turns on the Article 8(2) breach and the declaration of no binding effect.

Technical controls matter. They cannot make the institutional choices about who receives benefits, burdens, voice and remedy. People make those.

Model governance, system governance and symbolic ethics all travel under the word responsible. They cover very different amounts of the system.

Model governance controls the model artifact and its measured behaviour — version, evaluation, limits and monitoring, the model card and the release test. It is useful. It is also narrower than the deployed system, because it cannot observe harms outside its data boundary. That boundary has been measured. In Science in 2019, Obermeyer and colleagues dissected a commercial health-risk algorithm that reached millions of patients. It was accurate on the label it had been built to predict: health care cost. It produced large racial bias in the thing the programme actually allocated. Their abstract: “At a given risk score, Black patients are considerably sicker than White patients, as evidenced by signs of uncontrolled illnesses. Remedying this disparity would increase the percentage of Black patients receiving additional help from 17.7 to 46.5%.” Every artifact-level test the algorithm passed, it passed honestly. The harm sat in the choice of target variable, and no measurement of the artifact can report that.

System governance controls the complete decision workflow — model, people, policy and operations together. It tracks downstream consequences and recourse, and it gives someone the authority to stop or redesign. It is the level at which 17.7 against 46.5 percent, or Michigan's 85 against 44, is a governable fact rather than a finding published after the damage.

Symbolic ethics states broad values with no enforceable mechanism. It can align language and aspiration. It often lacks owners and evidence, and it can conceal unresolved trade-offs: a principle statement with no release gate behind it.

The sequence opens by naming the decision and closes by writing the conditions under which the system stops: define the decision; draw the system boundary; map impacts and power; assign accountable owners; set evidence and stop rules.

For some deployers the first three steps are no longer a matter of house style. They are law. The EU AI Act was published on 12 July 2024 and applies from 2 August 2026. Before a public body turns a high-risk system on, it has to assess what that system will do to people's fundamental rights. Article 27(1) reads: “Prior to deploying a high-risk AI system referred to in Article 6(2), with the exception of high-risk AI systems intended to be used in the area listed in point 2 of Annex III, deployers that are bodies governed by public law, or are private entities providing public services, and deployers of high-risk AI systems referred to in points 5 (b) and (c) of Annex III, shall perform an assessment of the impact on fundamental rights that the use of such system may produce.”

Six elements are mandated in that assessment. Three of them are steps 3 and 4 of this sequence written as law: the categories of people likely to be affected, the specific risks of harm to them, and the internal governance and complaint mechanisms to be used if those risks materialise. Annex III point 5(a) covers AI used by public authorities to evaluate eligibility for essential public assistance benefits and services. That is the same family of decision as a benefits agency's fraud determination. Naming who is affected and where they can complain is required content, not good practice.

Four drills, each ending in something a reviewer can hold: a boundary sketch, a power map, a list of gaps, and one written condition for not building at all.

The fourth is not this lesson's invention. NIST makes it an output of the MAP function: “After completing the MAP function, Framework users should have sufficient contextual knowledge about AI system impacts to inform an initial go/no-go decision about whether to design, develop, or deploy an AI system.” The framework lists among its own benefits “explicit processes for making go/no-go system commissioning and deployment decisions”. A national standards body writes the option of not building into the process. A team with no such condition on paper has skipped a documented step, not exercised restraint.

Governance nobody can see, measure or correct is a document. Michigan ran a fraud-detection programme with no human review from October 2013 to August 2015. It took a review of 62,784 never-appealed cases to establish that 85 percent of the findings made by computer program alone were reversed, and that more than $20.8 million had to go back. New York City had a statute, public postings and a named audit obligation; 155 student investigators found 18 posted audit reports across 391 employers. The Hague had a piece of legislation and voided it on transparency and verifiability. In none of the three did the missing thing turn out to be a better model.

So write it down in advance: when does sociotechnical governance require the team to redesign, restrict, remedy or retire the system? NIST puts a go/no-go decision at the end of MAP. MANAGE 4.1 requires appeal, override and decommissioning after deployment. Article 27 of the EU AI Act makes the affected categories and the complaint mechanisms mandatory content for public-service deployers.

Each of those is somebody's job on a named day. A principle is not.