Risk Appetite, Acceptance Criteria, and Stop Rules

Michigan's Unemployment Insurance Agency ran fraud adjudication through MiDAS with essentially no human review between October 2013 and August 2015. The case for the system was aggregate: determinations at machine speed, and money recovered from them. The harm landed on people with no financial buffer. It landed before anyone checked whether the determination was right. Collection ran through wage garnishment and tax-refund seizure while the check was still outstanding.

The numbers arrived from outside, years later. A federal court recorded in 2018 that “the Michigan Auditor General eventually determined that of the 22,427 robo-adjudications reviewed, over 93% did not involve fraud at all”. The Michigan Supreme Court added the human count in 2022. Roughly 40,000 people had been wrongfully accused of unemployment fraud in that period. An Agency study concluded that approximately 93% of the automated system's fraud determinations were incorrect.

None of that is a modelling failure better statistics would have caught in time. The system ran for the best part of two years because no limit had been written down. Nobody outside the programme could halt it.

Risk appetite is the kinds and levels of risk an organization is willing to pursue or retain in service of legitimate objectives. Acceptance criteria turn that position into measurable release conditions. Stop rules keep the authority to pause when the evidence deteriorates. A defensible rule combines quantitative thresholds, qualitative prohibitions, confidence requirements, compensating controls and escalation. It also separates residual risk that leadership knowingly accepted from risk that stayed invisible because nobody measured it.

That distinction is not only an ethical nicety. At least one regulator has made it enforceable, and in the direction organizations least expect. An employer who never collected the required impact data does not get a clean result under the Uniform Guidelines on Employee Selection Procedures. It gets an inference drawn against it. The text is at 29 CFR § 1607.4(D): “Where the user has not maintained data on adverse impact as required by the documentation section of applicable guidelines, the Federal enforcement agencies may draw an inference of adverse impact of the selection process from the failure of the user to maintain such data, if the user has an underutilization of a group in the job category, as compared to the group's representation in the relevant labor market or, in the case of jobs filled from within, the applicable work force.”

Not measuring is not clearance. Under that paragraph the absence of the data is itself evidence.

An aspirational principle states a value. An acceptance criterion states the evidence a decision needs. A stop rule states something that has to happen tonight.

The third column did its work in the Women's Health Initiative. Its estrogen-plus-progestin arm was a randomised primary-prevention trial of 16,608 postmenopausal women aged 50-79 with an intact uterus. Recruitment ran through 40 US clinical centres between 1993 and 1998. Planned duration: 8.5 years. It did not run that long.

The trial's 2002 report in JAMA says why: “On May 31, 2002, after a mean of 5.2 years of follow-up, the data and safety monitoring board recommended stopping the trial of estrogen plus progestin vs placebo because the test statistic for invasive breast cancer exceeded the stopping boundary for this adverse effect and the global index statistic supported risks exceeding benefits.” The hazard ratios, with nominal 95% confidence intervals, were 1.26 (1.00-1.59) for invasive breast cancer on 290 cases, and 1.29 (1.02-1.63) for CHD on 286 cases. NHLBI announced the halt on 9 July 2002.

Nothing in that was improvised. The boundary was fixed before the data existed. The telemetry was an accumulating test statistic. And the board that recommended the halt did not have to go through the people invested in the trial continuing. That is the whole column, working.

Objective, appetite, tolerance, acceptance evidence, stop rule: each row narrows the one above it until a launch becomes decidable. The rows are not an invention of this lesson. NIST named them in the Artificial Intelligence Risk Management Framework, AI RMF 1.0, published in January 2023. Each row there carries an outcome an organization can be held to.

The framework will not fill the tolerance row for you: “While the AI RMF can be used to prioritize risk, it does not prescribe risk tolerance”. What it requires is that the row be filled, and written down. MAP 1.5 is “Organizational risk tolerances are determined and documented”. MEASURE 2.6 asks whether a system's “residual negative risk does not exceed the risk tolerance”. That is the acceptance-evidence row stated as a test. MANAGE 2.4 supplies the authority row, in language stronger than most internal policies manage: “Mechanisms are in place and applied, and responsibilities are assigned and understood, to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use.”

The bottom of the ladder is not a discussion either. Three conditions make a risk level unacceptable: significant negative impacts imminent, severe harms actually occurring, catastrophic risks present. Where an AI system presents them, the framework says “development and deployment should cease in a safe manner until risks can be sufficiently managed.” Cease. Not review.

Classifying consequences is the cheap step. Naming who may invoke a stop rule without commercial permission is the step that gets negotiated away.

The five steps are not new material. Choosing indicators and setting evidence quality are what MEASURE 2.6 tests when it asks whether residual negative risk exceeds the tolerance. Precommitting decisions is what the Women's Health Initiative did. It wrote its stopping boundary before enrolment, not after the breast-cancer statistic started moving. And the last step is the one MANAGE 2.4 puts in the same breath as the mechanism itself: responsibilities are assigned and understood.

Michigan's Unemployment Insurance Agency had a system that could accuse. From October 2013 to August 2015 it had nobody positioned to stop it.

Only the last drill produces a fact rather than a document: invoke the emergency stop and time how long it takes. If that sounds like a formality, the SEC has published what the absence of it costs.

On 1 August 2012, while processing 212 small retail orders, Knight Capital's SMARS router sent millions of orders over a 45-minute period. It obtained over 4 million executions in 154 stocks, for more than 397 million shares, and lost over $460 million. The SEC's order the following year names the control that was missing: “Knight also did not have procedures in place to halt SMARS’s operations in response to its own aberrant activity.” Knight had no supervisory procedures for incident response either. The civil money penalty was $12,000,000.

Why that control is the one that goes missing had been set out the year before. Risk checks cost time: “applying risk controls before the start of a trade can slow down an order”, Carol Clark wrote in a Chicago Fed Letter in 2012. Speed is what the business bought. The halt is what speed is traded against. Clark named the missing piece as well — “A “kill switch” that could stop trading at one or more levels”.

An appetite written after launch describes what already happened. It does not constrain it. The Michigan figures — 22,427 robo-adjudications reviewed, over 93% not involving fraud, roughly 40,000 people wrongfully accused — were produced by an Auditor General, a federal court and a state supreme court. All three were working after the fact. Knight's 45 minutes were reconstructed by the SEC a year later. In both records the arithmetic was eventually excellent, and far too late to be a control.

The alternative is on the table in the same lesson. A boundary set before enrolment, and a board authorised to act on it, ended a trial at a mean of 5.2 years against a planned 8.5. AI RMF 1.0 asks under MAP 1.5 that tolerances be determined and documented. Under MANAGE 2.4 it asks that responsibilities to supersede, disengage or deactivate be assigned and understood. Article 9(8) asks for prior defined metrics and probabilistic thresholds. Article 9(5) asks that the residual risk left over be judged acceptable by someone the record can name.

So define what would make risk appetite and release authority require the board to redesign, restrict, remedy or retire this system. Define it in the tense those instruments use. The future.