Safety Cases, Hazard Analysis, and Misuse Cases

The function being defended, the hazards that threaten it, the barriers in between, the evidence that those barriers work, and the exposure nobody has removed. Hazard H73 sat in that last box for years. It was carried as an open item in a document that had already been signed off.

Hazard logs track items. Risk assessments score them. Only a safety case has to show why the evidence supports the conclusion. In English health IT that distinction is statutory rather than editorial.

Two NHS England information standards carry it. DCB0129 covers the manufacture of health IT systems; DCB0160 covers their deployment and use. Both were published on 7 June 2018 and took effect on 1 July 2018, under section 250 of the Health and Social Care Act 2012. Between them they require a health IT system to hold a hazard log AND a clinical safety case report, with a registered senior clinician named as Clinical Safety Officer. NHS England defines the second document in the same terms the defence world uses: “The clinical safety case is a structured argument which is supported by a body of relevant evidence that provides a compelling, comprehensible, and valid case that a system is safe for release.”

Two documents are demanded because they do different work. A list of hazards with owners and statuses is not an argument that the system is safe for release. A regulator that wanted only the list would have asked for only the list.

A safety case is a structured, challengeable argument that a system is acceptably safe for a defined use and environment. It links claims to hazards, controls, evidence, assumptions, operational boundaries, residual risks and named decision owners. Assumptions is the entry that ages. A safety case nobody revisits argues about a system that no longer exists.

MCAS was never evaluated as a complete and integrated function in the 737 MAX certification documents. The Joint Authorities Technical Review established that and reported it to the FAA on 11 October 2019. Its 28 members were drawn from the FAA, NASA and nine civil aviation authorities. Its Finding 6 says what the safety analysis had actually been built on: “The MCAS design was based on data, architecture, and assumptions that were reused from a previous aircraft configuration without sufficient detailed aircraft-level evaluation of the appropriateness of such reuse, and without additional safety margins and features to address conditions, omissions, or errors not foreseen in the analyses.” The assumptions were inherited. Nobody owned the question of whether they still held.

Hazard analysis asks how harm can arise: misuse, foreseeable abuse, interaction failure, degraded data, human factors, organizational pressure. Scenario analysis, fault trees, misuse cases, FMEA and STPA each illuminate a different causal structure. Misuse cases are not a loose metaphor. Sindre and Opdahl introduced them in 2000 and developed them in a journal paper five years later. Their definition runs to one line: “A misuse case is the inverse of a use case, i.e., a function that the system should not allow.” That line turns a vague worry about abuse into a sequence somebody has to write down and defend against. None of these methods is a universal proof of safety.

A hospital deploying a clinical AI can point to accuracy, certification and clinician review and still have no safety case. What that gap looks like in practice was measured at Michigan Medicine.

The Epic Sepsis Model was a widely implemented product before anyone outside the vendor validated it. Wong and colleagues did that in 2021, in JAMA Internal Medicine, across 27,697 patients and 38,455 hospitalizations. The hospitalization-level AUC came back at 0.63 (95% CI, 0.62-0.64). The results section of their abstract states the operational consequence: “The ESM also did not identify 1709 patients with sepsis (67%) despite generating alerts for an ESM score of 6 or higher for 6971 of all 38 455 hospitalized patients (18%), thus creating a large burden of alert fatigue.”

A model that misses 67% of the cases it exists to catch, while firing on 18% of all admissions, produces both the hazard path and the reason the control against it will erode.

Build a claim, then attack it. Step 1 bounds it to a given application in a given environment. Step 4 asks for evidence that discriminates — the external validation, not the vendor's headline number. The last step records the conditions that would withdraw the claim: the assumptions with owners, the change triggers, the stop decisions. Skipping it is how 40% of a hazard log stays Open. It is also how architecture and assumptions from a previous aircraft configuration end up load-bearing in a new certification.

Safety cases are arguments with evidence attached, so the argument and its defeaters both have to be kept current. Record the assumption, incident or change trigger that would force the owner to redesign, restrict, remedy or retire the system.

The method has already been carried across to frontier AI. A 2024 paper by Joshua Clymer and three co-authors asks how to justify the safety of advanced AI systems. It opens by naming the decision the argument has to serve: “As AI systems become more advanced, companies and regulators will make difficult decisions about whether it is safe to train and deploy them.” It sorts the available arguments into four categories: total inability to cause a catastrophe, sufficiently strong control measures, trustworthiness despite capability to cause harm, and deference to credible AI advisors.

Hold those four next to the Nimrod finding. Each one can be argued honestly or produced as paperwork. The difference is whether an independent reviewer could have rejected it.