Explainability: Intrinsic, Local, Global, and Counterfactual

Explanations come in five kinds, from the structure of the model itself to the procedure around the decision. They are not interchangeable. The last kind is not a simpler register for readers who cannot follow the mathematics. It is the one a binding European regulation reaches for, and the one an affected person can act on.

A clinician asks why a patient was flagged. The flag in the wards is not hypothetical. The Epic Sepsis Model is a proprietary sepsis prediction model deployed at hundreds of US hospitals. In 2021 JAMA Internal Medicine published an external validation of it, run at Michigan Medicine over 27,697 patients and 38,455 hospitalizations between December 2018 and October 2019. Sepsis occurred in 2,552 of those hospitalizations, 7% of the total.

The numbers came back poor. Area under the curve 0.63 (95% CI, 0.62–0.64). Sensitivity 33%, specificity 83%, positive predictive value 12%. The model raised alerts on 18% of all hospitalizations and still missed 1,709 of the 2,552 sepsis patients, 67% of them. At an ESM score of at least 6, the number needed to evaluate was 8. Wong and colleagues put the finding in one sentence: “This external validation cohort study suggests that the ESM has poor discrimination and calibration in predicting the onset of sepsis.”

Ask the vendor of a model like that why this patient was flagged, and what comes back is a global feature-importance chart. Average model behaviour. Not the patient’s case, and not the reliability of the flag.

Explainability methods answer different questions. A reader has to know which question is being answered before the answer means anything. How does the model work globally? Which inputs influenced this one output? What examples are similar? What change would alter the result? How did the process and the data produce the decision? The question should choose the method. An intrinsic explanation reads the structure of the model itself. A post-hoc method builds a second, simpler account of how the first one behaved, after it has already been trained. Local attribution, global importance, surrogates, counterfactuals, examples, concepts, and process traces differ in scope, fidelity, stability, actionability, and vulnerability to misuse.

That line between intrinsic and post-hoc is not a neutral taxonomy. It carries a stake, and in 2019 Cynthia Rudin named the stake in Nature Machine Intelligence: stop explaining black box machine learning models for high stakes decisions, and use interpretable models instead. A second, simpler account fitted after training is not the model, she holds. Systems making high-stakes decisions should be designed to be inherently interpretable. The paper puts the risk plainly: “People have hoped that creating methods for explaining these black box models will alleviate some of these problems, but trying to explain black box models, rather than creating models that are interpretable in the first place, is likely to perpetuate bad practices and can potentially cause catastrophic harm to society.”

The clinician facing an ESM alert asks a local question and receives global importance. That is how a chart can be accurate and useless in the same moment.

The question the user is actually asking comes first, and the method comes last. Reverse those two and the project ends up holding an explanation that nobody can use.

Step four is the one usually left as a list of nouns, so here is a test with a published result. Randomization-based sanity checks for explanation methods come in two forms, a model parameter randomization test and a data randomization test. Adebayo and colleagues proposed them in 2018 and ran them on an Inception v3 model trained on ImageNet. Gradients and GradCAM passed. Guided BackProp and Guided GradCAM did not. Those two are invariant to higher-layer parameters, and the maps they produce are visually comparable to a plain edge detector, which uses neither the model nor the training data. The abstract says it flatly: “Through extensive experiments we show that some existing saliency methods are independent both of the model and of the data generating process.” A method can fail that badly and still return a picture the reviewer finds convincing. That is why step four is a test and not an impression.

Attribution, counterfactuals, and process explanation serve different audiences. The one most useful to an affected person often contains nothing computed by the model at all.

The counterfactual register has a founding example, and it is two sentences long: “You were denied a loan because your annual income was £30,000. If your income had been £45,000, you would have been offered a loan.” Those lines open a section of Wachter and colleagues’ 2018 paper in the Harvard Journal of Law & Technology. Nothing in the two sentences reports the internal logic of the system. A counterfactual describes a dependency on the external facts that led to a decision. The “closest possible world” the authors set out is the smallest change to the world that obtains a desirable outcome.

The process register has a binding instrument. Article 86 of the EU AI Act, Regulation (EU) 2024/1689, is titled "Right to explanation of individual decision-making". Article 86(1) reads: “Any affected person subject to a decision which is taken by the deployer on the basis of the output from a high-risk AI system listed in Annex III, with the exception of systems listed under point 2 thereof, and which produces legal effects or similarly significantly affects that person in a way that they consider to have an adverse impact on their health, safety or fundamental rights shall have the right to obtain from the deployer clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision taken.” The role of the system in the procedure, and the main elements of the decision. No attribution, no saliency map, no coefficient.

Take five real stakeholder questions and watch how quickly the method choice stops being a technical one.

Explanation methods are chosen against a question. So the question asked, what the method cannot do, and what the reader is allowed to conclude from it all belong together in the record.

There is a named standard to write that record against, rather than a house convention. NIST set out four principles for explainable AI in a 2021 report, NISTIR 8312: “We have termed these four principles as explanation, meaningful, explanation accuracy, and knowledge limits, respectively.” A system delivers accompanying evidence or reasons. Explanations are understandable to the intended consumer. Explanations correctly reflect the system’s process for generating the output. And the system operates only under the conditions for which it was designed, and only when it reaches sufficient confidence in its output. Meaningful and Explanation Accuracy are kept apart on purpose, and Knowledge Limits is a principle in its own right. That is the vendor chart’s failure written as a standard. It is the 0.63 area under the curve written as one too.

Set the fidelity, stability, or user-study result that would force the reviewer to redesign, restrict, remedy, or retire the system.