Skip to content
AI.info

Responsible AI

Faithfulness, Stability, and the Limits of Explanations

Evaluate explanation fidelity, sensitivity, stability, manipulation, user reliance, privacy, and the difference between explanation and justification.

By the end you can

Key idea

Where a explanation assurance control can still fail

A clear explanation can increase trust in a bad decision. That is a measured result, not a caution.

Participants worked with New York City apartment-price models that were functionally identical. The models varied in two ways only: the number of features, and whether the model was clear or a black box. Those shown the clear model with few features could better simulate its predictions. They did not follow them more closely when following was beneficial. Apartment 12 was a synthetically generated one-bedroom, three-bathroom apartment, and both models made the same overly high prediction on it. There a one-way ANOVA over the four primary conditions gave F(3, 594) = 7.96, p < 0.001. On apartment 11, the other unusually configured apartment, the difference was not significant. Poursabzi-Sangdeh and colleagues ran this in 2021 as a sequence of pre-registered experiments, N = 3,800. Their abstract states the harm directly: “Furthermore, showing participants a clear model meant that they were less able to detect and correct for the model's sizable mistakes, seemingly due to information overload.” The clearer model made its readers worse at catching it out.

So a user study should measure whether the reliance an explanation produces stays calibrated, whether readers can spot the errors, and whether they act better as a result. Not only whether they liked it. Faithfulness is also method- and claim-specific. Complex models may not admit a single complete explanation, so whoever signs the explanation off should preserve uncertainty and use multiple forms of evidence. The loan system's reasons read well, came from a template, and sometimes named factors the model had not used to reach the denial.

An explanation signed off for one claim carries no assurance for the next one made in its name.

Example

Polished reasons, generated from a template

A loan system produces polished natural-language reasons for every denial. The reasons are generated from a template. They sometimes cite factors that did not materially affect the model output.

Under Regulation B that is not merely poor craft. 12 CFR 1002.9(b)(2), implementing the Equal Credit Opportunity Act, requires that a statement of reasons for adverse action “must be specific and indicate the principal reason(s) for the adverse action”. A statement resting on the creditor's internal standards, or on a failed credit score, is declared insufficient. The sentence the template breaks sits in the Consumer Financial Protection Bureau's official interpretation, comment 9(b)(2)-2: “The specific reasons disclosed under §§ 1002.9(a)(2) and (b)(2) must relate to and accurately describe the factors actually considered or scored by a creditor.” Comment 9(b)(2)-4 adds that where a credit scoring system is used the reasons “must relate only to those factors actually scored in the system”. Those interpretations date to 21 December 2011, as subsequently amended. The test is not how the reason reads. It is whether the factors it names are the factors the model used.

  • Plausibility: The explanation sounds coherent and respectful. That is the property Jacovi and Goldberg measure by asking humans — and the one they say human judgement cannot be used to establish faithfulness with.
  • Faithfulness gap: The stated reason may not track the actual decision mechanism, and comment 9(b)(2)-2 requires disclosed reasons to "accurately describe the factors actually considered or scored by a creditor".
  • Stability problem: Small irrelevant changes can alter the explanation. Ghorbani and colleagues used a perturbation that moved neither the label nor the confidence, and overlap in the top-1000 salient pixels still fell below 30% on average.
  • Persuasion risk: Applicants may accept an outcome because the narrative sounds authoritative. Bansal and colleagues found acceptance rising in exactly that direction, whether or not the recommendation was correct.
  • Recourse harm: People may change the wrong behavior based on an unfaithful reason, and where a scoring system is used comment 9(b)(2)-4 confines the reasons to factors actually scored in it.

Visual

Readable and stable, and still wrong about the model

Stability, readability, and faithfulness come apart: an explanation can have the first two and still misrepresent the model it describes. Each of the five properties is tested separately, and stability and sensitivity have a measured failure behind them rather than a warning.

Take 512 correctly-classified ImageNet images and perturb them so that neither the predicted label nor the prediction confidence moves. The picture of what the model looked at moves anyway. With a random sign perturbation at L-infinity = 8 there was on average less than 30% overlap in the top-1000 most salient pixels between the original and the randomly perturbed image. That held across all three interpretation methods tested: simple gradients (saliency maps), integrated gradients and DeepLIFT. Ghorbani and colleagues published the result in 2019, and their abstract puts the property failure in one line: “we show that interpretation of deep learning predictions is extremely fragile in the following sense: two perceptively indistinguishable inputs with the same predicted label can be assigned very different interpretations.” The prediction was constant. The explanation was not.

Sensitivity asks the opposite question — that meaningful changes be reflected — and it is asked with a different test.

FigureProcess · 5 steps
  1. 1

    Faithfulness

    The explanation reflects the model behavior it claims to describe.

  2. 2

    Stability

    Similar relevant cases produce appropriately similar explanations.

  3. 3

    Sensitivity

    Meaningful changes are reflected while irrelevant changes are not.

  4. 4

    Usefulness

    The audience understands and performs the intended task better.

  5. 5

    Justification

    The decision is supported by legitimate evidence and policy, not merely described.

A saliency map that survives scrambled parameters

An explanation can be plausible, useful, faithful, stable, complete, actionable, or legally sufficient. These are different properties. Whoever runs the assurance review should test the property the decision actually requires, rather than reward visual or verbal appeal. Useful tests randomize the model parameters, randomize the data, perturb the input, and delete or insert features. They also check whether counterfactuals stay consistent, whether two methods agree, whether a user understands, how far that user then relies on the output, and whether the task itself gets done better.

The randomization tests have names and a result list. Scramble the model's parameters, or permute the training labels at random, and see whether the saliency map notices. Those are the model parameter randomization test and the data randomization test. Adebayo and colleagues introduced them in 2018, in Sanity Checks for Saliency Maps. The paper reports the outcome as its third contribution: “Of the methods we tested, Gradients & GradCAM pass the sanity checks, while Guided BackProp & Guided GradCAM fail.” The cascading experiment reinitialised Inception v3, trained on ImageNet, across 17 blocks of weights. A revision dated 6 November 2020 narrowed the finding. Guided BackProp and Guided GradCAM are “invariant to higher layer parameters”, with the masks changing only once the lower convolutional layers are randomized. Two methods passed, two failed, and the failure has a location in the network.

A saliency map that survives scrambled parameters was never reading the model, and only the tests above can tell that from a convincing picture.

One test buys one property; a review that runs a single check can vouch for nothing else.

Case

Rudin in 2019, and the explanations that can be gamed

One argument says the whole enterprise is the mistake. In 2019, in Nature Machine Intelligence, Cynthia Rudin published a Perspective aimed squarely at systems already running in high-stakes settings. Her title says it plainly: stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Her abstract says what she thinks the alternative costs: “Some people hope that creating methods for explaining these black box models will alleviate some of the problems, but trying to explain black box models, rather than creating models that are interpretable in the first place, is likely to perpetuate bad practice and can potentially cause great harm to society.” Her objection is not that explanation is hard. It is that explaining a black box, instead of replacing it, keeps the black box.

Then came the demonstration. A scaffolding technique can hide a biased classifier behind whatever explanation its owner would rather show. Slack and colleagues built the attack and presented it at AIES 2020: “Specifically, we propose a novel scaffolding technique that effectively hides the biases of any given classifier by allowing an adversarial entity to craft an arbitrary desired explanation.” The classifier they scaffolded was deliberately discriminatory. It predicts purely from race on COMPAS — 6,172 Broward County, Florida defendants, 51.4% African-American — and on Communities & Crime, 1,994 communities; purely from gender on German Credit, 1,000 applicants, 69% male. They then measured how often the attack displaced the sensitive attribute from the top-ranked attribution.

With the single-feature attack, LIME's top attribution moved to the substitute feature in 100% of held-out points on COMPAS and in 100% on Communities & Crime. On German Credit it moved in 91% of points, where the substitute was the existing 'Loan Rate % Income' feature rather than a synthetic uncorrelated one. Against SHAP the same attack succeeded on 100% of Communities & Crime points, 84% of COMPAS points and 85% of German Credit points. In the LIME setting the adversarial classifier reproduced the original biased classifier's predictions 100% of the time. Fidelity was lower under SHAP: 91% on COMPAS, 80% on German and 75% on Communities & Crime. The regulator reading such a report is reading the output of the pipeline that produced it.

A convincing picture is not evidence that the model reasoned that way.

Comparison

Plausible narrative, Faithful technical explanation, or Decision justification?

Sounding reasonable, tracking the model, and justifying the decision are three separate achievements. A consequential decision needs the third. The first can be produced without either of the others.

The first two have published definitions. Jacovi and Goldberg set them against each other in 2020: ““Plausibility” refers to how convincing the interpretation is to humans, while “faithfulness” refers to how accurately it reflects the true reasoning process of the model”. In a setting such as recidivism prediction, where the interpretation's faithfulness carries legal consequences, they argue that a plausible but unfaithful interpretation may be the worst-case scenario. That is the loan template's position exactly. Their guidelines then close the shortcut a satisfaction survey offers: “Faithfulness evaluation should not involve human-judgement on the quality of interpretation”. Asking readers whether an explanation convinced them measures plausibility, by definition. It reports nothing about faithfulness. Decision justification is a third thing again: policy, evidence, rights and process. A faithful description of a mechanism may serve those. It does not supply them.

FigureComparison · 3 columns

Plausible narrative

Sounds reasonable to a person.

  • Can improve communication
  • May be disconnected from model behavior
  • Vulnerable to post-hoc rationalization
  • Insufficient for audit or recourse

Faithful technical explanation

Tracks defined model behavior under tests.

  • Supports debugging and assurance
  • May be difficult to understand
  • Still not causal or normatively justified
  • Needs scope and uncertainty

Decision justification

Shows why the decision is legitimate and supportable.

  • Includes policy, evidence, rights, and process
  • May use but exceeds model explanation
  • Can be challenged and remedied
  • Essential for consequential decisions

Example

Randomization, perturbation, and reliance

These four tests are designed for the explanation to fail. One that passes all of them is faithful, which is not the same as justified.

  • Randomization test: Check whether the explanation changes when model parameters or labels are randomized. These are the two tests of Adebayo and colleagues, which Gradients and GradCAM passed and Guided BackProp and Guided GradCAM failed.
  • Irrelevant-feature test: Modify a feature that should not matter and watch what the explanation does. Ghorbani and colleagues' random sign perturbation at L-infinity = 8 left the label and the confidence untouched and still cut top-1000 salient-pixel overlap below 30% on average.
  • Reliance study: Measure whether explanations help users identify wrong outputs rather than simply trust more. Bansal and colleagues ran mixed-method studies on Amazon book reviews, beer reviews and LSAT questions, with around 100 Mechanical Turk participants per condition. Explanations did not improve complementary team accuracy over simply displaying the AI's confidence. They did increase the chance participants accepted the recommendation, correct or not. The paper records it plainly: “One concerning observation was that explanations increased blind trust rather than appropriate reliance on AI.”
  • Reason audit: Compare natural-language reasons with model evidence and decision policy for sampled cases — in US credit, against the requirement in 12 CFR 1002.9(b)(2) and comment 9(b)(2)-2 that the disclosed reasons accurately describe the factors actually considered or scored.

Steps

State the claim, then try to break it

State what the explanation claims, then try to break it. The attempts are the named ones: the two randomization tests, a perturbation that leaves the predicted label and the confidence unchanged, a comparison against a second method, and a reason audit against the factors actually considered or scored. What survives is what may be shown to a user. It survives only for the claim that was tested.

FigureProcess · 5 steps

  1. 1. State the explanation claim

    Specify what behavior, decision, or process the method purports to explain.

  2. 2. Choose failure tests

    Randomize, perturb, compare, remove, insert, and test known cases.

  3. 3. Measure user effect

    Assess comprehension, error detection, reliance, and decision quality.

  4. 4. Separate justification

    Review policy, evidence, rights, and procedural legitimacy independently.

  5. 5. Bound deployment

    Restrict claims, add uncertainty, change method, or withhold misleading explanations.

Whether it helps, not whether it persuades

Explanation assurance measures whether a reason reflects the system and helps the reader, not whether it persuades. That is why Jacovi and Goldberg keep human judgement of interpretation quality out of faithfulness evaluation. They refuse the other shortcut too: “Do not trust 'inherent interpretability' claims”. Explanations from inherently interpretable models must be held to the same standards as post-hoc methods. And the apartment experiments are the reminder that a clear model can leave its readers worse at catching its mistakes.

Decide which randomization, stability, or reliance result would force the reviewer to redesign, restrict, remedy, or retire the system.

Key takeaways