Evaluating the Ability of Explanations to Disambiguate Models in a Rashomon Set

AI summary

Overview

Research area: Explainable AI (XAI), specifically the evaluation of local feature-importance explanations, situated at the intersection with the "Rashomon set" literature on predictive multiplicity and with algorithmic fairness.

Technical level: Intermediate. The paper assumes familiarity with feature-attribution explainers (LIME, SHAP), ground-truth and sensitivity-based evaluation metrics, and the idea of a Rashomon set, but it avoids heavy mathematical machinery beyond linear-model illustrations and a short algorithm.

Scope (one sentence): The paper argues that explanation evaluation metrics should help distinguish between similarly performing models in a Rashomon set, and shows that its proposed metric, AXE, can do so while also detecting adversarial "fairwashing" of explanations.

What This Paper Is About

A Rashomon set is a group of models that make similar predictions but can differ in how they reach those predictions — for example, in how fair they are towards protected groups or whether they rely on protected input features. Explanations are supposed to reveal those internal differences and help practitioners choose which model to deploy, but different explainers (LIME, SHAP, Gradients, Integrated Gradients) can give contradictory explanations for the same prediction, and the metrics used to score explanations can themselves hide the very differences practitioners want to see. The paper develops three principles for explanation evaluation and proposes AXE, a ground-truth-agnostic scoring method that satisfies all three, then tests whether AXE can catch deliberately manipulated explanations.

Key Contributions

1. Three evaluation principles. The paper formalizes local contextualization (explanations and their quality scores should depend on the datapoint being explained), model relativism (scores should change when a different model from the Rashomon set with a different internal mechanism is explained), and on-manifold evaluation (scores should not depend on off-manifold model behaviour).

2. A diagnostic of existing metrics. It classifies prior metrics into ground-truth based metrics (FA, RA, SA, SRA, RC, PRA) and sensitivity based metrics (PGI, PGU), and maps each against the three principles in Table 1. Ground-truth metrics satisfy only on-manifold evaluation; PGI and PGU satisfy only local contextualization and model relativism; AXE is listed as satisfying all three.

3. AXE, a new evaluation framework. AXE (sensitivity and ground-truth Agnostic eXplanation Evaluation) scores an explanation by how well the top-n most important features can predict the model's own output using k-Nearest Neighbours, defined in Algorithm 1 with hyperparameters n and k (written AXE_n^k).

4. An adversarial fairwashing experiment. The paper replicates a state-of-the-art fairwashing attack (Slack et al. 2019) across three datasets and four adversarially constructed models, and compares AXE against PGI and PGU.

Main Findings

• Explanations disagree in practice. In Figure 2, a neural network predicts diabetes on the "Pima Indians" dataset, and a single positive prediction is explained four different ways: Gradients names Diabetes Pedigree Function as the most important input pushing toward a positive prediction; SHAP keeps Glucose as most important but makes BMI negative; LIME makes Glucose and BMI positive and Insulin negative; Integrated Gradients makes Glucose and BMI positive and Blood Pressure negative.

• Ground-truth metrics collapse model differences. In the two-feature linear example of Figure 4, model m has coefficients β1 = 0.7 and β2 = 0.3, giving ground truth e* = (0.7, 0.3). Any explanation satisfying i1 > 0, i2 > 0, i1 > i2 receives the same FA, RA, SA, SRA, RC and PRA scores. A model m′ in the same Rashomon set with, for example, β1′ = 0.5 and β2′ = 0.3 is therefore rewarded for receiving the exact same explanation, even though the relative importance of X1 versus X2 ranges from 1 to infinity in the limit.

• Ground-truth metrics also violate local contextualization. Because the comparison is always against a fixed vector e*, entire regions of explanation space (not just individual points) share the same quality value of 1, 0.5 or 0, which incentivizes a single "global" explanation rather than per-datapoint explanations.

• Existing sensitivity metrics can be gamed. The paper notes that by defining the perturbation method appropriately, PGI can be made to emulate SHAP or LIME exactly, thus scoring particular explainers perfectly.

• AXE detects adversarial fairwashing. Table 2 reports that on German Credit, COMPAS and Communities and Crime, across models m_L1 (LIME, 1 foil), m_S1 (SHAP, 1 foil), m_L2 (LIME, 2 foils) and m_S2 (SHAP, 2 foils), AXE always assigns the highest average quality to the explanation set E_ρ that correctly identifies the protected feature X_ρ. For example, on German Credit with m_L1 and n = 1, AXE gives q̄(E_ρ) = 1.000 versus q̄(E_φ) = 0.680 and q̄(E_ω) = 0.617.

• PGI and PGU fail on the same task. Using n = 1, PGU fails to rank the correct explanation above the spurious ones 7 out of 10 times and PGI fails 3 out of 10 times, giving an overall error rate of 50% for the two sensitivity metrics, versus 0% for AXE. The abstract states AXE detects adversarial fairwashing with a 100% success rate.

• AXE can flag protected attributes. The paper states that unlike prior strategies based on model sensitivity or ground-truth comparison, AXE can determine when protected attributes are used to make predictions.

• Prior evaluation practices are weak. The paper cites prior work reporting that of approximately 300 papers proposing new model explanation methods, one in three evaluated explanations anecdotally.

Methodology in Plain English

The authors first set out notation: an input vector x with N features, a binary model m: ℝ^N → {0,1}, an explainer ℰ that produces a signed feature-importance vector e, a per-explanation quality score q in [0,1], and an evaluation framework (X, m, ℰ, Q) where Q averages q over a dataset of ν datapoints.

They then define the three principles by asking what a scoring metric should do when (a) the datapoint changes, (b) the model changes to another member of the Rashomon set, and (c) the model's behaviour changes only off the data manifold. They illustrate the failure of ground-truth based metrics with a hand-constructed two-feature linear model where entire regions of explanation space get identical scores.

AXE itself is a simple procedure. For each datapoint and its explanation, it takes the n features with the largest absolute importance values, keeps only those features, trains a k-Nearest Neighbours classifier to predict the model's output (not the ground-truth label) using that feature subset, and then reports the accuracy of those predictions across the dataset. Crucially, a separate k-NN model is fitted per datapoint rather than one global surrogate, which is what makes the metric locally contextualized. Because the k-NN operates only over existing datapoints in the dataset, it never queries off-manifold inputs. Because it predicts the model's own output rather than the data label, the score depends on the specific model being explained.

Finally, they reproduce the Slack et al. (2019) attack. A discriminatory model m is wrapped in models m_S or m_L that preserve the same discriminatory decision procedure and similar predictions on the data manifold, but alter behaviour off-manifold so that SHAP or LIME reports a benign "foil" feature (X_φ or X_ψ) as most important. All such models form a Rashomon set. The researchers then manually build four explanation sets — E_ρ (protected feature most important), E_φ and E_ψ (foil features most important), and E_ω (everything else) — and check whether each metric ranks E_ρ above the spurious sets. PGU values are flipped negative so that higher is better, matching PGI and AXE.

Why This Matters

Impact on research. The paper reframes explanation evaluation as a tool for model selection under predictive multiplicity, rather than as a search for a single ideal explanation. It offers a concrete critique of ground-truth based metrics used in recent benchmark work and proposes a testable, ground-truth-free alternative, together with a released code and dataset repository.

Real-world applications.

• Lending decisions: the German Credit dataset is used to model a classifier adversarially modified to favour men over others; explanation evaluation determines whether that behaviour can be exposed.
• Criminal justice risk assessment: the COMPAS and Communities and Crime datasets are used to model classifiers modified to favour white people over others.
• Healthcare: the diabetes prediction example on the Pima Indians dataset shows how explanations for the same prediction can contradict each other, which matters when clinical decisions depend on them.
• Regulatory auditing and model deployment: helping auditors compare candidate models that look equivalent on accuracy but differ in whether they rely on protected attributes such as race or gender.

Industry relevance. Organisations deploying models in regulated domains need evidence that explanations are trustworthy and that model choice has not been made to hide discriminatory behaviour. The paper's result that two widely used sensitivity metrics mis-rank spurious explanations 50% of the time, while AXE does not, is directly relevant to any group building explanation-based audit tooling. The paper also cites the Wellcome Trust, Sloan Foundation, Department of Health and Social Care, EPSRC, Luminate Group and the Alexander von Humboldt Foundation as funders.

Future Directions

• Extending beyond the abridged setting. The authors note that additional experiments and theory situating AXE in the explainability literature appear only in the extended version, "Evaluating Model Explanations without Ground-truth", so a fuller comparative study of AXE against other metrics is left to that work.
• Generalizing beyond tabular data and local feature importance. The paper restricts itself explicitly to local feature-importance explanations for tabular datasets, leaving image, text, and other modalities, as well as global or example-based explanations, unaddressed.
• Choosing and justifying n and k. AXE has two hyperparameters (top-n features and the number of neighbours k), and the paper's fairwashing experiment uses n = 1; how sensitive results are to these choices, and how to select them in practice, is not reported here.
• Defences beyond detection. The paper shows that off-manifold manipulation misleads both explainers and sensitivity-based evaluation, but it does not report a method for preventing the manipulation itself, only for detecting it after the fact.

Target Audience

Researchers and graduate students working on explainable AI, explanation evaluation, or the Rashomon effect and predictive multiplicity; fairness and AI-audit practitioners who need to compare candidate models that perform similarly on accuracy; and regulators or policy researchers interested in how explanation quality can be measured without access to ground-truth explanations. Readers without prior exposure to feature-attribution methods or to the Rashomon set concept will need to consult the referenced background literature first.

Authors’ abstract

Explainable artificial intelligence (XAI) is concerned with producing explanations indicating the inner workings of models. For a Rashomon set of similarly performing models, explanations provide a way of disambiguating the behavior of individual models, helping select models for deployment. However explanations themselves can vary depending on the explainer used, and need to be evaluated. In the paper "Evaluating Model Explanations without Ground Truth", we proposed three principles of explanation evaluation and a new method "AXE" to evaluate the quality of feature-importance explanations. We go on to illustrate how evaluation metrics that rely on comparing model explanations against ideal ground truth explanations obscure behavioral differences within a Rashomon set. Explanation evaluation aligned with our proposed principles would highlight these differences instead, helping select models from the Rashomon set. The selection of alternate models from the Rashomon set can maintain identical predictions but mislead explainers into generating false explanations, and mislead evaluation methods into considering the false explanations to be of high quality. AXE, our proposed explanation evaluation method, can detect this adversarial fairwashing of explanations with a 100% success rate. Unlike prior explanation evaluation strategies such as those based on model sensitivity or ground truth comparison, AXE can determine when protected attributes are used to make predictions.