The Pulse
New Paper Questions Whether XAI Faithfulness Scores Measure Trust
A position paper on arXiv argues that common metrics for feature-importance explanations can mismeasure how closely an explanation reflects a model. The authors identify five weaknesses and propose four research directions, including evalua
AI.info Team · Written with AI ·
Feature-importance explanations can earn a high faithfulness score even when they fail to reflect how an AI model uses its inputs. A position paper submitted to arXiv on September 24 argues that this mismatch leaves researchers without a dependable way to judge whether explanations match the models they describe—and, by extension, whether those explanations support trust.
The paper, “Faithfulness Without Anchors,” focuses on metrics for feature importance: methods that assign scores to parts of an input to indicate how much each part contributes to a model’s prediction. Its seven authors argue that the problem is not simply that explanations can be unfaithful. The tests used to assess faithfulness can also be unreliable or measure something other than the relationship between an explanation and the model.
When removing features distorts the test
Many faithfulness tests remove or alter features, then track how the model’s output changes. The premise is that removing a feature marked as important should affect the prediction more than removing one marked as less important. Researchers often compress those changes into a single score based on a curve’s area or its correlation with feature rankings.
The authors identify five weaknesses in this approach: research focuses disproportionately on classification rather than regression; some measures blur faithfulness with other explanation qualities; feature removal depends on arbitrary baselines; perturbations can push inputs outside the data the model was trained to handle; and common correlation- and area-based scores can reward patterns that do not match the intended meaning of faithfulness.
That last problem can produce misleadingly strong scores. In examples in the paper, a correlation measure can give different perturbation curves the same perfect score, even when one curve reflects a sound explanation and another does not. An area-based measure can also reward a case where only the top-ranked feature affects the prediction, though the authors say a faithful ranking should reflect proportional contributions across features.
Out-of-distribution inputs complicate the evidence
Feature removal can create inputs unlike anything the model encountered during training. A classifier may respond to those unfamiliar inputs with erratic or overconfident predictions, breaking the assumption that confidence should fall steadily as important information disappears. The paper argues that a metric built around that assumption cannot reliably grade the explanation when the model itself behaves unpredictably under perturbation.
To illustrate the issue, the authors describe a ResNet-18 image classifier trained on HAM10000, a dataset of seven skin-lesion classes. In their example, the model treats one class as a default category and remains highly confident in that classification even when an image is reduced to black patches. The example shows a limitation of the test setup, not a clinical evaluation of the classifier or evidence about medical decisions.
Four proposed directions for XAI testing
“In this position paper, we outline five major shortcomings of current faithfulness metrics and propose four possible avenues for research to bolster their reliability,” write Marco Zullich and six co-authors. Their proposals include testing explanations on models with better-calibrated uncertainty and more reliable behavior on out-of-distribution inputs; assessing explanations for classes other than the model’s chosen prediction; developing metrics suited to regression; and aligning each metric with what faithfulness is meant to measure.
The distinction between classification and regression matters because the usual test has no obvious translation between them. In classification, researchers can track changes in class confidence. In regression, they must decide whether to measure changes in the predicted value or changes in prediction error—two different questions, the authors argue, that can produce scores that are difficult to compare across datasets or examples.
A narrower claim than a verdict on explainability
The paper is a position paper about measuring feature-importance explanations, not a new user study showing that explanations universally fail to build trust. Its opening summarizes prior work suggesting that explanations can encourage overreliance on incorrect predictions, but its technical argument centers on whether existing faithfulness metrics can substantiate claims about the match between an explanation and a model.
The authors also limit the scope of their review: they do not examine other XAI tasks, including counterfactual explanations and concept-based methods. Their closing distinction is pointed: the field, they argue, lacks a principled way to assess faithfulness. Until that assessment improves, a score attached to an explanation may say less about trustworthiness than its precision suggests.