Challenges in Evaluating Explanation Methods for Static and Evolving Data

AI summary

Overview

Research area: Explainable Artificial Intelligence (XAI) — specifically the evaluation of explanation methods, fairness/bias mitigation in image recognition, counterfactual explanations, and explanations for evolving data streams affected by concept drift.

Technical level: Intermediate. The paper is a position-and-experience paper rather than a technical algorithm paper; it assumes familiarity with XAI terminology (saliency maps, counterfactuals, prototypes, concept drift) but presents its arguments at a conceptual level.

Scope: A single-author discussion paper by Jerzy Stefanowski (Poznan University of Technology, Institute of Computing Sciences, Poland) surveying why XAI evaluation is insufficient, illustrated by two case studies on image recognition and by the author's work on adapting explanations to non-stationary data.

The paper is accepted for publication in J. Nalepa (ed.) Explainable AI in Space, Proceedings of the EASi 2026 Workshop at IJCAI-ECAI 2026 Bremen, Springer CCIS vol 3107 (the year is printed as 2016 in the paper). Keywords listed are: Explainable AI, Evaluation of XAI, Debiasing, Human-grounded studies, Counterfactuals, Evolving data. The work was funded by the National Science Centre, Poland, under the OPUS programme, grant number 2023/51/B/ST6/00545.

What This Paper Is About

XAI research keeps producing new explanation methods, but far less effort goes into checking whether those explanations are actually correct or useful. The paper argues that evaluation is the weak link: measures are narrow, human studies are rare, and methods built for static data break down when data and models change over time. Its goal is to use concrete case studies to show how evaluation could be done better, both automatically and with human participants.

Key Contributions

1. A structured critique of XAI evaluation practice, distinguishing human-centered from computer-centered evaluation using the taxonomy of application-grounded, human-grounded, and functionally-grounded evaluation, and listing the proxy measures commonly used (fidelity, identity, stability, feature importance, compactness/simplicity, representativeness).

2. Two case studies on image recognition, showing contrasting evaluation paths: the DetoxAI system, which uses XAI to detect and remove bias in convolutional neural networks, and a human-grounded survey study comparing three saliency-map methods for explaining CNN predictions.

3. A discussion of counterfactual evaluation ambiguity, including the argument that no single counterfactual method is definitively best, and a proposed ensemble-of-explainers approach reduced via a dominance relation, a Pareto front, and a multiple-criteria Ideal Point Method.

4. An extension of XAI beyond static settings, describing the author's work on adapting prototype explanations (for tree ensembles such as bagging and random forests) and group counterfactual explanations to evolving data streams with concept drift, including new measures for comparing explanation sets before and after a change.

Main Findings

• Evaluation is chronically underdeveloped. The paper cites a review reporting that 97% of reviewed works claimed their explanations serve users, but most did not evaluate them through any user study. Another review of 381 XAI papers found that only 5% explicitly focused on a deeper evaluation of XAI methods.

• High fidelity is not enough. Guidotti's experiments are cited to show that focusing on high fidelity when evaluating proxy explanations may be incorrect with respect to background knowledge — rule-based explainers failed to retrieve all features in rule conditions as the number of features increased, and explainers failed to consider all relevant features or assign them the correct weight.

• DetoxAI case: bias was detected and largely removed. In the CelebA dataset, wearing items such as a necktie was found to be highly correlated with gender, creating a proxy feature for discrimination, and the models consistently misclassified individuals wearing neckties as not smiling. Using saliency maps generated with Concept-wise Relevance Propagation, the bias was detected; applying concept unlearning methods (including approaches based on Savani and Zhang, LEACE, AClarc, and post-hoc threshold optimization) led most methods to eliminate the network's undesirable focus on bias, observed qualitatively in saliency maps and quantitatively through improved group-level fairness metrics such as EqualizedOdds and Demographic Parity.

• Human study case: ProtoPNet was preferred overall. In an online survey with 148 completed responses, ProtoPNet received 1,045 total ranking points, ahead of ACE (763) and RISE (643). ProtoPNet won for most individual animals, with the strongest victory for the zebra, panda, and tiger; ACE won for the bison and was nearly tied for the elephant.

• Experience changed preferences. Respondents with prior experience using XAI methods showed a stronger preference for ProtoPNet, while those without such experience preferred ACE. Apart from that, analysis of demographic responses revealed no significant variation in the distribution of responses across respondent categories.

• Explanations correlated with expected anatomical features. The ACE score had the strongest correlation with the expected presence of an elephant's trunk, while ProtoPNet was associated with the mouth in photos of hippos and with the presence of stripes on tigers or zebras. The paper notes that it was not analyzed whether observing the explanation increased respondents' trust, though respondents generally did not express a lack of trust in the system's predictions.

• Survey design mattered as much as results. A pilot study with participants led the authors to focus on 10 well-known, easily recognizable, diverse animals from ImageNet (such as elephant, crocodile, hippopotamus, panda, tiger, zebra, lion, bison, and frog). The differing graphical formats of the three methods confused pilot respondents, so saliency maps were converted into a uniform graphical interface. Respondents saw up to six explanations per photo, were not told method names, and selected 0 to 2 of the best.

• Counterfactual metrics conflict. Properties such as validity, proximity, sparsity, actionability, plausibility, and discriminative power are not aligned and can even be contradictory, so there is no single definitive best counterfactual method. A proposed multi-criteria framework — an ensemble of multiple base explainers, a dominance relation, a Pareto front, and the Ideal Point Method for final selection without extra preference elicitation — produced counterfactuals competitive with the best single methods across several datasets.

• Explanations can be adapted to drift. For prototypes explaining tree ensemble predictions, new measures — mean minimal distance between prototypes, mean prototype centroid displacement, and prototype reassignment impact — supported detecting concept drift, and combining them with feature parallel plots supported analysis of drift causes concerning the most influential features. For group counterfactuals, generalizing the GLANCE algorithm and tracking cluster centroids and their counterfactual vectors before and after drift showed that cosine similarity between vectors serves as an interpretable proxy for locality of drifts and for checking whether the decision boundary moved.

• Interacting signals give a fuller picture. Analyzing data-level changes, model performance drops, and explanation changes jointly in the group-counterfactual work provided a more comprehensive understanding of drift than analyzing each layer independently.

Methodology in Plain English

The paper is organized as a guided tour through problems rather than a single experiment. It first frames why evaluating explanations is hard, using published statistics and prior critical studies to show how often evaluation is skipped or done narrowly. It then walks through two concrete image-recognition cases: one where evaluation is largely automated (using saliency maps produced by Concept-wise Relevance Propagation to find a spurious gender-correlated feature, then measuring fairness metrics before and after concept unlearning), and one where evaluation is human (running an online survey in which lay respondents ranked anonymous explanations from ACE, ProtoPNet, and RISE, alongside demographic questions and questions about which animal features they considered important).

From there the paper turns to counterfactual explanations, arguing that because methods optimize different, sometimes conflicting criteria, the better strategy is to generate several candidate explanations and filter them using multi-criteria decision analysis. Finally it moves to evolving data streams, where instead of explaining a single fixed model, the approach is to track how sets of prototypes or groups of counterfactuals shift over time, quantify those shifts with dedicated measures, and visualize them alongside data-level statistics and model performance drops to explain what changed and why.

Why This Matters

The paper's central claim is that XAI is developing faster than its ability to demonstrate usefulness — a situation the author likens to an "illusion of progress" in the tradition of David Hand's critique of supervised classification. It argues that machine-learning-style, unified evaluation standards are still missing, that measures do not generalize across explanation types, and that application- and human-grounded evaluation are too neglected. For research, it pushes toward comparative studies, multi-criteria evaluation, better annotated benchmarks, and interdisciplinary work drawing on HCI, cognitive science, and psychology.

Real-world applications the paper points to:

• Face recognition and image systems: the CelebA case shows how a harmless-looking attribute like wearing a necktie can become a proxy for gender and degrade fairness metrics, with consequences for automated decisions about people.
• Medicine: cited reasons include clinical trust (physicians verifying logic before high-risk treatments), detecting errors, dataset biases and false correlations, legal accountability since responsibility lies with humans, and support for scientific discovery.
• Aerospace and other critical systems: mission-critical verification, failure forensics, extreme domain adaptation, and strict regulatory compliance, with spacecraft health monitoring and anomaly detection given as examples where explainability is a prerequisite for operational trust.
• Finance and other high-stakes decision settings: counterfactuals are described as giving actionable insight that lets individuals understand and potentially contest automated decisions.

Industry relevance: the paper implies that deploying XAI tools without rigorous evaluation carries operational and compliance risk, especially for autonomous systems in high-stakes environments, and that practitioners should treat explanation quality as a multi-criteria trade-off rather than a single score.

Future Directions

• Build a unified evaluation framework. Develop fundamental research on new experimental standards, since current measures depend heavily on explanation type and task and do not generalize; broaden the set of measures and use multi-criteria decision analysis.

• Invest seriously in human-grounded studies. Open questions listed include how to organize interaction sessions with users, how to prepare them, how to validate the correctness of their answers, whether to move to NLP sequential dialogue instead of one-step answers, how to incorporate causality and background knowledge, and whether explanations actually increase trust in predictions and use of AI systems.

• Create better benchmarks. More benchmark datasets with annotated ground-truth information for explanations are needed, including work with synthetic generators.

• Evaluate explanations over time. Existing measures focus on static settings at single points in time; longer temporal dynamics and possible interactions between changes in data, models, and explanations require intensive fundamental research, with applications in space exploration, Earth observation, and medicine suggested as sources of inspiration.

Target Audience

Researchers and graduate students working on XAI, interpretability, and fairness, especially those designing evaluation protocols or human studies. It also suits practitioners in regulated or safety-critical domains (medical AI, aerospace, finance) who need to justify model behaviour, and data-stream mining researchers interested in explaining concept drift. The paper is least suited to readers looking for new algorithms or detailed experimental numbers, since the author's own methods are referenced rather than fully specified here. Notably, some quantities are not reported in this text — for example, the CNN trained on animal images is described only as achieving "high recognition accuracy" with no numeric value given.

Authors’ abstract

This paper addresses the limitations of Explainable Artificial Intelligence (XAI) with respect to insufficient evaluation. They are illustrated through the DetoxAI image recognition system for bias detection and concept unlearning. Then, an example of a human-grounded evaluation of methods for explaining image classification is presented. The paper further explores methods for adapting explanations to evolving data streams with concept drift. Experiences with adapting counterfactuals for this problem are discussed. Finally it is related to the challenges of tracking the co-evolution of data, models, and explanations.\footnote{This paper has been accepted for a publication in J.Nalepa (ed) Explainable AI in Space. Proceedings of EASi 2026 Workshop at IJCAI-ECAI 2026 Bremen, Springer CCIS vol 3107 (2016).}