Beyond Satisfaction: From Placebic to Actionable Explanations For Enhanced Understandability

AI summary

Overview

• Research area: Human-Computer Interaction and Explainable AI (XAI), specifically the evaluation of explanation quality.
• Technical level: Intermediate. The paper is readable without deep machine learning background, but the argument depends on understanding study design, ANOVA, Tukey's HSD, and two one-sided tests (TOST) for equivalence.
• One-sentence scope: A between-subjects online user study with 189 participants comparing no explanations, placebic explanations, and actionable explanations on a Social Security filing-age optimization task, showing that subjective satisfaction ratings do not distinguish explanation quality while objective performance measures do.

What This Paper Is About

Explainable AI research frequently measures the quality of an explanation by asking users how satisfied they were, but it is unclear whether satisfaction actually tells us whether an explanation helped anyone understand anything. This paper tests that question directly by pitting subjective survey ratings against objective task performance, using a domain where correctness can be measured precisely. The goal is to show that evaluations relying only on satisfaction can be misled, and that objective mental-model measures should be added.

Key Contributions

1. A controlled three-condition comparison. The authors ran a between-subjects study with 189 participants recruited from Prolific, split into three equal groups of 63: no explanations (None), placebic explanations (Placebic), and actionable explanations (Actionable), all completing the same Social Security filing age optimization tasks.

2. A direct demonstration that satisfaction fails to discriminate explanation quality. Actionable and placebic explanations produced statistically equivalent satisfaction with the agent (TOST p = .048, Hedges' g = -.04), even though actionable explanations produced significantly better objective performance.

3. Evidence that placebic explanations are no better than nothing. Using equivalence testing, the Placebic and None groups were statistically equivalent on both testing error and bonus compensation.

4. A methodological recommendation plus released code. The authors argue for integrating objective task performance metrics alongside subjective assessments in XAI evaluation, and provide the study code at https://github.com/Shymkis/social-security-explainer.

Main Findings

• Actionable explanations improved accuracy. One-way ANOVA on testing error across the three protocols showed a significant difference, F(2,186) = 5.893, p = .0033, η² = .06. Tukey's HSD found the Actionable group's mean testing error was significantly smaller than the Placebic group's (p = .0025). Actionable versus None was not significant (p = .0733), and Placebic versus None was not significant (p = .4672).

• Actionable explanations improved bonus compensation. ANOVA on bonus compensation was significant, F(2,186) = 5.255, p = .0060, η² = .05. Tukey's HSD found Actionable earned significantly higher bonuses than Placebic (p = .0075) and than None (p = .0362). Placebic versus None was not significant (p = .8452).

• Placebic explanations were statistically equivalent to no explanations. With an equivalence bound of 0.5 for testing error (errors ranged from 0 to 8), TOST gave t(124) = 1.821, p = .036, Hedges' g = -.23. With an equivalence bound of 0.1 for bonus compensation (range 0 to 1), TOST gave t(124) = -3.137, p = .001, Hedges' g = .11.

• Users were equally satisfied with placebic and actionable agents. Satisfaction with the agent showed no significant difference between the two explanation groups, F(1,124) = 0.04, p = .843, η² < .001, and TOST indicated equivalence, t(124) = 1.681, p = .048, Hedges' g = -.04. The None group was excluded from this question because it received no explanations.

• Satisfaction with the practice section showed no protocol differences. ANOVA gave F(2,186) = 2.342, p = .099, η² = .02. Tukey's HSD: Actionable versus None p = .0811, Placebic versus None p = .6074, Actionable versus Placebic p = .4511. This failed to support the hypothesis that placebic explanations would satisfy users more than no explanations.

• Perceived explanatory power was inconclusive. ANOVA gave F(1,124) = 2.894, p = .0914, η² = .02, and TOST found no significant equivalence either, t(124) = 0.213, p = .416, Hedges' g = -.30.

• The Actionable group learned the most across sections. Average error per selection fell from 3.23 to 2.87 (-11.3%) for None, from 3.36 to 3.05 (-9.3%) for Placebic, and from 2.95 to 2.52 (-14.5%) for Actionable. The Actionable group had both the smallest error and the largest improvement; the Placebic group had the largest error and the smallest improvement.

• Subjective ratings tracked performance more than explanation type. Pearson correlations with testing error were significant for satisfaction with practice (df = 187, t = -5.813, p < .001, r = -0.391) and for explanatory power (df = 124, t = -3.772, p < .001, r = -0.321), but not for satisfaction with the agent (df = 124, t = -0.755, p = .452, r = -0.068).

• Hypothesis outcomes. H1 (actionable beats placebic on performance) and H2 (placebic no better than none) were supported. H3 (actionable not more satisfying than placebic) was supported. H4 (placebic more satisfying than none) was not supported. H5 (no difference in perceived explanatory power) was not supported, since neither a significant difference nor a significant equivalence was found.

Methodology in Plain English

The researchers chose a task where "doing well" could be scored objectively: picking the optimal age to file for Social Security benefits under hypothetical scenarios. The ground truth came from opensocialsecurity.com, a calculator that produces inflation-adjusted expected values for filing decisions based on lifespan, marital status, and income.

Each scenario varied marital status, primary insurance amount (PIA), current age, and lifespan. Participants moved sliders to select filing ages, submitted, and then saw the optimal ages and their error. Participants were recruited through Prolific, restricted to US residents, at least 18 years old, with an approval rate of at least 98%.

Everyone completed a practice section of 10 filing decision tasks in 7.5 minutes. Only the Placebic and Actionable groups received explanatory text from an AI agent after each task; the None group received nothing. Placebic explanations restated known facts or tautologies without revealing model reasoning, while actionable explanations were written by the researchers to reflect the underlying model accurately.

Then everyone completed a testing section with 10 more scenarios under the same time limit, with no explanations and with bonus compensation tied to accuracy. Across the 10 test scenarios there were 18 selections, each weighted equally, and the bonus for a single selection dropped by 25% of its maximum value per year of error, so an error of four or more years earned no bonus for that selection.

Afterward, participants rated their agreement with a set of statements on a 7-point Likert scale, covering three subjective metrics: satisfaction with the practice section, satisfaction with the agent, and the agent's explanatory power. Each metric was the average of responses to three statements. The analysis used one-way ANOVA, Tukey's HSD for pairwise differences, TOST for pairwise equivalence, and Pearson correlations between survey metrics and error.

Why This Matters

This paper targets a specific weakness in how XAI systems are validated. If a placebo explanation and a genuinely informative explanation receive the same satisfaction scores, then a large body of evaluation practice may be unable to tell good explanations from useless ones.

• Research impact: It provides direct empirical evidence that satisfaction is an unreliable proxy for understandability, and shows that objective performance testing is more discriminating. It also adds placebic explanations as a comparison condition that few XAI studies include.

• Real-world applications:

  • Decision-support tools that help people choose complex financial or retirement options, such as Social Security filing ages.
  • Pedagogical and tutoring agents that must transfer domain knowledge to non-experts rather than merely appear helpful.
  • Recommendation systems where users need to build an accurate model of why a suggestion was made, not just accept it.
  • Content moderation or other systems where perceived bias and trust can be affected by explanation quality, a context the paper cites from prior work.

• Industry relevance: Teams shipping AI assistants and expert agents often justify design decisions with satisfaction surveys and user feedback. This work suggests those signals can be satisfied by placebic design while the underlying user understanding has not improved, which matters for products where users must act correctly on the system's advice.

Future Directions

• Broaden beyond satisfaction. The authors note that other subjective metrics such as trust, confidence, and reported understandability may behave differently from satisfaction, and that this study covered only reported satisfaction.

• Test with domain experts. The paper states its results are limited to users without machine learning expertise or domain knowledge, and cites evidence that subjective metrics may be more informative when provided by experts.

• Consider alternative explanation goals. Understandability is the focus here, but the authors point to the growth of human-AI team studies where team performance is the primary goal, and suggest examining how metric choice interacts with the system's objective.

• Compare types of bad explanations. The study used restatements of known instructional facts as its placebic condition; the authors distinguish these from tautologies, off-topic statements, and random, misleading, or outright false explanations, and call for comparing user responses across these categories to decide on appropriate evaluation metrics.

Target Audience

This paper is most useful for XAI and HCI researchers who design or evaluate explanation methods, particularly those who use user studies and satisfaction questionnaires. It also suits practitioners building expert or pedagogical agents for non-expert users, and methodologists looking for a concrete example of combining subjective and objective evaluation in a single study. Readers without a statistics background can follow the argument from the hypotheses, the group comparisons, and the discussion, though the equivalence testing results require some familiarity with TOST to interpret fully.

Authors’ abstract

Explainable AI (XAI) presents useful tools to facilitate transparency and trustworthiness in machine learning systems. However, current evaluations of system explainability often rely heavily on subjective user surveys, which may not adequately capture the effectiveness of explanations. This paper critiques the overreliance on user satisfaction metrics and explores whether these can differentiate between meaningful (actionable) and vacuous (placebic) explanations. In experiments involving optimal Social Security filing age selection tasks, participants used one of three protocols: no explanations, placebic explanations, and actionable explanations. Participants who received actionable explanations significantly outperformed the other groups in objective measures of their mental model, but users rated placebic and actionable explanations as equally satisfying. This suggests that subjective surveys alone fail to capture whether explanations truly support users in building useful domain understanding. We propose that future evaluations of agent explanation capabilities should integrate objective task performance metrics alongside subjective assessments to more accurately measure explanation quality. The code for this study can be found at https://github.com/Shymkis/social-security-explainer.