Human Cognitive Biases in Explanation-Based Interaction: The Case of Within and Between Session Order Effect

AI summary

Authors: Dario Pesenti, Alessandro Bogani, Katya Tentori, Stefano Teso (CIMeC, University of Trento; Teso also DISI) arXiv: 2512.04764v1 [cs.AI], 04 Dec 2025 · License: CC BY 4.0

Overview

Research area: Human factors in Explainable AI (XAI) and Explanatory Interactive Learning (XIL), at the intersection of cognitive psychology (order effects/decision biases) and interactive machine learning.

Technical level: Intermediate. No deep algorithm background is required — the paper's core is an experimental design with mixed-effects statistical modeling — but familiarity with XAI concepts such as saliency maps, local explanations, and the XIL debugging loop helps.

Scope: Two pre-registered-style, larger-scale behavioral user studies (n = 713 total) testing whether the order in which a model's correct and incorrect explanations appear biases users' feedback quality, their agreement with the model, and their reported trust, both within a single debugging session and between two consecutive sessions.

What This Paper Is About

XIL algorithms let users debug a model by showing them its predictions together with explanations (e.g. a bounding box or saliency region) and asking for corrective feedback. Because items are presented sequentially, prior work raised the concern that order effects — primacy (early information weighted more) and recency (late information weighted more) — could degrade the quality and reliability of that feedback. The authors argue the existing evidence (notably Nourani et al., 2021, and Honeycutt et al., 2020) is inconclusive because those designs diverge from typical XIL use cases, so they run two controlled studies that mimic realistic XIL debugging to measure the real size of the problem.

Key Contributions

1. Two larger-scale controlled user studies (n = 713 total) that simulate a realistic XIL debugging task, deliberately designed to be more ecologically valid than the designs used in earlier order-effect studies (participants cannot self-select stimuli and can correct the model's outputs).
2. The first study to separate within-session from between-session order effects in the same XIL-like paradigm, manipulating the distribution of model errors while holding overall model accuracy fixed (60% in both settings), so that any group difference is attributable to presentation order alone.
3. Behavioral rather than only self-report measurement: two objective, overlap-based dependent variables — accuracy of participants' feedback relative to ground truth, and agreement with the model — alongside a four-item, 7-point Likert questionnaire adapted from Honeycutt et al. (2020) and Hoffman et al. (2019) on perceived accuracy and trust.
4. A clear practical conclusion for XIL design: order effects are limited and inconsistent, and do not appear to threaten the successful deployment of XIL approaches.

Main Findings

• Feedback quality was high and stable across conditions. In Experiment 1, overall accuracy was 0.76 ± 0.10 (Inc), 0.75 ± 0.08 (Const), and 0.76 ± 0.08 (Dec). In the second session of Experiment 2 it was 0.78 ± 0.08 (Inc), 0.78 ± 0.07 (Const), and 0.78 ± 0.08 (Dec).

• Stimulus properties mattered far more than order. Participants were more accurate on easy (0.88 ± 0.08) than difficult images (0.62 ± 0.12; F(1,36) = 63.33, p < .001) in Experiment 1, and on correct (0.80 ± 0.11) than incorrect model placements (0.69 ± 0.10; F(1,36) = 10.65, p = .002). The same pattern held in Experiment 2 (easy 0.89 ± 0.06 vs difficult 0.66 ± 0.11, F(1,36) = 47.81, p < .001; correct 0.84 ± 0.11 vs incorrect 0.69 ± 0.09, F(1,36) = 19.55, p < .001).

• Within-session order effect on accuracy was small and significant. The order × placement interaction was significant in Experiment 1 (F(2,12875) = 4.56, p = .011), driven by a narrower gap between incorrect and correct images in the Dec condition (0.70 ± 0.08 vs 0.79 ± 0.11) than in the Const condition (0.67 ± 0.11 vs 0.80 ± 0.10; p = .008). The authors state this small, single-contrast difference may also reflect random noise.

• Within-session order effect on agreement looked like a small primacy effect. The three-way interaction between order, placement, and difficulty was significant (F(2,12862) = 7.99, p < .001). For incorrect placements, agreement on difficult vs easy images was 0.19 ± 0.08 vs 0.14 ± 0.06 in Inc, 0.24 ± 0.11 vs 0.15 ± 0.12 in Const (p = .027), and 0.24 ± 0.12 vs 0.13 ± 0.03 in Dec (p < .001). The authors read this as early exposure to model inaccuracy (the Inc condition) reducing reliance on the model, especially on ambiguous stimuli — mostly driven by difficult, partially wrong images.

• No between-session order effect on agreement. In the second session of Experiment 2, agreement was essentially identical across conditions: Inc 0.63 ± 0.10, Const 0.62 ± 0.10, Dec 0.61 ± 0.09. The paper reports no significant order condition effect on agreement here.

• Between-session accuracy effect was also tiny. The order × placement interaction was significant (F(2,12915.6) = 8.61, p < .001), with a slightly steeper accuracy drop for incorrect images in Inc (0.84 ± 0.11 correct vs 0.68 ± 0.09 incorrect) than in Const (p = .002) and Dec (p = .001). The authors explicitly caution that the magnitude is small and may be noise.

• Self-reported perceptions showed no order differences at all. Questionnaire index in Experiment 1: Inc 3.06 ± 1.03, Const 3.11 ± 1.11, Dec 3.13 ± 1.00 (Kruskal-Wallis p = .909). Experiment 2: Inc 3.38 ± 1.21, Const 3.43 ± 1.17, Dec 3.34 ± 1.23 (p = .821). Participants were apparently unaware of the behavioral effect observed in Experiment 1.

• The between-session manipulation worked as intended. Agreement in the first session of Experiment 2 tracked the manipulated model accuracy (F(2, 305.92) = 483.39, p < .001): Inc 0.42 ± 0.36 < Const 0.57 ± 0.38 < Dec 0.70 ± 0.35, all post-hoc p < .001.

• Agreement tracked correctness of the explanation, as expected. In Experiment 1, agreement was 0.80 ± 0.11 for correct boxes vs 0.18 ± 0.08 for incorrect boxes (F(1,36) = 174.38, p < .001). In Experiment 2's second session: 0.84 ± 0.11 vs 0.31 ± 0.10 (F(1,36) = 112.04, p < .001).

• Difficulty selectively reduced agreement on correct placements. Placement × difficulty interaction in Experiment 1: F(1,36) = 12.80, p = .001 (correct box: easy 0.92 ± 0.10 vs difficult 0.67 ± 0.16, p < .001; incorrect box: easy 0.14 ± 0.08 vs difficult 0.22 ± 0.11, p = .458). In Experiment 2: F(1,36) = 7.06, p = .012 (correct box: easy 0.94 ± 0.08 vs difficult 0.74 ± 0.15, p = .006; incorrect box: easy 0.28 ± 0.10 vs difficult 0.34 ± 0.14, p = .816).

• Overall conclusion: order effects do not pose a significant issue for the successful employment of XIL approaches.

Methodology in Plain English

The task. Participants debugged a binary face classifier — fictitious, unbeknownst to them — that decided whether a blurred image contains a human face. For each image they saw the image plus a bounding box representing where the model thought the face was. If they agreed, they clicked "Confirm"; if not, they moved the box onto the face. They had 6 seconds per image and could change only the box's position, not its size.

Images. All images came from the Open Images Dataset V7. The authors selected images with a single, unoccluded, front-facing human face and applied heavy Gaussian blur (85 × 85 kernel, σ = 40) to make the task non-trivial and avoid ceiling effects compared with Honeycutt et al. (2020). Model box placements were generated to be correct (perfect overlap with ground truth), partially wrong (25% overlap), or wrong (no overlap).

Difficulty calibration. Two pilot studies (N = 19 and N = 20, recruited via Prolific with the same criteria) selected easy images from those with the highest participant accuracy and difficult images from those with the lowest. The first pilot selected the 40 images for Experiment 1; the second selected the 56 images for the first session of Experiment 2. Experiment 2's second session reused the 40 images from Experiment 1, except four wrong images (two easy, two difficult) whose boxes were altered to partially wrong, because order effects in Experiment 1 were most evident on partially wrong trials.

Independent variables. Order (between-subjects: increasing Inc, constant Const, decreasing Dec), Placement (within-subject: correct vs incorrect, with incorrect collapsing partially wrong and wrong for analysis), and Difficulty (within-subject: easy vs difficult). Crucially, all conditions saw the same images and the same 60% overall model accuracy; only the timing of correct vs incorrect boxes differed.

Experiment 1 (within-session). 359 participants (M_Age = 34.58 ± 9.72; 51% female), split Inc = 119, Const = 121, Dec = 119. After 6 warm-up trials, one session of 40 images. Model accuracy moved from 40% to 80% (Inc), stayed at 60% (Const), or moved from 80% to 40% (Dec) between the first and second halves of the session. Image order was randomized once and fixed within each condition.

Experiment 2 (between-session). 354 participants (M_Age = 36.33 ± 10.64; 47.5% female), split Inc = 121, Const = 117, Dec = 116. Two sessions of 40 images each. First-session model accuracy differed by condition (Inc 40%, Const 60%, Dec 80%); after a 5-second pause in which participants were told the model was being updated from all users' feedback, the second session was identical in images and order for everyone (60% accuracy). Within sessions, errors were spread evenly to minimize within-session effects.

Recruitment and incentives. Prolific, with an approval rate of at least 98%, paid at the platform's suggested hourly rate (£1 for Experiment 1, £1.30 for Experiment 2). Three randomly selected participants who answered five randomly selected trials correctly received a £10 bonus; both random draws were actually carried out and the bonus awarded.

Power and exclusions. A simulation-based a priori power analysis in R indicated that at least 330 participants evaluating 40 images each would give 82% power to detect a small-to-medium interaction among the three independent variables and their main effects. Excluded were trials with no response and trials whose last input came at 5.95 sec or later out of the 6 available. Exclusion rates were low and balanced (Experiment 1: 8%, 7%, 8%; Experiment 2: 6%, 6%, 7%), and prior programming/debugging experience did not differ across conditions (p = .188 and p = .278). The authors note that not excluding the late-response trials did not change the results.

Analysis. Mixed linear models with order condition, placement, difficulty and their interactions as fixed effects, and random intercepts for participants and images; Bonferroni-corrected post-hoc comparisons for multi-contrast effects; Kruskal-Wallis rank sum tests for the averaged questionnaire index. Accuracy and agreement were computed as intersection-over-ground-truth-area and intersection-over-model-box-area respectively.

Why This Matters

Impact on research. The paper directly counters the concern — raised by Nourani et al. (2021) and partially by Honeycutt et al. (2020) — that explanatory interaction is unsafe because presentation order corrupts user feedback. By using a realistic XIL-style task, behavioral outcome measures rather than self-reports alone, and a between-subjects comparison where Inc and Dec groups act as each other's control, the authors show the effect is small, confined to within-session settings and specific stimuli (difficult images), and invisible in self-reported trust. It also reframes the field's emphasis: most XIL work has focused on algorithmic factors (how to integrate feedback, which explanation type to use), while this study argues human factors are less of a barrier than assumed.

Real-world applications (drawn from the paper's own examples and task design):

• Medical imaging diagnosis — XIL can surface confounding cues in X-ray scans, such as background artifacts correlated with the label but not causally related, that standard accuracy metrics miss.
• Spam filter personalization — laypeople tailoring a spam filter to their own needs through explanatory feedback.
• Scientific data curation — domain experts rectifying confounded feature usage in scientific studies.
• Image classification and face detection / tagging — the exact task used here: users accept or correct bounding-box explanations of where a detector looked.

Industry relevance. Teams shipping human-in-the-loop or "feedback-driven" AI (annotation and debugging interfaces, model-customization tools, data labeling pipelines) can treat item ordering as a minor implementation detail rather than a design risk. The finding that participants appear to reset expectations after being told the model was updated suggests that simply informing users about model updates may be enough to prevent carry-over effects across sessions — a cheap, practical safeguard.

Future Directions

• Extend beyond image classification to other prediction tasks, and beyond bounding boxes/saliency maps to other widely used explanation families such as concept-level explanations (Stammer et al., 2021) and concrete examples (Zylberajch et al., 2021).
• Investigate other types of order effects — for instance those arising from the sequence in which more or less challenging items or explanations are presented, which could affect accuracy by modulating users' confidence in their own ability.
• Determine the optimal procedure for structuring debugging sessions, including how and when users should be informed of model updates, since the observed reset between sessions is described as warranting further investigation.
• Examine whether XIL implementations are effective across different types of tasks, and whether other cognitive biases — beyond order effects — might more substantially hinder real-world adoption.

Target Audience

Researchers and practitioners in explainable AI, human-in-the-loop machine learning, and interactive machine learning who design or evaluate XIL systems; HCI and cognitive-science researchers studying human factors and decision biases in AI interaction; and industry teams building user-facing model debugging, customization, or annotation tools who need evidence on whether presentation order is a real design hazard. The paper is readable without deep technical background in ML algorithms, though the statistical results (mixed models, F statistics, Bonferroni-corrected post-hocs) are presented at a level suited to readers comfortable with experimental psychology or empirical ML evaluation.

Note: The provided text is truncated partway through Appendix A.3 (the pilot-study accuracy table for easy vs difficult images is cut off mid-row), so subsequent appendix content is not covered here. The paper does not report effect sizes for the questionnaire analyses beyond the mean index values and p-values.

Authors’ abstract

Explanatory Interactive Learning (XIL) is a powerful interactive learning framework designed to enable users to customize and correct AI models by interacting with their explanations. In a nutshell, XIL algorithms select a number of items on which an AI model made a decision (e.g. images and their tags) and present them to users, together with corresponding explanations (e.g. image regions that drive the model's decision). Then, users supply corrective feedback for the explanations, which the algorithm uses to improve the model. Despite showing promise in debugging tasks, recent studies have raised concerns that explanatory interaction may trigger order effects, a well-known cognitive bias in which the sequence of presented items influences users' trust and, critically, the quality of their feedback. We argue that these studies are not entirely conclusive, as the experimental designs and tasks employed differ substantially from common XIL use cases, complicating interpretation. To clarify the interplay between order effects and explanatory interaction, we ran two larger-scale user studies (n = 713 total) designed to mimic common XIL tasks. Specifically, we assessed order effects both within and between debugging sessions by manipulating the order in which correct and wrong explanations are presented to participants. Order effects had a limited, through significant impact on users' agreement with the model (i.e., a behavioral measure of their trust), and only when examined withing debugging sessions, not between them. The quality of users' feedback was generally satisfactory, with order effects exerting only a small and inconsistent influence in both experiments. Overall, our findings suggest that order effects do not pose a significant issue for the successful employment of XIL approaches. More broadly, our work contributes to the ongoing efforts for understanding human factors in AI.