The Impact of Concept Explanations and Interventions on Human-Machine Collaboration

AI summary

Overview

Research area: Human-Computer Interaction and Explainable AI (XAI), specifically human evaluation of Concept Models such as Concept Bottleneck Models (CBMs) in collaborative human-machine tasks.

Technical level: Intermediate. The paper assumes familiarity with deep neural networks, interpretability concepts, and basic statistics (t-tests, error bars), but the study design and findings are explained in accessible terms.

One-sentence scope: Two human studies (an expert dermatology study and a lay-person Blackjack study) test whether CBMs and their intervention capability actually improve human-machine task accuracy, interpretability, and trust when people — not automated metrics — are the ones intervening.

What This Paper Is About

Deep neural networks (DNNs) are often treated as black boxes, so Concept Models (CMs) such as Concept Bottleneck Models (CBMs) were introduced to predict human-defined concepts as an intermediate step before predicting task labels, improving interpretability. The original CBM paper showed that replacing incorrect concept predictions with ground truth values — known as intervening — improved model task accuracy, and claimed improved human-machine collaboration, but that result was produced with automated metrics and without human evaluation. This paper runs the first human studies using CBMs in collaborative task settings to determine whether those findings hold when humans perform the interventions.

Key Contributions

1. The authors perform the first human studies using CBMs in a joint human-machine task setting, analysing the interaction between humans and the CBM. They find interventions often increased trust in a model but that this trust was sometimes misplaced, and that the CBM decision-making process is not aligned with that of the humans.

2. They show the initial promise of interpretability from high-level concepts is upheld, but that understanding the model's decision-making process required participants to actively interact with the model. Providing concept predictions without the capability to intervene had a similar effect on task accuracy.

3. They establish that although they used CBM, the findings also apply to other CMs with intervention capabilities that predict task labels in the same feed-forward fashion from input to concepts to task label — namely Concept Embedding Models, Sidecar CBM, and hybrid CBM.

4. They provide two open datasets and study codebases (a Skincon-based expert study and a new Blackjack dataset), running two studies that span both human-grounded (simulated task, lay participants) and application-grounded (real-world task, expert participants) settings following an established taxonomy.

Main Findings

• CBMs improve alignment, not necessarily accuracy: CBMs improved interpretability compared to standard DNNs, leading to increased human-machine alignment, but this increased alignment did not translate to a significant increase in task accuracy. In the lay-person study, a one-tailed t-test gave a p-value of 0.27 when comparing participants who performed interventions against all NoExp and CExp participant groups, and 0.195 when compared against participants who did not intervene but had the capability.

• Concept explanations alone helped accuracy: In the lay-person study, a one-tailed t-test with a p-value of 0.042 showed CExp improved task accuracy compared to NoExp. In the lay-person study, Acc - CExp participants reached 91.0% accuracy and Inacc - CExp reached 81.4%.

• Interventions increased alignment: Across all participant groups in the lay-person study, interventions improved human-machine alignment (WithInt 86.7% vs NoInt 77.1%), with p-values of 0.041 for all participants and 0.04 for those capable of performing interventions. Concept explanations also raised alignment, with a p-value of 0.036 when comparing CExp to NoExp participants.

• Direction of alignment effects differed between studies: In the expert study, a slight decline in alignment was observed with WithInt participants (78.1% vs 81.8%) and CExp+Int+SMap participants (80% vs 81.7%), with a one-tailed t-test p-value of 0.03 confirming that the absence of interventions led to increased alignment. Initial alignment with interventions (65.6%) was closer to the model's actual accuracy (70%), indicating interventions helped participants calibrate trust.

• Intervention behaviour differed by model quality: 65.4% of interventions were performed on the inaccurate model and 34.6% on the accurate model. Participants with an accurate model primarily performed feature adjustments, while those with an inaccurate model split interventions nearly evenly between error correction (47.2%) and feature adjustments (52.7%). Binary interventions were common with inaccurate models (47 vs 11 concepts intervened).

• Participants explored before settling: The number of interventions declined over time in both studies, consistent with participants initially exploring model sensitivity before developing a mental model. However, for incorrectly predicted concepts in the lay-person study, participants consistently performed around 2 interventions per sample. One exception was Acc - CExp+Int participants, who showed a spike at game 13 with an average of 7 interventions and a standard error of ±2, which the authors note was not representative because it occurred once.

• Test-time intervention accuracy did not improve as in the original CBM work: With human-performed interventions, task accuracy for the most part did not improve compared to no interventions. The exception was the lay-person study's inaccurate model, where interventions increased or matched the model's initial task accuracy. Concept precision and recall increased or matched in the expert study and the inaccurate-model lay-person study.

• Saliency maps showed no significant alignment impact: The difference in alignment between CExp+Int and CExp+Int+SMap participants in the expert study was not statistically significant (p-value of 0.59). A one-tailed t-test gave a p-value of 0.09 for accuracy being higher with saliency maps.

• Explanations rated around average to good: In the expert study, SCS overall scores were almost all 0.68 or slightly below, which the paper describes as indicating an average response; the subgroup who selected "strongly agree" for skin disease experience scored 0.71. In the lay-person study all overall scores were 0.70 or above, with the highest score reported as 0.78.

Methodology in Plain English

The researchers ran two human studies in which people worked alongside a model that acted as a second opinion.

Expert study: 12 participants — doctors, consultants or trainees with dermatology expertise — diagnosed skin conditions in 10 images drawn from the Skincon dataset, restricted to samples labelled "malignant melanoma" and "seborrhoeic keratosis," shown in random order. Participants were split into two groups, both with access to the same model and both with intervention capability: CExp+Int and CExp+Int+SMap.

Lay-person study: 104 participants, mostly university staff or students, each played 15 games of Blackjack — the first without the model and the other 14 with model predictions. Each game had a maximum between 1 and 7 moves using a single deck; betting was removed and a score was added that rose or fell with wins and losses. Participants could choose hit, stand, or surrender. They were split into eight groups varying the model (accurate vs inaccurate) and the explanations provided (NoExp, CExp, CExp+Int, CExp+Int+SMap).

In both studies, participants could adjust intervention sliders for each concept, and adjusting a slider automatically updated the model's predicted task label. For every sample or move, participants reported whether they were influenced by the AI's suggestion, by the concepts the AI detected, or not influenced at all. At the end they completed a closing survey including System Causability Scale (SCS) questions.

The researchers classified interventions into two kinds: error correction (a concept intervened at most once per sample with the intervened value in the range 0 ≤ c̄ ≤ 0.1 or 0.9 ≤ c̄ ≤ 1) and feature adjustment (all other interventions, including

Authors’ abstract

Deep Neural Networks (DNNs) are often considered black boxes due to their opaque decision-making processes. To reduce their opacity Concept Models (CMs), such as Concept Bottleneck Models (CBMs), were introduced to predict human-defined concepts as an intermediate step before predicting task labels. This enhances the interpretability of DNNs. In a human-machine setting greater interpretability enables humans to improve their understanding and build trust in a DNN. In the introduction of CBMs, the models demonstrated increased task accuracy as incorrect concept predictions were replaced with their ground truth values, known as intervening on the concept predictions. In a collaborative setting, if the model task accuracy improves from interventions, trust in a model and the human-machine task accuracy may increase. However, the result showing an increase in model task accuracy was produced without human evaluation and thus it remains unknown if the findings can be applied in a collaborative setting. In this paper, we ran the first human studies using CBMs to evaluate their human interaction in collaborative task settings. Our findings show that CBMs improve interpretability compared to standard DNNs, leading to increased human-machine alignment. However, this increased alignment did not translate to a significant increase in task accuracy. Understanding the model's decision-making process required multiple interactions, and misalignment between the model's and human decision-making processes could undermine interpretability and model effectiveness.