SimXAI: A Simple Drop-in Layer Increases Interaction Quality and Understanding in Conversational XAI

AI summary

Overview

Research area: Natural Language Processing / Explainable AI (conversational XAI, LLM tool calling)

Technical level: Intermediate — assumes familiarity with dialogue-system architectures (intent recognition, dialogue management, NLG) and with LLM function/tool calling.

Scope: The paper proposes SimXAI, a lightweight LLM-based conversational layer that replaces the multi-module conversational stacks of existing conversational XAI agents while leaving the underlying explanation resources untouched, and evaluates it with a new multi-aspect suite plus a 30-participant user study.

What This Paper Is About

Most conversational explainable AI (ConvXAI) agents use "GUS-style" pipelines with three separate parts: intent recognition, dialogue management, and template-based response generation. These parts require fine-tuned models, handcrafted intent data, rules, and templates, which makes them hard to adapt and weak at follow-up questions that depend on context. The authors ask whether a single off-the-shelf tool-calling LLM can serve as a drop-in replacement for that entire conversational layer, keeping the host system's explanations unchanged and adding a way to measure whether such a layer is actually good.

Key Contributions

1. SimXAI, a modular LLM-centric conversational layer. It uses one off-the-shelf tool-calling LLM plus general function specifications to map user requests onto the host system's existing XAI resources. It requires no fine-tuning, handcrafted intent data, templates, or rules, and is described as data- and model-agnostic. The authors stress that the tool-calling mechanism itself is not novel; the contribution is the ConvXAI-specific separation of a replaceable conversational layer from the XAI resources.

2. An extensible XAI function interface. Based on the question bank of Liao and Varshney (2022) and Liao et al. (2020), the authors implement 11 functions — one for retrieving a model prediction and ten for explanatory questions such as why, why not, how to be that, how to still be this, what if, and queries about model, data, performance, and output information. Specifications are written generically rather than tied to a dataset or model.

3. A multi-aspect evaluation suite for ConvXAI layers, covering parsing (single-turn, multi-turn, and dialogue-level), response-to-tool faithfulness, and contextualization (context relevance and contextual answer coherence), each with manually annotated ground truth, metrics, and an automated pipeline.

4. Quantitative evaluation and a user study showing that swapping only the conversational layer of TalkToModel for SimXAI improves task accuracy and perceived interaction quality, with the underlying XAI resources held fixed.

Main Findings

• Parsing accuracy is high across all complexity levels: 86% for single-turn (CoXQL + Compass), 88% for multi-turn (Compass-MultiTurn), 92.5% for SYN-Sentiment, and 96% for SYN-HeartRate. Accuracy is higher on multi-turn and dialogue-level data, which the authors interpret as dialogue context helping the model resolve intent.

• Responses stay grounded in tool output: response-to-tool faithfulness was 93.6% on SYN-QA-Sentiment and 91.5% on SYN-QA-HeartRate — i.e., over 91% in most cases. The authors note this matters because replacing rules and templates with an LLM raises hallucination risk.

• Context-dependent turns are mostly handled coherently: contextual answer coherence reached 90.9% (sentiment) and 94.74% (heart rate).

• LLM-as-a-judge is a promising but uneven proxy: agreement with human annotators was around 90% for response-to-tool faithfulness, around 90% for context relevance in sentiment but only 76.60% in heart rate prediction, and above 80% for contextual answer coherence across both domains.

• User study — accuracy: participants answered 92.1% of questions correctly with SimXAI versus 59.0% with original TalkToModel (p < .001).

• User study — efficiency: 10.0 versus 12.4 chat turns, 5.8 versus 7.4 minutes, and 2.70 versus 1.32 turns per correct answer (SimXAI lower on all).

• User study — perceived quality (BUS-11, 1–5 scale): largest gains in interaction clarity (4.60 vs 3.33), context awareness (4.37 vs 3.13), response comprehensibility (4.57 vs 3.60), goal facilitation (4.37 vs 3.10), information sufficiency (4.37 vs 3.23), and response accuracy (4.47 vs 3.63), all p < .001. Response time was the only item where SimXAI scored lower (4.03 vs 4.37), but the difference was not statistically significant (p > 0.1); Cronbach's alpha was 0.925 and 0.856 respectively.

• Preference: 80.0% of participants preferred SimXAI overall, 16.7% preferred TalkToModel, and 3.3% had no preference; the same proportions held for perceived task effectiveness.

• Qualitative fixes: swapping in SimXAI corrected context-handling failures in TalkToModel, InterroLang, and XAgent — for example, TalkToModel misinterpreting the referent of "it" in a follow-up "What if it was 105?" question.

• Synthetic dialogue realism: five ML-background raters scored generated dialogues 3.75 on average on a 5-point realism scale.

Methodology in Plain English

The authors reframe a ConvXAI agent as two separable things: a conversational layer that talks to the user, and XAI resources (predictions, explanations, model/data cards) that produce the actual explanatory content. They then replace only the conversational layer with a single LLM — ToolACE-2.5-Llama-3.1-8B — given a small set of documented functions.

At each turn, the system decides whether a function call is needed at all. If it is, it picks a function, fills in the arguments from the current query and the dialogue history, and if required arguments are missing it searches the history and, failing that, asks the user a clarifying question. It then calls the function and writes a natural-language answer grounded in the returned output.

To test this, the authors assemble parsing datasets of increasing difficulty: existing single-turn benchmarks (CoXQL, Compass), a new short multi-turn set built from Compass by merging related questions and replacing repeated instance text with pronouns, and two synthetic dialogue-level datasets generated with GPT-5.1 in "constrained" (covering all 11 functions) and "unconstrained" modes for sentiment analysis and heart-rate prediction. Faithfulness is assessed on 71 question-answer pairs where a gold function output exists and a binary judgment records whether the response preserves it. Contextualization is split into whether a query needs prior context at all, and whether the resulting answer coheres. Human annotations are compared against an LLM judge (Llama-3.1-8B-Instruct) to see whether evaluation can be automated.

Finally, they conduct a within-subjects user study with 30 lay participants on Prolific ($8/hour) using the Diabetes dataset from the original TalkToModel study. Each participant uses both agents across three tasks totalling nine questions, with agent order randomized and specific values varied to reduce learning effects. They measure answer accuracy, turns per correct answer, breakdown indicators, the 5-point BUS-11 usability scale, and a pairwise preference questionnaire.

Why This Matters

The paper argues that the bottleneck in conversational XAI is not the explanations themselves but the rigid handcrafted conversational control around them. Showing that a single 8B tool-calling LLM can replace that control without degrading faithfulness — and that users then perform markedly better — reframes how ConvXAI systems could be built and evaluated. It also contributes a reusable benchmark suite for a component that prior work mostly evaluated turn-by-turn without dialogue context.

Real-world applications:

• Clinical or health-risk model interfaces, where lay users need to ask follow-up questions about a prediction (the study's diabetes setting) without re-specifying the patient each time.
• Credit, loan, and insurance decision tools, where applicants ask "why was I rejected" and then "what would change it" — directly reflected in the XAgent loan example in the paper.
• Data science and ML operations tooling, letting analysts query model performance, data information, and feature attributions in natural language on top of existing SHAP or counterfactual backends.
• Regulated-domain explanation portals (e.g., under transparency requirements), where an organization already has validated explanation methods and only needs a conversational front end that does not alter the underlying outputs.

Industry relevance: the layer is drop-in, open-source, and requires only function implementations over existing resources, which lowers integration cost for organizations that already own XAI infrastructure. The latency trade-off (response time rated slightly lower, though not significantly) is a practical consideration for deployment, and the authors attribute it to TalkToModel's parsing models ranging from 60M to 6B parameters versus the tool-augmented LLM used here.

Future Directions

• Fully automated evaluation. The authors plan to strengthen automated assessment of LLM-based ConvXAI layers, motivated by the uneven LLM-judge agreement on context relevance (76.60% in heart rate prediction versus around 90% in sentiment).
• How LLM choice affects tool calling. The paper explicitly does not study this; all experiments use one lightweight 8B LLaMA 3.1 model, and the authors note results may vary with other models. The same gap applies to the choice of LLM judge.
• Backend explanation quality. SimXAI only improves conversation and assumes appropriate XAI resources exist; a response can be faithful to a tool output that is itself incomplete or misleading. Connecting conversational quality to validated explanation faithfulness is left open.
• Mitigating latency and hallucination. Both are named as limitations of the LLM-based design, alongside extending the function set when no predefined function matches a user's intent.

Target Audience

Researchers and practitioners building conversational or agentic interfaces on top of explainable AI systems — particularly those who already have explanation backends (feature attributions, counterfactuals, model/data cards) and want a conversational front end without maintaining intent models and templates. It is also useful for HCI and XAI evaluation researchers interested in benchmarking dialogue-level contextualization and response-to-resource faithfulness, and for engineers who need a concrete, open-source reference architecture for tool-calling over XAI resources. Readers should be comfortable with dialogue-system terminology and LLM function calling; the paper is not an introduction to XAI methods themselves.

Authors’ abstract

Conversational explainable AI (ConvXAI) agents often use GUS-style architectures with separate modules for intent recognition, dialogue management, and template-based response generation. While faithful to the underlying XAI methods, these pipelines are hard to adapt and often struggle with context-dependent follow-up questions. We present SimXAI, a lightweight LLM-centric conversational layer that replaces these modules with a single tool-calling LLM that bridges access to existing XAI resources. Rather than a standalone agent, SimXAI serves as a modular interface that enables conversational access to external XAI resources. We further introduce an evaluation suite for single- and multi-turn parsing, faithfulness, and contextualization. Across existing and new datasets, SimXAI achieves strong function-selection accuracy and high faithfulness while better handling context-dependent requests. In a user study replacing only the conversational layer of TalkToModel, a representative GUS-based agent, users completed tasks more accurately and in fewer turn. Overall, SimXAI improves GUS-based design while preserving faithfulness and improving conversational quality.