Explainability Assistant: A Conversational XAI Interface for Interpreting Energy Consumption Models

AI summary

Overview

Research area: Explainable AI (XAI), conversational interfaces, and applied machine learning for building energy management.

Technical level: Intermediate. The paper is written accessibly, but readers will get more from it with basic familiarity with post-hoc explanation methods (SHAP, counterfactuals), LLM function calling, and regression/classification model evaluation.

Scope: The paper presents and evaluates a conversational XAI system that lets non-ML experts interrogate energy consumption forecasting models through natural language, replacing rigid grammar-based parsing with LLM function calling.

What This Paper Is About

Machine learning models that forecast building energy consumption are accurate but opaque, and the facility managers who rely on their predictions lack practical ways to ask why a given prediction was made. Traditional XAI dashboards expose explanations, such as feature importance and what-if analysis, but demand technical expertise and force users into fixed workflows. This paper builds a conversational assistant that translates plain-language questions directly into explainability operations, aiming to make model interpretation accessible to domain experts without data science training.

Key Contributions

1. A conversational architecture built on LLM function calling. Instead of fine-tuning language models on custom grammars as the prior TalkToModel system did, the Explainability Assistant supplies a modern LLM with JSON specifications of available explanation functions and lets it select and parameterize the right calls.

2. A lightweight, modular system design. The front-end (Next.js) and back-end (FastAPI) are decoupled, explanation execution runs locally, and new datasets or model types can be onboarded through configuration changes alone rather than retraining.

3. Empirical parsing accuracy gains. The system reaches 94% intent-parsing accuracy with Gemini-2.5-Flash, compared to the 76.8% reported for the previous grammar-based conversational XAI system.

4. Expert validation in a real energy domain. Three energy specialists compared the conversational interface against a traditional Explainer Dashboard on parallel tasks, showing maintained or improved task accuracy and unanimous preference for the conversational system.

Main Findings

• Parsing accuracy clears the production threshold. Gemini-2.5-Flash achieved 94% exact-match intent parsing on dataset A+B and 93% on A+C; Gemini-2.0-Flash reached 88% and 91%. GPT-5-mini (74%, 79%) and Llama-3.3-70B (78%, 74%) trailed noticeably, showing that model choice matters for reliability.

• Structured output improves transparency. The LLM must return a JSON payload with a mandatory free-text explanation field describing its planned actions, so users can inspect what the system intends to do before any function executes.

• Data stays local. Only the user query and tool schemas travel to the external LLM provider; prediction and explanation execution happens entirely inside the back-end, so proprietary datasets and model internals are not exposed.

• Task accuracy held up under expert use. Participants answered 100% of questions correctly with the conversational assistant versus 93% with the dashboard, though the sample was too small for statistical significance.

• Usability ratings improved sharply and converged. All three experts rated "Easy to use" and "Would use again" at 5 for the assistant, compared with dispersed ratings (2-5) for the dashboard.

• Trust ratings stayed deliberately moderate. "Trust outputs" and "Understand system" scored 3-5 for the assistant, which the authors read as appropriate epistemic caution rather than blind reliance, consistent with their design goal of supporting calibrated trust.

• Architecture transfers across problem types. Switching from electricity demand regression to heart disease classification required only updating JSON function specifications, with no changes to core logic and no fine-tuning.

Methodology in Plain English

The researchers built a chat interface backed by a server that holds the ML model and the explanation tools. When a user types a question, the server sends the query, the conversation history, and a catalogue of available functions to a large language model. The LLM picks which functions to call and with what arguments. The server runs those functions locally, using SHAP for feature importance and DiCE for counterfactual generation, and returns results to the user.

To test how well the LLM understood user intent, the authors assembled three datasets of questions paired with correct function calls. One set of 20 examples was written by hand to cover the system's 20 functions; two larger sets of 80 each were generated with frontier models and then manually verified and corrected. Four cost-effective models were scored on exact-match accuracy across combined sets.

For the user study, three energy professionals with 10-20 years of sector experience completed five parallel tasks on a conventional dashboard and five on the conversational assistant, drawn from a simulated building energy management scenario. They then rated each interface on ease of use, trust, understanding, and future adoption using five-point scales, plus open-ended feedback.

Why This Matters

Impact on research. The work shows that the accuracy ceiling of grammar-based conversational XAI is an artifact of the architecture, not a fundamental limit. Moving to LLM function calling raises intent parsing above 90% without task-specific fine-tuning, which reframes how conversational XAI systems should be built going forward. The paper also supplies open-source code, gold parse datasets, and evaluation scripts, making its claims directly testable.

Real-world applications:

• Building energy management: facility managers can ask why a forecast spiked and what input change would lower it, turning model output into operational decisions.
• Clinical decision support: the demonstrated heart disease classifier case shows the same interface can explain risk-factor contributions to clinicians without bespoke development.
• Industrial process monitoring: operators in manufacturing or utilities could interrogate anomaly detection and demand forecasting models in operational language.
• Regulatory and audit contexts: the JSON explanation field and local execution support traceability and data governance requirements for automated decision systems.

Industry relevance. The system needs only configuration changes to support new models and datasets, which lowers deployment cost for companies that already have trained models but no interpretability layer. Because sensitive data never leaves the back-end, the architecture fits enterprises with strict privacy constraints. The evaluation also suggests vendors can retire complex custom XAI dashboards in favor of conversational front-ends for many user-facing scenarios.

Future Directions

• Larger and more diverse user studies. The current validation rests on three experts in one domain; broader populations across healthcare, finance, and industry are needed to establish generalizability.

• Counterbalanced experimental design. Because participants always used the dashboard first, learning effects may have inflated the conversational system's ratings. Randomized interface ordering would control for this.

• Hybrid conversational-visual interfaces. Participants noted that dashboards remain useful for global orientation, suggesting value in combining dialogue with targeted visualizations rather than replacing one with the other.

• Longitudinal adoption research. Tracking how experts use the system over weeks or months would reveal whether initial usability advantages translate into sustained practice, and whether query patterns shift as familiarity grows.

Target Audience

Researchers working on conversational XAI and human-AI interaction will find the parsing benchmarks and architectural comparison directly useful. Applied ML engineers and product teams building interpretability features for non-technical users will benefit from the modular front-end/back-end separation, the function-calling pattern, and the privacy-preserving execution model. Domain experts in energy management, and by extension other operational forecasting fields, are the intended end users and will recognize the usability problems the paper targets. Practitioners evaluating whether to build on TalkToModel-style grammars or migrate to LLM function calling will find the 76.8% versus 94% comparison especially decisive.

Authors’ abstract

Energy consumption forecasting relies on increasingly complex machine learning (ML) models, such as Genetic Programming-based symbolic regressors, whose predictions can be difficult for facility managers and building operators to interpret. Explainable Artificial Intelligence (XAI) techniques address this opacity, but traditional XAI dashboards require substantial technical expertise and provide limited flexibility for dynamic, context-aware inquiry. Conversational XAI systems offer a promising alternative; however, previous approaches, such as TalkToModel, were constrained by rigid custom grammars and achieved only 76.8% intent-parsing accuracy. This paper introduces the Explainability Assistant, an open-source conversational XAI system that leverages the function-calling capabilities of modern Large Language Models (LLMs) to overcome these limitations. The system achieves 94% intent-parsing accuracy, supports flexible natural language interaction, and adapts to different ML problem types without task-specific fine-tuning. We present the system's architecture and report results from a comparative evaluation conducted with energy domain specialists, contrasting the Explainability Assistant with a traditional XAI dashboard. The evaluation suggests improved usability and consistent task accuracy, with all experts unanimously preferring the conversational interface for practical use.