Simplifying Outcomes of Language Model Component Analyses with ELIA

AI summary

Overview

Research area: Natural Language Processing / mechanistic interpretability, with a focus on human-computer interaction and explainable AI (XAI) tooling.

Technical level: Intermediate. The paper presumes familiarity with interpretability concepts such as attribution, activation vectors, and circuit tracing, but the framing is about usability and communication rather than new mathematical machinery.

Scope: The paper designs, builds, and empirically evaluates ELIA, a Streamlit web application that wraps three LLM interpretability techniques in an interactive interface and adds vision-language-model-generated natural language explanations plus automated faithfulness checks.

What This Paper Is About

Mechanistic interpretability has produced powerful tools for reverse-engineering the internals of large language models, but those tools output heatmaps, graphs, and high-dimensional plots that require specialist expertise to read. The authors argue this creates an accessibility gap that excludes developers, domain experts, and policymakers. Their goal is to close that gap by building ELIA, an interactive web application that integrates three interpretability techniques and uses a vision-language model to translate the resulting visualizations into plain-language narratives, then testing whether this actually improves comprehension across different experience levels.

Key Contributions

1. An integrated interpretability application (ELIA) that unifies three complementary analyses — Attribution Analysis, Function Vector Analysis, and Circuit Tracing — inside a single user-centered Streamlit interface, with full English and German support.

2. A novel explanation methodology that feeds the visualizations produced by these analyses, together with structured textual data, to a vision-language model (Qwen2.5-VL-72B) to automatically generate natural language explanations (NLEs).

3. An automated faithfulness verification pipeline that extracts atomic factual claims from each generated narrative, checks quantitative claims against ground-truth analysis data, and uses the explainer model as a constrained fact-checker for semantic claims, reporting verified/contradicted status to users.

4. A mixed-methods user study with 18 participants providing empirical evidence that AI-powered explanations reduce comprehension gaps between users with different levels of prior LLM experience, and that users prefer interactive, explorable interfaces over static visualizations.

Main Findings

• Faithfulness scores were high overall but uneven across methods. Verification results ranged from 86.8% (Saliency, 59/68) to 100.0% (Function Type, 47/47; Layer Evolution, 36/36). Intermediate scores included Integrated Gradients at 91.8% (56/61), Occlusion at 88.0% (66/75), Function Vector Placement at 95.8% (23/24), Categories at 91.7% (44/48), Circuit Overview at 97.8% (45/46), Subnetwork at 96.4% (132/137), and Features at 96.0% (120/125).

• Prior LLM experience did not significantly predict comprehension. A Spearman correlation test found no statistically significant relationship between a user's prior LLM experience and their correctness score (ρ = 0.30, p = 0.23). Average correctness was 1.00 for Experts, 0.98 for Intermediates, and 0.95 for Novices.

• UX ratings differed significantly across pages. The Function Vectors and Circuit Trace pages received significantly more positive UX ratings than the Attribution Analysis page (Kruskal-Wallis H-test, p = 0.006). The "PCA Clarity" metric on the Function Vectors page received a perfect median score of 5, while "Ease of Use" for the Attribution Analysis heatmap received the lowest median score in the study (3).

• Interventions on traced circuits had larger effects than random baselines. Ablating the features and paths identified by the Cross-Layer Transcoder produced a substantially larger impact on the model's output probability than ablating random features or edges, using exemplary prompts covering knowledge retrieval, code generation, and literary analysis. The authors also compute the Circuit Performance Ratio (CPR) metric.

• Users preferred interactive, metaphor-rich visuals. Interview transcripts showed the 3D PCA plot and the Subnetwork Explorer were consistently praised as "very clear," while abstract static visuals such as heatmaps were found confusing. A recurring theme was the need for more integrated, automated guidance, including summaries, tooltips, and adaptive complexity.

• Users were often overwhelmed by too much information at once, particularly the main circuit graph — a tension between detail and simplicity that the authors identify as a central challenge.

• A worked example showed the three methods reinforcing each other. For the prompt "The capital of France is," attribution identified "France" as highest-saliency, function vectors placed the prompt in the "Abstractive Tasks" cluster scoring highly on "Next Item" and "Country Capitals," and circuit tracing showed a progression from early "article usage" and "country-related information" features, through middle-layer "country-related terms," to late-layer "Geographical knowledge" and "country-related phrases."

Methodology in Plain English

The authors built a web application on Streamlit, backed by PyTorch, the Transformers library, Plotly for visualizations, and the Inseq toolkit for attribution. Two models do the work: a "subject model" whose behavior is being analyzed (the 7-billion-parameter OLMo-2, specifically OLMo-2-1124-7B) and a vision-enabled "explanation model" (Qwen2.5-VL-72B) that turns visualizations into readable narratives. API calls to the explainer use low temperature and a fixed seed to keep output largely deterministic.

The Attribution Analysis page implements Saliency, Integrated Gradients, and Occlusion through Inseq, producing an interactive heatmap, plus an Influence Tracer that embeds the user's prompt and performs a k-nearest neighbors search against a pre-computed Faiss index of the Dolma training corpus. The Function Vector Analysis page builds a custom bilingual (English/German) dataset of instructional prompts — 6 function types and 120 total categories with 5 example prompts each — pre-computes a function vector per category by averaging final-layer, final-token activations, and compares new prompts by cosine similarity, visualizing results with a 3D PCA scatter plot, a bar chart, and a sunburst chart. The Circuit Trace Analysis page uses a Cross-Layer Transcoder trained offline on Dolma (batch size 16, 1,500 training steps, 512 interpretable features per layer, JumpReLU activation with threshold 0.0, L1 sparsity penalty λ = 1e-3, Adam at learning rate 3e-4 with cosine annealing and gradient clipping at max norm 1.0) to produce a layer-by-layer circuit graph, local path ablations, and Subnetwork and Feature explorers.

For verification, the explanation model is prompted again to extract its own verifiable claims as a JSON list (following the atomic fact extraction approach of FActScore), after which a verification module programmatically checks quantitative claims against ground-truth data and uses the explainer model — running at temperature 0.0 with a fixed seed and constrained by hard-coded rules, negative constraints, and synonym-mapping directives — to judge the plausibility of abstract semantic claims.

The evaluation was a within-subjects user study with 18 undergraduate computer science students, mostly novice or intermediate in LLM experience, averaging roughly 1 hour to complete. Participants rated each page on a 5-point Likert scale for visual clarity, ease of use, and plausibility, and answered three multiple-choice comprehension questions per page, complemented by interview transcripts.

Why This Matters

The paper reframes interpretability as a communication problem as much as a technical one. If the outputs of circuit tracing and attribution analyses remain legible only to specialists, the broader conversation about AI safety and reliability stays confined to a small circle — the exact concern the authors cite from Weidinger et al. (2023). ELIA offers evidence that a combination of automated explanation and careful interface design can widen that circle without sacrificing analytic depth, and it argues that usability and faithfulness should be treated as co-equal concerns by the interpretability community.

Real-world applications:

• Model auditing and compliance: Auditors or regulators without deep ML backgrounds could inspect why a model produced a given output, supported by verified plain-language narratives.
• AI safety and alignment research: Researchers can use the tool as a guided investigation interface rather than an expert-only diagnostic, lowering the onboarding cost for new team members.
• Developer debugging of deployed LLMs: Developers can trace token-level attribution, check semantic task alignment, and ablate specific circuits to isolate the source of unexpected behavior.
• Domain-expert review in sensitive fields: Clinicians, legal experts, or policymakers could review model behavior in their domain and contribute judgements on plausibility without needing interpretability training.

Industry relevance: The system runs on pre-computed artifacts (an offline-trained CLT, a pre-cached function vector space, a Faiss index) so that live usage only requires forward passes and asynchronous API calls — a design pattern directly relevant to teams that want interpretability features in production interfaces without large local compute costs.

Future Directions

• Broadening language, model, and explainer coverage. ELIA is currently limited to two languages (English and German), OLMo as the explained model, and Qwen-VL as the explainer. Generalizing to other subject models and explainer models is an open step.

• Adding richer intervention tools. The authors note that counterfactual editing or causal scrubbing might be necessary to provide a comprehensive user-centric view of interpreting language model behavior.

• Studying longer-term, professional usage. The user study was limited to subjective ratings and short-term interactions; the authors state that studying longer-term usage in professional settings remains necessary to prove the advantages of ELIA they have recorded.

• Reducing the detail-versus-simplicity tension. Qualitative findings pointed to users being overwhelmed by dense visualizations and relying on a facilitator for context, suggesting a need for summaries, tooltips, and adaptive complexity.

Target Audience

This paper is most useful to HCI and XAI researchers building interpretability interfaces, mechanistic interpretability practitioners who want their outputs to reach non-specialists, and product or platform teams considering embedding interpretability features in developer tools. It also serves as a compact reference for educators teaching model interpretability to students without a research background, and for anyone who wants an empirical account — including verification rates and study statistics — of whether LLM-generated explanations actually help non-experts understand model internals.

Authors’ abstract

While mechanistic interpretability has developed powerful tools to analyze the internal workings of Large Language Models (LLMs), their complexity has created an accessibility gap, limiting their use to specialists. We address this challenge by designing, building, and evaluating ELIA (Explainable Language Interpretability Analysis), an interactive web application that simplifies the outcomes of various language model component analyses for a broader audience. The system integrates three key techniques -- Attribution Analysis, Function Vector Analysis, and Circuit Tracing -- and introduces a novel methodology: using a vision-language model to automatically generate natural language explanations (NLEs) for the complex visualizations produced by these methods. The effectiveness of this approach was empirically validated through a mixed-methods user study, which revealed a clear preference for interactive, explorable interfaces over simpler, static visualizations. A key finding was that the AI-powered explanations helped bridge the knowledge gap for non-experts; a statistical analysis showed no significant correlation between a user's prior LLM experience and their comprehension scores, suggesting that the system reduced barriers to comprehension across experience levels. We conclude that an AI system can indeed simplify complex model analyses, but its true power is unlocked when paired with thoughtful, user-centered design that prioritizes interactivity, specificity, and narrative guidance.