KnowThyself: An Agentic Assistant for LLM Interpretability

AI summary

Overview

Research area: LLM interpretability tooling, multi-agent systems, and human-facing AI interfaces (a demonstration-track systems paper).

Technical level: Intermediate. The overall idea is accessible, but it assumes familiarity with terms such as token attribution, attention heads, and bias metrics like toxicity, regard, and HONEST.

Scope: The paper presents KnowThyself, a conversational, multi-agent platform that routes natural language interpretability questions to specialized tool agents and returns interactive visualizations with plain-language explanations.

The paper is a preprint (arXiv:2511.03878v1 [cs.AI], 05 Nov 2025) and has been accepted for publication at the Demonstration Track of the 40th AAAI Conference on Artificial Intelligence (AAAI'26).

What This Paper Is About

Understanding why a large language model produces a given output is hard, and the tools that exist for this purpose are scattered across separate libraries that each require programming expertise to run. KnowThyself aims to close that gap by letting a user upload a model, ask a question in ordinary language, and receive both a visualization and an explanation, all inside a single chat interface rather than a stack of notebooks and scripts.

Key Contributions

1. A multi-agent orchestration framework that coordinates a broad range of interpretation tasks, enabling flexible routing of user queries and producing coherent explanations rather than raw tool outputs.
2. A modular architecture in which each interpretation method is encapsulated as an independent agent, so new tools can be added without altering core components.
3. An interactive visualization interface that pairs graphical outputs with natural language explanations, intended to lower the technical barrier to effective model inspection.
4. A working, publicly released implementation (https://github.com/spygaurad/KnowThyself) combining an orchestrator LLM, an agent router, four specialized agents, and a conversational front end.

Main Findings

• Four-part system design: KnowThyself consists of an Orchestrator LLM for query reformulation and explanation, an Agent Router for selecting tools, Specialized Agents for analysis, and a Conversational Interface for interaction.
• Routing by embedding similarity: The Agent Router dispatches queries using embedding-based similarity search that matches user intent against agent descriptions, with the option to add LLM-based routing as the system scales.
• Four integrated agents: BertViz for attention visualization, TransformerLens for fine-grained layer- and head-level activations, a RAG explainer that grounds responses in domain literature, and BiasEval for safety and demographic disparity assessment.
• Bias measured with three established scores: BiasEval reports toxicity, regard, and HONEST scores, prompting models with the Real Toxicity Prompts, BOLD, and HONEST datasets.
• Concrete implementation stack: The system is built with LangGraph as a directed graph of agents over a shared state, uses the Ollama-hosted nomic-embed-text model for routing, Gemma3-27B for orchestration, and FAISS for the RAG agent's similarity search.
• Demonstration models included: GPT-2, BERT, and LLaMA2-13B are pre-included for demonstration, with large models served through Ollama; the system can run locally when resources permit, avoiding third-party APIs.
• Worked use cases: A user can upload a LLaMA2 checkpoint and ask how the model attends across tokens for the word "she"; the router selects TransformerLens, the orchestrator synthesizes an input sentence ("Maria went to the library because she needed a book.") when none is given, and the attention map is returned and explained. In the same session, a question about gender bias is identified as a new task, BiasEval chooses a submodule such as regard, samples prompts from BOLD, runs them on the user model, and computes scores.
• Stated limitations: The current implementation integrates only a limited set of tools, requires additional engineering to adapt non-modular libraries, and supports text inputs exclusively.
• No quantitative evaluation is reported: The paper does not report accuracy figures, benchmark scores, user-study results, or comparisons against other interpretability platforms.

Methodology in Plain English

The authors treat interpretability as an orchestration problem rather than a modeling problem. A supervisory LLM sits at the front of the pipeline and rewrites whatever the user typed into something a tool can act on, generating subtasks such as synthesizing an input sentence or selecting a tool. A router then compares the rewritten request against short descriptions of the available agents using embedding similarity, and sends it to whichever agent matches best. Each agent is a self-contained plug-in that carries its own dependencies, so, for example, TransformerLens brings along HookedTransformer while BertViz relies on HuggingFace Transformers, and neither one interferes with the others. To keep the system trustworthy, the RAG agent indexes documents and retrieves relevant literature that the orchestrator folds back into its explanation as grounding context. Once the tool returns its output, the orchestrator converts it into a narrative explanation, and everything is surfaced in a chat window where users upload models and ask follow-up questions.

Why This Matters

Impact on research. Interpretability research has advanced faster than its tooling has become usable, creating a gap between published methods and practitioners who could benefit from them. KnowThyself argues that wrapping existing methods in an agentic, conversational layer is itself a contribution, making emerging interpretation techniques easier for broader audiences to engage with and providing an extensible platform other researchers can build on.

Real-world applications:

• Auditing a deployed language model for gender or demographic bias using the built-in BiasEval agent before release.
• Debugging unexpected model behavior by inspecting attention patterns and layer-level activations without writing code.
• Supporting analysts, journalists, or auditors who need to interrogate an open model but lack deep machine learning engineering skills.
• Running sensitive interpretability analyses entirely on local hardware, which the paper notes is possible when resources permit, so that proprietary models and prompts are never sent to third-party APIs.

Industry relevance. Teams that ship LLM-based products face growing pressure to explain model decisions for transparency, trust, and accountability. A chat interface that returns visualizations plus grounded explanations offers a lower-friction path to that kind of inspection than maintaining custom scripts per method, and the modular agent design means an organization can bolt on its own internal tools.

Future Directions

1. Broaden tool coverage beyond the current four agents, since the paper notes the present implementation integrates only a limited set of tools.
2. Extend support to multimodal models, moving past the current text-only input restriction.
3. Improve routing precision for overlapping tasks, where distinct user intents may map to similar agent descriptions.
4. Add richer visualization capabilities to support deeper and more transparent interpretive insights.
5. Reduce the engineering effort needed to adapt non-modular libraries, which the authors identify as a current practical obstacle.

Target Audience

This paper suits LLM practitioners, interpretability researchers, and tooling or platform engineers who want to inspect model behavior without rewriting each method into ad hoc scripts. It is also relevant to AI governance, safety, and audit teams looking for a conversational front end to bias evaluation and attribution analysis, and to researchers interested in multi-agent architectures as a way to package and route scientific software. Readers seeking quantitative benchmark evidence should look elsewhere, since the paper is a demonstration-track system description and reports no empirical measurements.

Authors’ abstract

We develop KnowThyself, an agentic assistant that advances large language model (LLM) interpretability. Existing tools provide useful insights but remain fragmented and code-intensive. KnowThyself consolidates these capabilities into a chat-based interface, where users can upload models, pose natural language questions, and obtain interactive visualizations with guided explanations. At its core, an orchestrator LLM first reformulates user queries, an agent router further directs them to specialized modules, and the outputs are finally contextualized into coherent explanations. This design lowers technical barriers and provides an extensible platform for LLM inspection. By embedding the whole process into a conversational workflow, KnowThyself offers a robust foundation for accessible LLM interpretability.