Interpreto: An Explainability Library for Transformers

AI summary

Overview

Research area: Natural Language Processing / Explainable AI — interpretability tooling for Transformer language models.

Technical level: Intermediate. Readers need basic familiarity with HuggingFace models, tokenization, and the distinction between prediction tasks (classification vs. text generation), but the paper is a system/tooling description rather than a new algorithmic result.

Scope (one sentence): The paper presents Interpreto, an open-source MIT-licensed Python library that unifies attribution methods and concept-based explanations behind a single API for interpreting HuggingFace language models, covering both classification and text generation.

What This Paper Is About

Understanding why a Transformer language model produced a given output matters for debugging, bias detection, and safety, but the tools for doing so are scattered across separate packages, often target only one modality or task, and sometimes lack documentation or evaluation metrics. Interpreto addresses this by packaging two complementary families of explanation methods — attributions (importance scores on input tokens or spans) and concept-based explanations (higher-level features learned from internal activations) — into one library with a shared API for classification and text generation models. Its distinguishing feature is an end-to-end concept pipeline that runs from activation extraction through concept learning, interpretation, and scoring, which the authors state is uncommon in existing libraries.

Key Contributions

1. A attribution module for HuggingFace models supporting classification and text generation, with 10 attribution methods, 2 evaluation metrics, and control over attribution granularity.
2. A concepts module providing an end-to-end concept-based pipeline: it wraps nnsight to split HuggingFace language models, offers 15 concept-learning options (mostly via overcomplete), 3 concept-interpretation methods, concept-importance estimation, and 7 metrics.
3. A unified API and supporting ecosystem for both classification and generation, including visualization tools, metrics, tutorials, documentation, a demonstration website, and a video.
4. Validation and practical grounding, including unit tests for all functions, tests across more than 15 model architectures, sanity checks on manually constructed models, and reported use of Interpreto in an industrial setting by Mussot et al. (2026).

Main Findings

• Two explanation families, one library: Interpreto exposes both attribution methods (input-element importance) and concept-based activation analysis, which the paper describes as a component of mechanistic interpretability, under the same API.
• Attribution coverage: 10 methods are available — perturbation-based (4): KernelSHAP, LIME, Occlusion, and Sobol; gradient-based (6): GradientSHAP, Integrated Gradients, Saliency, SmoothGrad, SquareGrad, and VarGrad — plus 2 faithfulness metrics, Insertion and Deletion.
• Concept pipeline coverage: The concept space can be defined via neurons as concepts, dictionary learning (KMeans, PCA, SVD, ICA, NMF, Semi-NMF, Convex NMF), or sparse autoencoders (Vanilla SAE, Jump-ReLU SAE, Top-k SAE, Batch Top-k, Matching Pursuit SAE).
• Interpretation and scoring: Concepts can be interpreted with top-k vocabulary tokens, top-k activating examples/words/n-grams (MaxAct), or LLM-based labeling; importance can be estimated with concept-to-output gradients or concept × gradients. Metrics cover concept-space faithfulness via MSE and FID, sparsity, Stability, and general usefulness via ConSim.
• Granularity and defaults: Attributions can be produced at token, word, or sentence level, with word-level as the default; users can also enable the input × gradient product (on by default for gradient methods) and choose the output space (logits, softmax, or log softmax), with logits as the default.
• Task-dependent recommendations: For concept extraction, the paper recommends the [CLS] token for classification and tokens for generation, with interpretation granularity matched to activation granularity.
• Demonstration gallery: The website covers 6 models — 3 classifiers (DistilBERT/IMDB, BERT/emotion, RoBERTa/AG-News) and 3 generators (GPT-2, Qwen3-0.6B, Llama 3.1 8B).
• Computation costs: Attributions typically require 10–100 forward passes or 5–20 gradient computations (seconds). Concept pipelines are dominated by activation extraction and concept importance scoring: small runs take minutes on an RTX 3080, while large SAEs can take hours. The worked generation example (Qwen3-0.6B, 100 AG-News samples, Semi-NMF with 20 concepts, GPT-4.1-nano labeling) runs in under 3 minutes on an RTX 3080 (10GB).
• Classification example output: In the AG-News DistilBERT example, class-wise concept importances include 0.088 for a "World" concept, 0.112 for a "Sports" concept, 0.092 for a "Business" concept, and 0.121 for a "Sci/Tech" concept.
• Differentiation from prior tools: Compared with attribution libraries (Captum, Ferret, Inseq, SHAP), Interpreto is described as supporting both classification and text generation, providing evaluation metrics, and exposing granularity controls; compared with mechanistic interpretability libraries, its main differentiator is integrating model splitting, concept learning, interpretation, and scoring into a single documented package.
• Stated limitations: LLM-based concept labels are highly sensitive to prompt design and can be vague, redundant, overly precise, or lengthy; SAE-based methods can be computationally expensive; explanations are subject to human biases; and the authors note there is no single method that governs all cases.

Methodology in Plain English

The authors built a software library rather than proposing a new explanation algorithm. They organized interpretability tooling around two workflows.

The attribution workflow has three steps: instantiate an explainer with a HuggingFace model and tokenizer, compute explanations for given inputs (optionally specifying targets such as class indices or selected output tokens), and visualize the results. Internally, the pipeline follows three stages — constructing perturbed input variants, running forward and/or backward passes, and aggregating intermediate scores into element-wise importance scores — so that adding a custom method means writing only the method-specific core and inheriting the rest.

The concept workflow also has four steps: split the model into a feature extractor and a predictor and collect an activation dataset; train a concept model on those activations to define a concept space; assign human-meaningful labels to each concept; and quantify each concept's contribution to predictions. Each step is modular, so users can mix and match split points, concept models, and interpretation methods.

To validate correctness, the authors provide unit tests for all functions, test more than 15 model architectures, and run sanity checks on manually constructed models in both tasks. They also demonstrate the library through a website gallery of precomputed explanations with copyable runnable code snippets, and report a user scenario in which a practitioner debugging an emotion recognition model uses KernelSHAP attributions and then class-wise concept explanations to uncover an inconsistent class-index mapping across data sources.

Why This Matters

Interpreto lowers the practical barrier to applying attribution and concept-based interpretability to HuggingFace language models, from early BERT variants to LLMs. The authors argue that easier access can support model auditing, debugging, and documentation, including identifying biases and recurring failure modes. They also caution that interpretability outputs can be misread as faithful causal explanations, since results depend on method choice, hyperparameters, and presentation, and recommend treating explanations as diagnostic evidence rather than ground truth.

Real-world applications implied by the paper:

• Model debugging: tracing incorrect predictions, as in the emotion-recognition scenario where concept analysis revealed a class-index mapping inconsistency.
• Bias detection and auditing: using class-wise global concept analyses and attributions to inspect what drives predictions across categories.
• Safety and documentation: supporting model cards, audits, and failure-mode identification for deployed language models.
• Industrial evaluation: the paper cites Mussot et al. (2026) using Interpreto in an industrial setting, supporting its accessibility and practical usefulness.

Industry relevance stems from the library's focus on off-the-shelf HuggingFace models, its MIT license, its pip and uv installability, and its use of external LLM APIs (for example, GPT-4.1-nano via OpenAI) as an optional labeling path — making it applicable to teams already working with standard Transformer models and cloud tooling.

Future Directions

• Supervised concepts: adding CAVs and probes as an alternative to unsupervised discovery, which the paper lists as planned for later within the concepts module.
• More methods and metrics: adding attribution methods such as RISE and RFEM, and metrics such as AOPC comprehensiveness and sufficiency, along with interpretation-specific metrics (for example, Detection, Fuzzing, Clarity, and Purity).
• Linking the two modules: enabling input-to-concepts attributions, so attribution and concept explanations can be composed.
• Broader model coverage: extending the library to ViT with the longer-term aim of multi-modal Transformer support.
• Open questions the paper raises: there is no single method that governs all interpretability needs, so users must compare methods via metrics; the meaning of attribution scores depends on the method, so similar scores from different methods can mean different things; and it remains unclear whether a non-interpretable concept stems from a bad model, a bad concept space, or a bad interpretation. Circuit-level mechanistic interpretability methods, data attribution, and feature visualization are explicitly declared out of scope.

Target Audience

This paper is most useful to NLP practitioners and ML engineers who need to explain HuggingFace model behavior in classification or generation settings; to interpretability researchers who want a single package covering model splitting, concept learning, interpretation, and scoring; to teams working on model auditing, bias detection, and safety documentation; and to instructors or students looking for documented tutorials and runnable examples. Readers seeking vision-specific tooling, circuit-level mechanistic interpretability, data attribution, or feature visualization are directed elsewhere by the authors.

Authors’ abstract

Interpreto is an open-source Python library for interpreting HuggingFace language models, from early BERT variants to LLMs. It provides two complementary families of methods: attribution methods and concept-based explanations. The library bridges recent research and practical tooling by exposing explanation workflows through a unified API for both classification and text generation. A key differentiator is its end-to-end concept-based pipeline (from activation extraction to concept learning, interpretation, and scoring), which goes beyond feature-level attributions and is uncommon in existing libraries. See GitHub: https://github.com/FOR-sight-ai/interpreto and the demo website: https://for-sight-ai.github.io/interpreto-demo/.