FaCT: Faithful Concept Traces for Explaining Neural Network Decisions

AI summary

Overview

Research area: Interpretable machine learning — specifically concept-based explanation methods for deep neural networks, combining B-cos networks, sparse autoencoders, and vision foundation models.

Technical level: Intermediate to Advanced. The paper is accessible in its motivation and results, but the core derivations rely on dynamic-linear transforms, B-cos attributions, and sparse dictionary learning.

Scope: The paper introduces FaCT, a model architecture whose decisions are made through interpretable concepts, so that concept contributions to logits and concept visualizations at the input can be computed exactly rather than approximated, plus a new foundation-model-based metric (C²-score) for evaluating concept consistency.

What This Paper Is About

Most existing concept-based explanation methods are post hoc: they decompose a model's activations after training, then rely on approximate attribution techniques to estimate how much each concept mattered to the output and to visualize where in the input the concept fired. Those approximations can be unfaithful to the model's actual decision-making, and the concepts themselves are often constrained by assumptions such as being class-specific, limited to small object parts, or restricted to a predefined human-annotated set. FaCT addresses both problems by building a model whose forward pass explicitly routes through a shared, class-agnostic concept basis, so that explanations are faithful by design rather than estimated.

Key Contributions

1. A model with an inherent concept basis used in the forward pass. Concepts are shared across classes, exist across depth, and generalize across architectures (CNNs and ViTs).
2. Faithful measurement of each concept's contribution to the output. The paper shows quantitatively that this outperforms existing approximate concept-importance measures.
3. Faithful visualization of every concept at the input level. Through a user study with control groups, the authors show how these visualizations affect interpretability.
4. A novel concept-consistency evaluation metric (C²-score) that uses foundation-model features to evaluate concepts across images, applicable to both shared concept sets (theirs) and class-specific sets (prior work).

Main Findings

• Logits decompose exactly into concept contributions. Because the model's later layers are B-cos transforms and the sparse autoencoder is bias-free, every logit equals a sum of concept contributions, and every concept activation equals a sum of input-pixel contributions. For example, the yellow-color concept contributes 4.3% of the School Bus logit.
• Competitive ImageNet performance with a sparse concept bottleneck. FaCT maintains competitive accuracy compared to original B-cos networks despite being trained with a sparse concept representation, even at intermediate layers, with a largest drop of less than 3% across settings.
• Large gains in concept consistency. For DenseNet at Block 3/4, the C²-score rises from 0.11 to 0.39; compared to B-cos channels overall, the C²-score rises from 0.09 to 0.37.
• Concepts vary widely in spatial extent. Measuring the number of highest-attribution pixels needed to cover 80% of the total attribution, the authors find a diversity of concept sizes across layers and architectures — e.g., a small late-layer "Bike Helmet" concept for DenseNet and a large early-layer "Wooden Texture" concept for ViT — in contrast to prior work that assumes fixed sizes.
• Concepts are more interpretable to users. In a study with 38 volunteer participants rating concepts on a 5-point scale, participants found FaCT's concepts far more interpretable than B-cos channels, for both early- and late-layer concepts.
• Faithful input attributions improve interpretability, especially early on. Providing attributions led to an average increase of about 0.5/5 for earlier-layer concepts. The early-layer concepts with the largest score increase were low-level patterns such as background (+3.0/5) and up-facing curves (+2.5/5); for the "curve" concept the explanation increased the score by Δ = 2.5 (/5).
• Concept contributions are sharper under deletion. Using the concept-deletion metric, FaCT's contributions (Eq. 9) produce significantly steeper drops in both the top-1 logit and overall accuracy than attribution methods such as Saliency or Sobol indices used in prior work.
• Shared concepts support misclassification analysis. For a Basketball image misclassified as Volleyball, concepts A–F were labeled via CLIP-Dissect as 'balls', 'jerseys', 'rugby', 'flexible', 'volleyball', and 'basketball'; the mutually contributing concepts (confounders like ball, jersey, person with shirt, limbs) explain the confusion.
• Concepts can be named with existing tools. Applying CLIP-Dissect to the SAE latents, using common English words, produces plausible text labels for the concepts.

Methodology in Plain English

The authors start from B-cos networks, a class of models already known to produce faithful, model-inherent attributions because their layers can be rewritten as a dynamic-linear combination of the input. They take ImageNet-pretrained B-cos checkpoints (ResNet-50, DenseNet-121, and ViT_c-S) and insert sparse autoencoders at chosen intermediate layers. Each SAE learns a dictionary of concepts and encodes each spatial patch of features into a sparse set of concept activations. Crucially, the SAE encoder has no biases, so the concept activations are a strictly linear function of the features, and the decoder's output — reconstructed purely from concepts — is what the remaining network consumes to produce the logits. This makes the concepts part of the forward pass rather than an afterthought.

Training details: SAEs are trained for reconstruction on ImageNet's training set, with 50 samples per class held out as a validation set. The authors sweep TopK ∈ (8, 16, 32) and K ∈ (8192, 16384) total concepts. Each concept is visualized by its top-activating test images together with its input attribution and image category.

For evaluation, the authors design C²-score. They feed images through a fixed feature extractor — DINOv2, upsampled with LoftUp and centered by the dataset mean — and compute, for each concept on each image, an embedding that weights the extractor's features by the concept's attribution map. Consistency is the weighted cosine similarity of these embeddings across pairs of images, and the C²-score is the average consistency minus the consistency of a random concept with a random attribution map. This subtraction makes class-specific and shared concept sets comparable.

Why This Matters

Impact on research. The work challenges a prevailing pattern in concept-based interpretability — extract concepts post hoc, then approximate their importance and localization. By making concepts part of the forward pass in a way that provably sums to the logit, it shifts the field toward architectures where faithfulness is a design property rather than a post-hoc claim. The C²-score also offers an evaluation route that does not depend on human part annotations, which are limited to a small set of classes and assume every concept maps to an annotated part.

Real-world applications:

• Healthcare and other sensitive domains where understanding which high-level concepts drove a decision matters for accountability.
• Dataset and model auditing: a shared, class-agnostic concept basis makes it easier to spot confounding concepts that cause systematic confusion between classes.
• Human-in-the-loop review of model errors, where reviewers can see which concepts were shared between the predicted and true class.
• Model debugging and steering in settings where concept-level intervention is preferable to pixel-level attribution.

Industry relevance. The method is built on top of existing pretrained checkpoints and adds sparse autoencoders at selected layers, so it can be layered onto standard training pipelines. Competitive ImageNet accuracy with a largest drop under 3% means the interpretability does not come at a prohibitive accuracy cost. The paper also reports code availability at github.com/m-parchami/FaCT.

Future Directions

• Textual alignment of concepts. The authors note that some users may want concepts aligned with text labels, and show CLIP-Dissect can be applied to SAE latents; more robust neuron-labeling for these shared concepts remains open.
• Generalization beyond ImageNet. The paper states that generalization of FaCT to other datasets is demonstrated in Appendix Sec. K (results on CUB), but broader cross-domain validation is a natural next step.
• Understanding universally contributing concepts. The authors find that a small set of concepts always contributes to every decision and causes a sharp accuracy drop when deleted at Block (2/4). Strategies to handle or isolate such concepts deserve more study.
• Limitations and societal impact. The paper flags dedicated sections (Appendix Sec. L and Sec. M) on limitations and societal impact, indicating these are acknowledged but not fully resolved within the main text.

Target Audience

This paper is most valuable to interpretability and explainable-AI researchers, particularly those working on concept-based explanations, part-prototype networks, and concept bottleneck models. It also suits practitioners who need auditable model decisions in high-stakes domains, and engineers familiar with sparse autoencoders or B-cos networks who want to integrate faithful concept decomposition into existing vision pipelines. Readers without a background in attribution methods will need to work through the derivations in Section 3 to follow the faithfulness arguments.

Authors’ abstract

Deep networks have shown remarkable performance across a wide range of tasks, yet getting a global concept-level understanding of how they function remains a key challenge. Many post-hoc concept-based approaches have been introduced to understand their workings, yet they are not always faithful to the model. Further, they make restrictive assumptions on the concepts a model learns, such as class-specificity, small spatial extent, or alignment to human expectations. In this work, we put emphasis on the faithfulness of such concept-based explanations and propose a new model with model-inherent mechanistic concept-explanations. Our concepts are shared across classes and, from any layer, their contribution to the logit and their input-visualization can be faithfully traced. We also leverage foundation models to propose a new concept-consistency metric, C$^2$-Score, that can be used to evaluate concept-based methods. We show that, compared to prior work, our concepts are quantitatively more consistent and users find our concepts to be more interpretable, all while retaining competitive ImageNet performance.