FACE: Faithful Automatic Concept Extraction

AI summary

Overview

• Research area: Explainable AI / interpretable computer vision — specifically concept-based explanation methods for deep neural network classifiers.
• Technical level: Intermediate (requires familiarity with neural network classifiers, matrix factorization, and softmax/KL divergence, though the paper explains its own terminology).
• Scope: The paper proposes FACE, an NMF-based concept extraction framework regularized with KL divergence to make learned concepts align with a classifier's actual predictions, and evaluates it on ImageNet, COCO, and CelebA with ResNet-34 and MobileNetV2.

What This Paper Is About

Deep classifiers make decisions using internal features that are often hard to relate to human-understandable ideas. Concept-based explanation methods try to bridge this gap by automatically discovering semantic "concepts" (like fur, ears, or texture) from a model's internal activations. The problem this paper targets is that existing automatic methods reconstruct activations without checking whether the reconstructed representation still drives the model to the same decision — so concepts can look interpretable while misrepresenting what the model actually relies on. FACE's goal is to produce concepts that are both interpretable and faithful to the model's true predictive behavior.

Key Contributions

1. A KL-regularized NMF objective. FACE adds a Kullback-Leibler divergence term between the classifier head's output on the original activations and on the reconstructed activations (UW^T), so concept learning is supervised by the model's own predictions rather than by reconstruction alone.
2. Theoretical justification for faithfulness. The authors show via a first-order Taylor argument and a concrete temperature-scaled softmax example that small reconstruction error does not guarantee small prediction deviation, and use Pinsker's inequality to bound total variation distance between predictive distributions by sqrt(2ε), where ε is the empirical KL value.
3. A practical optimization scheme. Because the KL term relies on the downstream classifier, standard multiplicative updates no longer apply; the authors use projected gradient descent with alternating updates of U and W, initialized with Non-negative Double Singular Value Decomposition (NNDSVD).
4. Systematic evaluation. Experiments across ImageNet, COCO, and CelebA with ResNet-34 and MobileNetV2, comparing FACE against ICE and CRAFT on factorization quality, faithfulness, and sparsity, plus ablations on regularization strength and decomposition rank.

Main Findings

• FACE achieves the lowest KL divergence in all settings. Across ResNet-34 and MobileNetV2 on ImageNet, COCO, and CelebA, FACE had the lowest D_KL between predictions on original and reconstructed activations — for example, 0.220 ± 0.000 (ResNet-34, ImageNet) versus 0.240 ± 0.003 for CRAFT and 0.359 ± 0.004 for ICE, and 0.021 ± 0.001 (ResNet-34, CelebA) versus 0.110 ± 0.066 for CRAFT and 0.212 ± 0.000 for ICE.
• FACE trades reconstruction error for predictive fidelity. FACE has higher MSE than ICE in every reported cell and higher MSE than CRAFT on ImageNet and COCO — e.g., 0.497 ± 0.002 (ResNet-34, ImageNet) versus 0.296 ± 0.001 for ICE. The authors state this is expected because the objective balances reconstruction with KL regularization.
• Reconstruction alone can destroy predictions. Starting from images classified with 100% accuracy by the original model, FACE retained 100% top-1 accuracy on reconstructed activations, while CRAFT and ICE showed notable drops, including only 40% accuracy for the "Train" class with CRAFT.
• Faithfulness and sparsity metrics favor FACE. FACE scored highest on Concept Insertion, Concept Deletion, and Gini-index sparsity across all models and datasets; on ResNet-34/ImageNet it reached C-Ins 0.969 ± 0.010, C-Del 0.891 ± 0.011, and C-Gini 0.895 ± 0.001, versus CRAFT's 0.932 ± 0.001, 0.752 ± 0.031, and 0.835 ± 0.031.
• CelebA shows a narrower gap. The authors attribute the smaller margin on CelebA (4 classes) to lower class complexity, where inserting a few key concepts suffices to recover accuracy while removal may not dramatically disrupt predictions.
• Regularization strength is dataset-dependent. On ImageNet and COCO, a small KL penalty (e.g., λ = 10⁻⁵) improved faithfulness, but λ ≥ 10³ caused a sharp drop in faithfulness and, on ImageNet, harmed classification accuracy. On CelebA, faithfulness kept improving up to around λ = 10⁵ with no adverse effect on accuracy — which the authors relate to ImageNet's 1000 classes and COCO's 200 classes being harder to align than CelebA's 4 classes.
• Rank improves faithfulness with diminishing returns. Sweeping r ∈ {5, 10, …, 50} showed sharp improvement between k = 5 and k = 25, after which gains plateaued; sparsity continued to improve with higher rank. The paper uses r = 25, following prior work.
• Concepts may diverge from human intuition. In the Gray hair class, CRAFT highlighted hair and ICE focused on the forehead, while FACE indicated the model relies on facial features — which the authors present as evidence that faithfulness, not visual plausibility, should be the target.

Methodology in Plain English

The authors take a standard pipeline: run images through a classifier's encoder to get penultimate-layer activations, then factorize those activations with NMF into a dictionary of concept vectors and a coefficient matrix, and score concept importance using Sobol indices. Their change is to add a second term to the factorization objective. Instead of only minimizing how far the reconstructed activations drift from the originals (Frobenius norm), they also minimize the KL divergence between the softmax distribution the classifier produces from the original activations and the one it produces from the reconstructed activations. This means the factorization is penalized whenever its low-rank approximation would change the model's output distribution, not just when it would change the activations.

Because this KL term involves the classifier, the classic multiplicative update rules do not apply, so the authors optimize with projected gradient descent, alternating between the two factor matrices and projecting back to non-negative values, starting from an NNDSVD initialization. They evaluate with perturbation-based metrics borrowed from the feature attribution literature — Concept Deletion (accuracy drop as important concepts are removed) and Concept Insertion (accuracy recovery as they are re-added) — plus the Gini index over concept importance scores to measure sparsity, and MSE and KL divergence to measure factorization quality. Results are averaged over five runs on correctly classified samples: 10,000 samples from 10 ImageNet classes, 5,000 from 5 COCO classes, and 4,000 from 4 CelebA attributes.

Why This Matters

Concept-based explanations are only useful if they describe what a model actually does; otherwise they mislead the very people trying to audit or debug the model. FACE reframes automatic concept discovery as a prediction-consistency problem rather than a reconstruction problem, and supplies a bound linking the KL penalty to prediction deviation. That makes it directly relevant to anyone using explanations for debugging, bias diagnosis, or safety review.

Real-world applications:

• Model debugging and failure analysis in deployed vision systems, where explanations that track the true decision process help engineers locate spurious or shortcut features.
• Regulatory and compliance review for high-stakes classifiers, where reviewers need evidence that a stated rationale matches model behavior.
• Dataset and bias auditing, since class-level concepts that faithfully capture model reliance can expose attributes the model leans on unintentionally.
• Automotive and safety-critical perception (the work was supported by Toyota InfoTech Labs through Unrestricted Research Funds), where understanding why a perception model made a decision is a prerequisite for validation.

Industry relevance: the method is explicitly described as lightweight — dominated by a single small matrix product and a linear head — making it tractable on low-resource hardware, which matters for teams that cannot afford large-scale retraining or heavy attribution computation. Code is released at https://github.com/dipkamal/FACE, and the authors credit the CRAFT codebase as a foundation.

Future Directions

• Instance-level explanations. FACE is currently class-specific and global; extending it to per-instance interpretability is listed as a limitation that affects explanation granularity.
• Human-centered evaluation. The authors note that because the KL loss changes the representation objective, its effect on human interpretability is unexplored, and they call user studies important future work.
• Hyperparameter transfer. Regularization strength requires tuning per dataset, and the ablations show optima ranging from 10⁻⁵ to 10⁵ depending on class complexity; a principled way to set λ remains open.
• Beyond CNNs. The authors state FACE suits CNN architectures and that applying it directly to transformers like ViT is non-trivial, which they frame as promising future work.

Target Audience

Researchers and practitioners in explainable AI and interpretable machine learning, particularly those working on concept-based explanations, post-hoc attribution, and model auditing. It will also be useful to applied ML engineers in safety-critical domains (automotive perception, healthcare imaging, biometrics) who need explanations that reflect actual model behavior, and to graduate students already comfortable with NMF, softmax classifiers, and perturbation-based faithfulness metrics. Readers without that background can follow the intuition, but the theoretical section assumes familiarity with Jacobians and Pinsker's inequality.

Authors’ abstract

Interpreting deep neural networks through concept-based explanations offers a bridge between low-level features and high-level human-understandable semantics. However, existing automatic concept discovery methods often fail to align these extracted concepts with the model's true decision-making process, thereby compromising explanation faithfulness. In this work, we propose FACE (Faithful Automatic Concept Extraction), a novel framework that augments Non-negative Matrix Factorization (NMF) with a Kullback-Leibler (KL) divergence regularization term to ensure alignment between the model's original and concept-based predictions. Unlike prior methods that operate solely on encoder activations, FACE incorporates classifier supervision during concept learning, enforcing predictive consistency and enabling faithful explanations. We provide theoretical guarantees showing that minimizing the KL divergence bounds the deviation in predictive distributions, thereby promoting faithful local linearity in the learned concept space. Systematic evaluations on ImageNet, COCO, and CelebA datasets demonstrate that FACE outperforms existing methods across faithfulness and sparsity metrics.