Flexible Concept Bottleneck Model

AI summary

Overview

Research area: Interpretable computer vision / explainable AI, specifically concept bottleneck models (CBMs) built on vision-language foundation models (VLMs).

Technical level: Advanced. The paper assumes familiarity with concept bottleneck models, hypernetworks, CLIP-style joint embedding spaces, sparsemax and gradient derivations, and standard vision benchmark protocols.

Scope: The paper proposes and empirically evaluates the Flexible Concept Bottleneck Model (FCBM), a framework that lets a CBM swap out its concept pool without full retraining, and tests it on five public benchmarks with two backbones.

What This Paper Is About

Concept bottleneck models make neural networks more interpretable by forcing them to predict human-understandable concepts before producing a final class prediction. Recent systems use large language models to generate the concept set and vision-language models such as CLIP to score images against those concepts, removing the need for expensive expert annotation. The catch is that these concept pools are fixed: adding a new concept, or switching to a newer foundation model, requires retraining the whole model. FCBM's goal is to make the concept layer dynamic, so concepts can be added, replaced entirely, or regenerated by a different LLM without paying the full retraining cost.

Key Contributions

1. A flexible and scalable CBM framework. FCBM replaces the fixed concept-to-label mapping of prior VLM-based CBMs with a design that adapts when new concepts are introduced, which the authors argue suits scenarios where different concept subsets are preferable or where users want to leverage a more advanced LLM.

2. Dynamic concept modeling via a hypernetwork. A hypernetwork generates the prediction weights from concept embeddings, so the mapping scales with the concept set rather than being baked into a fixed linear layer.

3. Sparse and interpretable concept selection. A modified sparsemax module with a learnable temperature parameter controls how many concepts are active per prediction, aiming to preserve interpretability and avoid concept redundancy.

4. Competitive accuracy plus concept-level generalization. The authors report accuracy comparable to state-of-the-art baselines with a similar number of effective concepts, and generalization to unseen concept pools with only a single epoch of fine-tuning.

Main Findings

• Accuracy at matched sparsity: With the average number of effective concepts (NEC) controlled at approximately 30, FCBM on ResNet50 scored 85.59 on CIFAR10, 64.77 on CIFAR100, 63.46 on CUB, 49.13 on Places365, and 66.34 on ImageNet. On ViT-L/14 it scored 97.21, 83.63, 80.52, 51.39, and 80.62 on the same five datasets. The paper states FCBM achieves the highest accuracy in most cases and ranks second in the remaining two, and that it outperforms all baselines on more than half of the benchmarks.

• The non-sparse standard model is the accuracy ceiling. The standard model scored highest on every dataset (e.g., 88.55 on CIFAR10 and 73.14 on ImageNet with ResNet50; 98.02 and 84.11 with ViT-L/14), which the authors attribute to it using the most informative concepts. Its sparse counterpart performs poorly, which the paper frames as evidence of the difficulty of maintaining accuracy under sparsity.

• Baseline comparison. On ResNet50, LF-CBM reached 86.16 on CIFAR10 and CF-CBM reached 64.23 on CUB, both ahead of FCBM on those two datasets; PCBM was the weakest of the sparse baselines (e.g., 76.43 on CIFAR10 with ResNet50) and the paper notes it was not originally designed for larger datasets.

• Zero-shot transfer to entirely new concept pools. Concepts regenerated by DeepSeek-V3 and GPT-4o were substituted for the trained concepts. With ResNet50 and DeepSeek-V3 concepts, FCBM scored 75.32 on CIFAR10, 35.57 on CIFAR100, 18.07 on CUB, 32.79 on Places365, and 29.23 on ImageNet; with GPT-4o concepts it scored 75.09, 34.54, 19.43, 32.27, and 26.69. On ViT-L/14 with DeepSeek-V3 concepts FCBM scored 94.89, 62.27, 23.06, 37.24, and 51.70.

• One epoch of fine-tuning closes much of the gap. FCBM adapts to an entirely new concept set with a single epoch of fine-tuning. For example, on ViT-L/14 CIFAR10 it moves from 97.21 with trained concepts to 94.89 zero-shot with DeepSeek-V3 concepts, and on Places365 from 51.39 to 37.24.

• Learnable temperature matters most for generalization. Removing sparsemax ("Hard", forcing the 30 most effective concepts) occasionally does well on trained concepts (e.g., 69.17 on ResNet50 CIFAR10) but has the weakest zero-shot behavior (e.g., 6.82 on CUB with DeepSeek-V3 concepts). Removing the learnable temperature ("FCBM temp.") yields sparsity that is typically very low, making effective concept selection difficult.

• Sparsity is stable across NEC settings. Testing NEC = 30, 50, 100, and the full concept set across all five datasets with both backbones showed accuracy increasing only slightly as NEC grows, with the most noticeable improvement on Places365, which the authors attribute to that dataset's higher prediction difficulty.

• Concept attribution example. On a Places365 image from the "campus" class, the top contributing concepts were similar across the trained and DeepSeek-V3 concept pools — for example, "a central quad or lawn" versus "a quad/lawn", "students" versus "Students", and "institution" versus "educational institution". With DeepSeek-V3 concepts the contributions were more dispersed, which the authors suggest may indicate incomplete alignment and potential classification errors.

Methodology in Plain English

FCBM keeps the two-stage structure of a CBM but changes what happens in the second stage.

Stage one is unchanged in spirit from prior VLM-based CBMs: a backbone extracts image features, and a concept predictor is trained to match CLIP-derived concept scores using the cosine-cubed loss from LF-CBM. Concepts themselves come from an LLM using the three prompt types introduced in that prior work — key features of a class, features commonly associated with a class, and a class's superclasses. The paper uses GPT-3 for the main experiments and later regenerates concepts with DeepSeek-V3 and GPT-4o.

Stage two is where the flexibility comes from. Instead of a fixed linear layer mapping concepts to classes, a hypernetwork takes each concept's text embedding and outputs that concept's contribution weights to each class. Because the hypernetwork maps from text-feature dimension to class dimension, its size does not depend on how many concepts exist, so the concept set can grow, shrink, or be swapped wholesale.

To make swapping work without retraining, the authors align distributions: at inference they normalize the new text features using the mean and standard deviation of the training text features, and similarly normalize the hypernetwork's outputs, so the final weights follow a distribution comparable to what was seen in training.

For interpretability, they apply a sparsemax operation with a learnable temperature to the generated weights. Sparsemax, unlike softmax, produces exact zeros, so only the most relevant concepts contribute. Higher temperature means fewer active concepts; lower temperature means more. Because the temperature is learned, the model balances accuracy and interpretability itself. The authors derive the gradient with respect to both the input vector and the temperature so the temperature can be optimized jointly with everything else. A temperature decay schedule (0.998 for CIFAR10, CIFAR100 and CUB; 0.92 for the large-scale datasets) lets the model start dense and sparsify over training, with decay stopping once NEC falls below roughly 30.

Evaluation uses prediction accuracy as the main metric and NEC to measure sparsity, with NEC held near 30 for fair comparison. Experiments were run on an AMD EPYC 7402 24-Core processor with an NVIDIA GeForce RTX 4090 and 512GB RAM, using the Adam optimizer at a learning rate of 0.001, up to 5,000 epochs for the small-scale datasets and 500 for the large-scale ones. Results are averaged over three seeds with standard deviations reported.

Why This Matters

The paper targets a practical failure mode of interpretable models: interpretability that cannot keep up with changing knowledge. If a concept layer is frozen, then any new biomarker, new object category, or upgraded vision-language foundation model invalidates it and forces an end-to-end retrain. FCBM argues that a hypernetwork plus sparse selection can decouple the concept set from the learned mapping, making the interpretable layer as updatable as the foundation models it depends on. Prior work such as OpenCBM supports adding or removing concepts at test time, but the paper notes it cannot handle complete replacement of the concept pool — which is exactly the scenario FCBM is built for.

Real-world applications:

• Medicine and clinical decision support. The introduction specifically cites newly discovered biomarkers as a case where the concept space must be extended; the conclusion proposes investigating whether novel biomarkers can be integrated without sacrificing interpretability or predictive performance.
• Domain knowledge that evolves over time. Any deployment where expert concepts are revised periodically — regulatory categories, product taxonomies, scientific nomenclature — can adopt a new concept set with one epoch of fine-tuning instead of a full retrain.
• Systems that want to upgrade their foundation model. Because the concept pool can be replaced entirely, a pipeline can move from one LLM's generated concepts to a newer LLM's concepts, or from one CLIP checkpoint to a stronger one, without rebuilding the classifier.
• Interactive human-in-the-loop tools. Sparse concept selection means each prediction comes with a short list of active concepts that a user can inspect or intervene on, which is the premise behind CBM-style human–AI interaction cited in the related work.

Industry relevance: Organizations deploying interpretable vision models face a recurring cost every time their concept vocabulary or foundation model changes. FCBM's one-epoch adaptation claim, if it holds on their data, converts that recurring retraining cost into a cheap fine-tuning step, which is the kind of operational argument that drives adoption of interpretability tooling in regulated industries.

Future Directions

• Evaluation in more complex, real-world scenarios. The conclusion explicitly lists this as future work, beyond the five benchmark datasets used here.
• Biomarker integration as a concrete test case. The authors pose the question of whether novel biomarkers can be folded into FCBM without losing interpretability or predictive performance.
• Caution around fine-grained and specialized domains. The paper's own results show weak zero-shot transfer on CUB, a fine-grained bird-species dataset (for example, 6.82 with hard truncation on ResNet50 and 18.07 for FCBM with DeepSeek-V3 concepts), and it advises users to be careful when applying LLM-generated concepts to fine-grained categories or highly specialized datasets.
• Understanding why concept contributions disperse under new pools. The Places365 case study shows more dispersed contributions with DeepSeek-V3 concepts, which the authors link to possible misalignment with the training concepts and potential classification errors — but the underlying mechanism is left open.

Target Audience

Researchers and practitioners working on interpretable machine learning, explainable computer vision, and concept-based models, particularly those who already follow the VLM-based CBM line of work (Post-hoc CBM, LF-CBM, CF-CBM, VLG-CBM, OpenCBM) and want to understand how to make a concept layer modifiable. It is also relevant to engineers building production systems where interpretable models must stay current with evolving domain vocabularies or with frequent foundation-model upgrades. Readers without background in CBMs, hypernetworks, or sparsemax will need to consult the cited prior work first, since the methodology section moves quickly through the mathematical formulation.

Authors’ abstract

Concept bottleneck models (CBMs) improve neural network interpretability by introducing an intermediate layer that maps human-understandable concepts to predictions. Recent work has explored the use of vision-language models (VLMs) to automate concept selection and annotation. However, existing VLM-based CBMs typically require full model retraining when new concepts are involved, which limits their adaptability and flexibility in real-world scenarios, especially considering the rapid evolution of vision-language foundation models. To address these issues, we propose Flexible Concept Bottleneck Model (FCBM), which supports dynamic concept adaptation, including complete replacement of the original concept set. Specifically, we design a hypernetwork that generates prediction weights based on concept embeddings, allowing seamless integration of new concepts without retraining the entire model. In addition, we introduce a modified sparsemax module with a learnable temperature parameter that dynamically selects the most relevant concepts, enabling the model to focus on the most informative features. Extensive experiments on five public benchmarks demonstrate that our method achieves accuracy comparable to state-of-the-art baselines with a similar number of effective concepts. Moreover, the model generalizes well to unseen concepts with just a single epoch of fine-tuning, demonstrating its strong adaptability and flexibility.