From Black-box to Causal-box: Towards Building More Interpretable Models

AI summary

Overview

Research area: Interpretable machine learning and explainable AI (XAI), combined with structural causal models (SCMs) from causal inference.

Technical level: Advanced. The paper is written in the formal language of structural causal models, causal diagrams, and counterfactual queries, with definitions, propositions, and theorems. Readers need comfort with graphical causal reasoning.

One-sentence scope: The paper formalizes when a class of prediction models can consistently answer counterfactual "what if" questions from observational data, proves that blackbox and concept-based models generally cannot, and derives a complete graphical criterion for architectures that can.

What This Paper Is About

Deep learning models predict well but cannot explain their reasoning, and a promising route to explanation is counterfactual reasoning: asking how a prediction would change if a high-level feature (such as whether a person is smiling) were different. The authors ask whether such counterfactual questions can be answered in a consistent, trustworthy way from observational data alone, given a particular model architecture. They show that the two dominant architectures, blackbox predictors and concept-based predictors, fail in general, and they then build a framework for models that succeed by design.

Key Contributions

1. A formal definition of causal interpretability. The authors introduce the notion that a model class is causally interpretable with respect to a counterfactual query when every pair of models in that class that agrees on the observational distribution also agrees on the answer to that query (Definition 2). This draws an explicit connection between XAI and causal inference.

2. Impossibility results for existing architectures. They prove that a blackbox prediction model, which maps image pixels directly to labels, is never causally interpretable with respect to any counterfactual query over the observed features (Proposition 1). They also show, contrary to prior belief, that concept-based models, which predict from all observed features, are not causally interpretable in general.

3. A complete graphical criterion and a uniqueness result. They define generalized concept-based prediction (GCP) models, which predict from an arbitrary chosen subset T of features, and prove that such a model class is causally interpretable with respect to a query on features W if and only if T is contained in W together with the non-descendants of W (Theorem 1). They further prove that the maximal such feature set is unique and equals the intersection over queries of W_i union the non-descendants of W_i (Theorem 2).

4. A closed-form evaluation formula and a trade-off theorem. They give a two-step formula for computing the counterfactual quantity from a classifier over T and a feature extractor over X (Theorem 3), and they characterize a fundamental trade-off: adding features to the predictor reduces the set of counterfactual queries the model can answer, and demanding more queries shrinks the maximal admissible feature set (Theorem 4).

Main Findings

• Blackbox models cannot answer counterfactual queries consistently. For any latent causal graph, the class of blackbox prediction models is not causally interpretable with respect to any query Q(W) for any W contained in the observed features (Proposition 1). The paper illustrates this with a facial attractiveness example in which two blackbox models agree perfectly on the observational distribution yet return opposite answers (0 and 1) to the question "would the prediction be attractive had the person not smiled?"

• Concept-based models are not safe either. Using the same attractiveness-style setup, two concept-based models that agree on the observational distribution give different counterfactual answers for the same query, specifically 0.3 versus 0.5 (Example 4). This contradicts the intuition that predicting from features rather than pixels removes the problem.

• The fix is a graphical condition on which features the model may use. A GCP model using feature set T is causally interpretable with respect to Q(W) exactly when T is a subset of W union the non-descendants of W (Theorem 1). Intuitively, a model cannot answer a counterfactual about a feature if it bases its prediction on any descendant of that feature.

• The best admissible architecture is unique. Among all feature sets that retain interpretability for a collection of queries, there is exactly one maximal set, given by the intersection over queries of W_i union ND(W_i) (Theorem 2). It contains every interpretable choice as a subset, so there is no ambiguity about which architecture maximizes predictive expressiveness.

• Only descendants matter in practice. Identifying the maximal admissible set requires knowledge only of the descendants of the target feature, not a full specification of the causal graph or modeling of unobserved confounders. The paper's example: for the query "what if the person had smiled?", only the descendants of smiling (such as cheekbone) need to be known.

• Counterfactuals have a simple two-step form when interpretability holds. When the model class is causally interpretable, the counterfactual probability equals a sum over the feature values of a classifier term P(Y-hat | w' intersected with T, t minus W) times a feature-extraction term P(t | x) (Theorem 3). This formula holds only for causally interpretable models.

• Interpretability and accuracy trade off directly. If one feature set is contained in another, the smaller set admits a superset of answerable counterfactual queries (Theorem 4, part i). Correspondingly, a larger collection of required queries yields a smaller maximal admissible feature set (Theorem 4, part ii).

• Experiments support the theory. The abstract and introduction state that experiments corroborate the theoretical findings. The specific datasets, benchmark names, and numerical experiment results are not reported in the provided content.

Methodology in Plain English

The authors start from a structural causal model, a standard causal-inference representation in which latent generative factors produce an image, and a classifier then produces a label from that image or from the factors. They extend this into what they call an augmented SCM, which layers the prediction step on top of the data-generating process, and they draw the resulting causal diagram.

They then define model classes by what the classifier is allowed to look at: pixels only (blackbox), all observed features (concept-based), or an arbitrary chosen subset of features (generalized concept-based). Interpretability is defined behaviorally rather than architecturally: a class is causally interpretable for a query if no two members of the class can agree on the observational distribution yet disagree on the counterfactual answer.

With these definitions in place, the analysis becomes purely graphical. The authors trace which features are descendants of the feature being intervened on and prove that interpretability holds exactly when the predictor uses no descendants. They then show that among all sets satisfying this condition, one is maximal, and they prove it is unique by expressing it as an intersection. Finally, they derive a closed-form expression for the counterfactual probability, which shows the query decomposes into a feature extractor and a classifier, and they prove monotonicity results that yield the accuracy-interpretability trade-off.

Why This Matters

Impact on research. The paper reframes interpretability as a property that can be proven about a model class rather than a heuristic added after training. It gives a precise reason why post-hoc explanation methods and concept-based models, including those that add causal structure without consistency guarantees, can silently give contradictory counterfactual answers. It also connects the XAI literature (referencing LIME, SHAP, Grad-CAM, sparse models, restricted final layers, and decision-tree-based models) to the structural causal model tradition.

Real-world applications (drawn from the domains the paper names):

• Healthcare: asking whether a diagnosis would have changed under a different treatment or presentation, where trusting a contradictory answer is unacceptable.
• Law and high-stakes decisions: justifying why an individual received a particular classification, such as asking whether a person would have been classified differently had their income been higher.
• Scientific discovery and medical imaging: reasoning about high-level, human-understandable features such as the presence of a tumor or a smile, rather than raw pixels.
• Fairness analysis: the paper notes that the query formulation is general and could cover quantities such as the natural direct effect used in fairness analysis.
• Facial attribute prediction: the running example throughout the paper, where users ask whether attractiveness would change if a person smiled.

Industry relevance. The design guidance is actionable and cheap: a team only needs to know the descendants of the feature being interrogated, not the full causal graph, to decide which features a model may consume. That makes the criterion practical for teams building concept-based or feature-based predictors in regulated settings where a model must defend its counterfactual behavior, and it clarifies the accuracy price of that guarantee.

Future Directions

• Extending beyond the query family studied. The paper includes at least one preliminary result for more general query types, and the practical evaluation and guarantees for those remain to be developed.
• Building the augmented causal models at scale. The theory assumes an augmented SCM over a generative process; how to obtain reliable feature annotations and the relevant descendant structure in large, messy real-world datasets is left open.
• Empirical benchmarking. The provided content says experiments corroborate the theory but does not report datasets or numbers; a fuller empirical comparison of generalized concept-based models against blackbox and concept-based baselines, including the accuracy cost of the trade-off, is a natural next step.
• Handling multiple or conflicting queries in practice. Since demanding more counterfactual queries shrinks the admissible feature set, understanding how to prioritize queries in real deployment, and how much predictive accuracy is sacrificed, remains an open design question.
• Relaxing the requirement of no descendants. Whether approximate or probabilistic notions of interpretability could allow limited use of descendant features without losing consistency is not addressed.

Target Audience

This paper is aimed at machine learning and causal inference researchers working on interpretability, explainability, and trustworthy AI, particularly those interested in formal guarantees rather than heuristic explanations. It will also be valuable to practitioners in high-stakes domains such as healthcare, law, and fairness auditing who need to justify counterfactual behavior of deployed models, and to graduate students with a background in graphical causal models who want a concrete bridge between SCM theory and modern predictive architectures.

Authors’ abstract

Understanding the predictions made by deep learning models remains a central challenge, especially in high-stakes applications. A promising approach is to equip models with the ability to answer counterfactual questions -- hypothetical ``what if?'' scenarios that go beyond the observed data and provide insight into a model reasoning. In this work, we introduce the notion of causal interpretability, which formalizes when counterfactual queries can be evaluated from a specific class of models and observational data. We analyze two common model classes -- blackbox and concept-based predictors -- and show that neither is causally interpretable in general. To address this gap, we develop a framework for building models that are causally interpretable by design. Specifically, we derive a complete graphical criterion that determines whether a given model architecture supports a given counterfactual query. This leads to a fundamental tradeoff between causal interpretability and predictive accuracy, which we characterize by identifying the unique maximal set of features that yields an interpretable model with maximal predictive expressiveness. Experiments corroborate the theoretical findings.