A Model of Causal Explanation on Neural Networks for Tabular Data

AI summary

Overview

• Research area: Explainable machine learning (XAI) with an explicit causal-inference component, applied to neural network predictors on tabular (row-and-column) data.
• Technical level: Advanced. The paper assumes familiarity with structural causal models (SCMs), directed acyclic graphs (DAGs), partial correlation, the causal Markov condition, and Shannon entropy.
• Scope: The paper introduces CENNET (Causal Explanations for Neural NETwork predictors), an architecture that combines neural networks with structural causal models to produce global and per-prediction explanations for classification tasks on tabular data, plus a new entropy-based index for scoring explanation quality.

What This Paper Is About

Most explanation methods for machine learning (such as LIME and SHAP) assume the reasons for a prediction are additive across variables, and they do not distinguish genuine causes from pseudo-correlations or indirect influences. This paper proposes CENNET, a method that analyzes the neurons in the layer just before a network's output — the "nearest neighbor latent unit" (NNLU) — and infers causal structure between those neurons and the input variables, so that explanations cite only directly causal variables or combinations of them. The paper also defines a new entropy-based explanation power index to compare single-variable and multi-variable reasons within the same framework.

Key Contributions

1. Removal of pseudo-correlations and indirect influences. The method is designed to eliminate pseudo-correlated and indirect variables that existing methods struggle to filter out, so that only direct causes are presented as reasons — reducing the risk that users absorb false causal knowledge about a model's behavior.

2. An information-theoretic guarantee. The paper proves (Theorem 1) that the entropy of a variable conditioned on its characteristic correlated variables (CCVs) achieves the lower bound of conditional entropy given any larger variable set that contains the CCVs, provided the variable is not an ancestor of that set. This gives the proposed variable selection a principled footing in information theory.

3. Handling of non-additive, combinatorial reasons. In classification tasks, the method supplies explanations for cases where multiple factors must combine to determine an individual prediction, cases that additive methods cannot express.

4. A new index for comparing single and combined reasons. The paper introduces entropy-based explanation powers (EEP), extended mutual information (EMI), positive explanation power (PEP), negative explanation power (NEP), and total explanation power (TEP), allowing single-variable reasons and multi-variable reasons to be scored and ranked on the same scale.

Main Findings

• Synthetic non-linear additive data: CENNET performed almost the same as LIME, Stabilized-LIME (S-LIME), and SHAP on the Non-Linear Additive dataset; ACV (Active Coalition of Variables) was the worst performer.

• Synthetic non-linear non-additive data: CENNET outperformed all existing methods in the Non-Linear Non-Additive experiment, even though the authors used a simple discretization scheme. This is the setting where explanations require pairs of interacting variables.

• Synthetic categorical data: CENNET outperformed all existing methods in the Category experiment, both for top-1 and top-5 ranking of important variable sets.

• Top-1 parity with SHAP on interacting pairs: In the Non-Linear Non-Additive experiment, CENNET was statistically at the same level as SHAP for top-1 ranking of important variable pairs, with no significant difference in a McNemar test at p = 0.324. CENNET's share of important variable pairs within the top-5 was higher than the other existing methods.

• Quasi-real data: CENNET outperformed the existing methods in seven out of ten experimental settings across the Alarm, Hailfinder, Insurance, and Carpo datasets, indicating it presents more direct causal variables as explanations.

• Selected quasi-real results (average rank of direct causal variables, lower is better, ± standard deviation):

  • Alarm (CATECHOL): CENNET 24.75 ± 30.08, versus SHAP 96.74 ± 96.11, LIME 103.18 ± 69.98, S-LIME 102.12 ± 68.05, ACV 118.11 ± 87.88.
  • Hailfinder (ScenRelAMCIN): CENNET 1.43 ± 0.94, versus SHAP 7.72 ± 3.76, ACV 7.96 ± 5.67, LIME 9.06 ± 4.49, S-LIME 9.25 ± 4.52.
  • Insurance (Antilock): CENNET 8.63 ± 6.99, versus SHAP 15.89 ± 13.96, ACV 31.86 ± 17.97, LIME 41.22 ± 12.61, S-LIME 41.15 ± 12.52.
  • Insurance (Vehicleyear): CENNET 29.14 ± 5.95, versus ACV 43.74 ± 29.26, SHAP 70.72 ± 16.17, S-LIME 78.36 ± 18.80, LIME 79.21 ± 19.65.
  • Carpo (N42): CENNET 86.23 ± 29.11, with ACV best at 81.69 ± 42.58 and SHAP at 95.36 ± 30.02.
  • Alarm (HISTORY): LIME and S-LIME tied at 1.00 ± 0.00, SHAP 1.64 ± 0.85, ACV 3.26 ± 1.71, CENNET 3.79 ± 0.84.

• Why NNLU analysis is needed: In preliminary experiments where CCVs were inferred directly for the final output variable, only a few variables were selected, making it hard to give diverse causal explanations for individual predictions. Specifically, X2, designed as a direct causal variable for the prediction variable in the Non-Linear Additive dataset, was not selected as a CCV for that output, and X1 was not selected as a CCV for the Category dataset output.

• Computational cost: The local explanation cost is O(lpt), where l is the number of neurons in the NNLU, p is the number of combinations of CCVs, and t is the number of samples to be explained. The global analysis is the cost of estimating the causal graphs, multiplied by the number of NNLU neurons, but the neuron-level work can be fully parallelized.

• Not reported: The paper does not report the predictive accuracy of the trained MLPs, runtime wall-clock measurements, regression-task results, or results on genuinely real-world datasets with known causal structure.

Methodology in Plain English

The method rests on the idea that correlation is not causation, and that explanations should name variables that directly influence a prediction rather than variables that merely co-move with it.

1. Structural causal models as the backbone. The researchers represent causal relations among variables as a directed acyclic graph, where an edge means one variable directly causes another. The graph is discovered from observational data using conditional independence tests — in practice a modified version of the PC algorithm — and gives the paper's operational definition of causality.

2. Characteristic correlated variables (CCVs). A variable X is a CCV of Y if no set of other variables can make their partial correlation zero, and X comes before Y in causal order. Intuitively, a CCV is a variable whose association with the target cannot be explained away by anything else in the data. The paper proves that conditioning on the CCVs of a variable achieves the minimum possible conditional entropy among all admissible conditioning sets.

3. Analyzing the layer before the output. Instead of attributing reasons straight to the final prediction, the method examines the "nearest neighbor latent unit" — the neurons immediately preceding the output layer. For each such neuron, the method infers a causal model over the input variables plus that neuron, and extracts the input variables that are CCVs for it. Because the network's output is essentially a weighted combination of these neurons, this captures how the network built its features while allowing more contributing variables to be found than a direct-to-output analysis would yield.

4. Turning causal structure into a per-prediction score. For an individual prediction, the method discretizes the relevant variables, computes an entropy-based explanation power (EEP) for each configuration of CCVs — essentially the log ratio of the conditional probability of the target given the configuration to its marginal probability — and multiplies it by the neuron's activation and its weight to the output layer. Summing over neurons with non-negative weights gives positive explanation power (PEP); summing over neurons with negative weights gives negative explanation power (NEP); their sum is total explanation power (TEP). Candidate reasons are then ranked by TEP.

5. Experimental design. The method was tested on three synthetic datasets (Non-Linear Additive, Non-Linear Non-Additive, Category) built with reference to the approach of Chen et al., and on four quasi-real datasets generated by sampling from expert knowledge models (Alarm, Hailfinder, Insurance, Carpo). Each dataset contained 10,000 examples split across training, validation, and test. Multi-layer perceptrons with two hidden layers were trained, with 5 neurons in the NNLU and 16 in the other hidden layer. Numerical variables were split into three equal-frequency categories for CENNET's causal analysis. Every method was scored by the average rank it assigned to the known important variables across test samples; smaller ranks are better.

Why This Matters

Impact on research. The paper connects two lines of work that are usually separate: causal discovery from observational data, and post-hoc explanation of neural networks. It argues that explanation quality should be evaluated against known causal ground truth rather than only against model behavior, and it offers entropy-based indices (EEP, EMI, PEP, NEP, TEP) as a shared currency for comparing single-variable and combinatorial reasons. It also pushes back on the additivity and linear-approximation assumptions that underlie LIME and SHAP, following concerns raised by Rubin and others that such approximations can distort what a model learned.

Real-world applications (as motivated by the paper):

• Medicine and epidemiology. Diagnoses that only arise when several factors combine — the paper's example of a disease requiring a given gender, an underlying condition, and elevated physical parameters together — could be explained as a combined cause rather than as three weak individual signals.

• Marketing. Preventing teams from accumulating incorrect beliefs about which factors cause customer outcomes, since explanations built on pseudo-correlated variables can be mistaken for causal drivers.

• Manufacturing. Presenting actionable, directly causal factors so that a prediction can plausibly be followed by an intervention to change the outcome.

• Crime analysis and environmental monitoring. The paper cites understanding the causes of crimes and of abnormal weather as everyday situations where causal explanation is desired.

Industry relevance. The authors are affiliated with Sony Computer Science Laboratories, Sony Corporation of America, and the method targets tabular data, where gradient boosting machines and neural networks are heavily used in practice and where interpretability tradeoffs are a live concern. Because explanations are built from a score that can be computed from nested combinations and the neuron-level causal searches can be parallelized, the approach is positioned as deployable on moderate-sized models; the paper notes that tabular datasets with underlying causal structure usually do not require very large networks, so the number of neurons in the layer before the output is not large.

Future Directions

• Scaling the causal discovery step. The global explanation requires inferring one causal model per neuron in the layer before the output. Whether this remains practical for networks with many more such neurons, or for deeper architectures, is not resolved.

• Beyond multi-layer perceptrons. The paper states that CENNET can be applied to various types of neural networks, but all reported experiments use MLPs, so performance on architectures such as ResNet, SNN, TabNet, or FT-Transformer is untested.

• Better discretization. The authors note they used a simple discretization method, splitting numerical variables into three equal-frequency categories, and suggest this choice when presenting their non-additive results.

• From observation to intervention. The paper argues that causal explanations increase the feasibility of linking prediction to intervention, but it does not report experiments where explanations were actually used to intervene and alter predicted outcomes.

• Evaluation on real data. The authors note that few real datasets exist with many variables and known causal relationships, which is why they relied on synthetic and quasi-real data; validating against expert-confirmed causal graphs in real domains remains open.

Target Audience

This paper is most useful to researchers and practitioners in explainable AI, causal inference, and applied machine learning who work with tabular data and neural network predictors. It will particularly interest those who need explanations that distinguish direct causes from correlations and that can express combinatorial reasons, as well as readers evaluating whether existing attribution methods such as LIME, SHAP, and their variants are adequate for their domain. A background in graphical causal models and information theory is helpful for following the theorem and the index definitions; the experimental sections are more broadly accessible.

Authors’ abstract

The problem of explaining the results produced by machine learning methods continues to attract attention. Neural network (NN) models, along with gradient boosting machines, are expected to be utilized even in tabular data with high prediction accuracy. This study addresses the related issues of pseudo-correlation, causality, and combinatorial reasons for tabular data in NN predictors. We propose a causal explanation method, CENNET, and a new explanation power index using entropy for the method. CENNET provides causal explanations for predictions by NNs and uses structural causal models (SCMs) effectively combined with the NNs although SCMs are usually not used as predictive models on their own in terms of predictive accuracy. We show that CEN-NET provides such explanations through comparative experiments with existing methods on both synthetic and quasi-real data in classification tasks.