ICON Decomposition: Multivariate Concept-Level Explanations of Deep Representations for Model Auditing

AI summary

Overview

Research area: Explainable AI (XAI), specifically concept-based explainability (C-XAI) and model auditing for shortcut learning, with applications in medical imaging and neuroimaging.

Technical level: Advanced. The paper combines partial least squares (PLS), Type I sum-of-squares allocation, and variance partitioning from the statistics literature, and assumes familiarity with linear probes, Concept Activation Vectors (CAVs), and out-of-distribution testing.

Scope: The paper introduces ICON (Independent Canonical cONcept) decomposition, a multivariate method that quantifies how much of a deep network layer's variance each supplied concept explains conditional on all other concepts and the outcome, and validates it on a synthetic benchmark, a skin-cancer benchmark, and two UK Biobank neuroimaging tasks.

What This Paper Is About

Deep neural networks often learn shortcuts, relying on spurious associations such as acquisition artifacts or demographic variables rather than genuine signal, so models need to be audited for which concepts they actually use. Existing C-XAI tools (linear probes, CAVs) ask whether a concept can be decoded from a layer in isolation, so their scores are inflated by correlations present in the auditing dataset and are not comparable across layers or between categorical and continuous concepts. ICON reframes the problem as multivariate variance decomposition: how much of a layer's variance is explained by each concept given the other concepts and the outcome, and how much is left unexplained by the supplied concept set.

Key Contributions

1. A new framing of C-XAI as multivariate variance decomposition. ICON asks how much of a layer's variance each concept explains conditional on all other concepts and the outcome, rather than asking whether each concept is individually decodable from the layer.

2. A method combining two established statistical tools. ICON combines partial least squares (Wold) to handle high-dimensional DNN representations with Type I (sequential) sum-of-squares allocation (Kutner et al.) to reduce false positives from correlated concepts, drawing on the variance-partitioning literature (Grömping; Clouvel et al.; Dinga et al.).

3. Scores that are comparable across layers and concept types, plus an explicit unexplained share. Because ICON scores are proportions of representation variance, the explained and unexplained shares sum to one at every layer, one metric covers categorical and continuous concepts alike, and the unexplained share warns when the supplied concept set is inadequate — a diagnostic that the authors state no univariate score offers.

4. Validation across three experiments. A ToyBrains simulation with known ground-truth concept importance (90 trained models), the ISIC 2019 skin-cancer benchmark with synthetic artifacts inserted at controlled rates, and two open UK Biobank neuroimaging questions (binge-drinking detection and the brain-age gap) with independent confirmatory tests.

Main Findings

• ICON recovers ground-truth importance more accurately than seven baselines in hidden layers. On ToyBrains, using n = 500 auditing samples and a randomly drawn set of q = 30 concepts from a pool of 100, at H_{L−2} ICON's total variation distance (TVD) error is 0.33 versus 0.52 for the best baseline (marginal R²), 0.69 for the linear probe, and 0.75 for CAV-signal. At H_{L−1}, ICON reaches 0.23 versus 0.38 for the next best baseline (partial R²).

• At the scalar output layer, ICON does not win. At the final decision logit H_L, partial R² is more accurate than ICON (0.14 versus 0.16). The authors state this is expected because at p_L = 1 the multi-response regime motivating ICON's PLS step does not arise, and ICON's advantage therefore lies at the high-dimensional internal representations where C-XAI methods are most often applied.

• The baselines lose most ground on confounders and correlated concepts. For every baseline, the two confounder roles plus the 60 correlated concepts together carry 69–82% of the total error at H_{L−1}. Across the three roles whose true importance is exactly zero, ICON misallocates 0.06 of TVD at H_{L−1}, against 0.16 for partial R² and 0.36 for CAV-signal.

• ICON is nearly insensitive to spurious correlation strength. Between r = 0 and r = 0.6, spurious importance rises by a factor of 7–12× for the linear probe and 7–17× for marginal R², but only 1.2–1.3× for ICON. Partial R² and standardised regression coefficients show no dose response to r but already at r = 0 assign 3–8× the importance ICON does.

• ICON's advantage grows with more concepts and shrinks with more data. As q grows from 5 to 100, ICON's error at H_{L−1} never exceeds 0.22, while baselines degrade to 0.72 (linear probe) and 0.85 (CAV-signal); baselines match ICON only at the smallest concept sets (Shapley R² at q ≤ 5 and partial R² at q ≤ 10). ICON also has the lowest error when the auditing dataset is small, and beyond n ≈ 500 its lead is stable, whereas the linear probe barely improves from n = 25 to n = 5,000 at H_{L−1}. Varying the interpreted layer width shifts every method's error only modestly.

• On naturally occurring ISIC artifacts, ICON confirms prior findings. Applied to a VGG-16 model, ICON shows camera reflection, band_aid, skin_marker and red_color are encoded at H_{L−1} (explained variance 9.8%, 3.4%, 3.0%, and 2.8% respectively), with reflection and red_color propagating into the final logit H_L (4.0% and 2.8%).

• ICON avoids false positives from collinearity in the ISIC setting. Evaluating a clean model (ℳ_micro-00) on auditing datasets of rising microscope–MEL collinearity (correlation rising from 0.30 at k = 10% to 0.93 at k = 90%), ICON stays below 0.01 at both final layers, while the three baselines assign 0.05–0.12. For the timestamp artifact, absent from every training set, ICON again stays below 0.01, whereas CAV-classic swings between 0.005 and 0.11 between adjacent auditing datasets and CAV-signal stays high at 0.09.

• Only ICON tracks the degree of shortcut learning. Across ten models evaluated on the same low-adulteration auditing dataset, accuracy gained from the microscope artifact rises from +0.4% at k = 10% to +10.3% at k = 90%, and only ICON's normalised importance rises correspondingly, from ≈ 0.00 for ℳ_micro-00 to 0.44 for ℳ_micro-90; the three baselines change by only 0.02–0.15. ICON also separates a clean model from a shortcut-affected one (ℳ_micro-00 vs ℳ_micro-40) in its layer-wise decomposition (n = 2,459 auditing images with the artifact in 10% of MEL images), whereas baselines separate the two models only subtly.

• ICON reports unexplained variance that shrinks toward the logits. In the ISIC models the unexplained share is large in early convolutional layers and shrinks toward the prediction logits, indicating that early layers encode concepts outside the supplied set, or encode them non-linearly, or both.

• Binge drinking: ICON identifies sex as a shortcut, confirmed by retraining. On a 3D ResNet-18 classifying binge drinkers from controls at 60 ± 0.5% accuracy (mean ± s.d. over five folds; chance 50%), ICON assigns ≈ 0.39 of the final-layer variance to Sex and essentially none to the outcome or other concepts, whereas the linear probe also surfaces the outcome itself and health-related concepts (BMI, EverAddicted, EmploymentStatus). Retraining on sex-balanced data drops performance to 53.3 ± 0.5% (five folds, n = 1,270 hold-out participants), confirming ICON's reading. A different architecture (SFCN) at 61.2 ± 0.7% over five folds reproduced the same confound, suggesting the shortcut originates in the data rather than modelling choices. Sex correlates with the binge-drinking outcome at r ≈ 0.23 in this cohort.

• Brain age: ICON finds no substantial concept beyond age; the probe surfaces many age-correlated concepts. The SFCN predicts age with MAE 2.89 ± 0.02 years and R² = 0.750 ± 0.004 (five folds, n = 5,000 hold-out participants). ICON assigns ≈ 0.63 of the final-layer variance to the outcome Age, with ≈ 0.36 unexplained, and surfaces scanner-related concepts only in early layers; a few other concepts such as Acquisition date carry tiny non-zero importance, together explaining under 1% of the variance in the prediction. The probe ranks many age-correlated concepts highly, including EmploymentStatus, BMI, Sex, SnapReactionTime, and scanner and imaging-site settings.

• Independent validations support ICON and not the probe on brain age. For the eight concepts selected from each method's top five at H_L, a valid OOD set could be constructed for six, and none changed prediction error significantly (|ΔMAE| ≤ 0.069 years, smallest p-value in any fold 0.07; Bonferroni α = 0.05/6). A conditional independence test on n = 3,738 auditing-dataset participants (Bonferroni α = 0.05/8) flagged three concepts — Acquisition date (χ² = 84.15, p < 0.001), Snap reaction time (χ² = 18.72, p < 0.001), and Alcohol use frequency (χ² = 28.04, p < 0.001) — exactly the three ICON ranks highest after the outcome. Employment status (probe score 0.360, rank 1), Sex (0.108, rank 2) and Household income (0.072, rank 4), ranked highly only by the probe, were flagged by neither validation.

• Layer-wise trajectories add interpretive value. For the brain-age model, representations are scanner-dominated in the initial epochs and Age comes to dominate only as the model converges, leaving a small scanner-related residual at convergence as Acquisition date.

Methodology in Plain English

The core move is to stop asking "can this concept be predicted from this layer?" and start asking "how much of this layer's variance do the supplied concepts and the outcome jointly account for, and how is that share divided among them?" To do that, ICON first uses partial least squares to compress the high-dimensional layer representation into a manageable form, then uses Type I (sequential) sum-of-squares allocation to divide the explained variance among the concepts. Sequential allocation means concepts are credited in order, so a concept that is merely correlated with an already-credited concept receives little or no importance. Because the result is a proportion of the layer's variance rather than a fit score in the units of a particular decoder, the numbers are directly comparable across layers and across categorical and continuous concepts, and whatever is left over is reported explicitly as the unexplained share.

The authors evaluate this in four ways. On the ToyBrains simulator, they can define the causal graph generating the images and outcome, and derive each concept's ground-truth importance at each layer by intervening on the graph, so they can measure estimation error with total variation distance against seven baselines across 90 trained models (six causal scenarios × five architectural variants × three seeds; balanced accuracy 75.8 ± 2.0%). On ISIC 2019, they insert synthetic microscope and timestamp artifacts into controlled percentages of melanoma images, which lets them create models with known shortcut reliance and auditing datasets with controlled collinearity, then check whether each method reports importance that follows the model's actual behaviour. On two UK Biobank neuroimaging questions, they compare ICON with linear probes and validate both methods' findings independently: retraining on sex-balanced data for the binge-drinking model, and an OOD test plus a conditional independence test for the brain-age model.

Why This Matters

Impact on research. The paper argues that C-XAI has largely re-derived variable importance without drawing on decades of statistics work on the same problem, and that the resulting univariate framing causes three specific failures: false positives from dataset correlations, scores incomparable across layers and concept types, and no way to say when the concept set is inadequate. ICON addresses all three in one framework, and by reporting an unexplained share it gives auditors a way to detect that a missing or non-linearly encoded concept may be at play.

Real-world applications:

• Clinical decision support auditing. Before deploying a diagnostic model, check whether it relies on acquisition artifacts, demographics or socioeconomic proxies rather than pathology — the paper's motivating examples include skin markings and band-aids in skin-cancer models and sex or recruitment site in brain-disease models.
• Neuroimaging biomarker validation. Deciding whether a brain-age or disease-prediction model reads a genuine biological signal or a confound such as scanner settings, imaging site, or sex.
• Biomedical cohort research. Auditing models trained on cohorts where clinical, demographic and socioeconomic variables are strongly interdependent and available for only a few hundred annotated samples, a regime where the paper reports ICON's advantage is largest.
• General pre-deployment model auditing for fairness and robustness. Auditing reliance on protected attributes such as sex, income or employment status, where ICON's conditional allocation reduces the risk of falsely flagging merely correlated variables.

Industry relevance. Any organisation that ships or reviews machine-learning models in regulated or high-stakes settings — medical imaging vendors, health systems, pharmaceutical and digital-health research groups, and internal model-risk or fairness review teams — needs post-hoc auditing that does not require retraining or the original training data. ICON preserves those post-hoc benefits, works with mixed categorical and continuous concepts, and needs concept annotations only for the auditing dataset.

Future Directions

• Correcting for the unobservable-signal proxy gap. The authors state that no method recovers the true predictive signal exactly because the image attribute carrying it is unobservable, and that every method must approximate it through the outcome; this proxy gap accounts for most of ICON's own residual error, which points to a clear target for improvement.

• Extending beyond linear encoding assumptions. The unexplained share can be large because concepts are encoded too non-linearly for a linear method to capture, so handling non-linear concept encoding is an open problem the authors themselves flag.

• Better diagnostics for incomplete concept sets. The paper establishes that a large unexplained share signals a missing concept or a non-linear encoding, but does not say how to identify which concept is missing; turning that warning into a search procedure is a natural next step.

• Broader validation and tooling. The provided content is truncated in the Discussion section, so the authors' own stated limitations, broader applications, and any code or software availability statements are not reported here. The paper reports no experiments beyond the ToyBrains simulator, ISIC 2019, and the two UK Biobank tasks.

Target Audience

This paper is most useful to XAI and interpretability researchers, statisticians working on variable importance and variance partitioning, and machine-learning researchers in biomedical imaging and neuroimaging who need to audit trained models. It also suits applied practitioners and reviewers who need a post-hoc auditing method that handles many correlated concepts of mixed types, and readers evaluating whether published claims about model reliance on confounds hold up under independent validation.

Authors’ abstract

Deep neural networks often exploit spurious associations in their training data, a failure known as shortcut learning. Concept-based explainability methods screen for shortcuts by testing whether concepts such as a patient's sex or scanner settings can be decoded from a network layer. Because each concept is evaluated in isolation, these methods can mistake correlations between concepts as evidence that the model uses them. We introduce ICON decomposition, which instead quantifies how much of a layer's variance each concept explains after accounting for all other concepts and the outcome. On synthetic data with known ground truth, ICON recovers concept importance more accurately than seven alternative baseline methods. On skin-lesion and brain-imaging models, it isolates the concepts on which a model genuinely relies, quantifies the representation unexplained by any of the supplied concepts, and yields sparse explanations that we validate by retraining and out-of-distribution testing.