Spectral-Spatial Interpretation

AI summary

Overview

Research area: Explainable AI (post-hoc attribution) applied to multi-channel and multispectral image models, combining cooperative game theory (Shapley values) with remote sensing and fine-grained vision.

Technical level: Advanced — the paper assumes familiarity with Shapley values, masking-based perturbation metrics (deletion/insertion AUC), and multispectral remote sensing bands.

Scope: The paper proposes Spectral-Spatial Interpretation (SSI), a full-grid attribution method that scores every position–channel pair (g, c) in a multi-channel image by solving a small Shapley game over the channels at each spatial position while holding the rest of the observed image fixed.

What This Paper Is About

Post-hoc image explanations typically answer only one question at a time: spatial heatmaps (Grad-CAM, RISE, Extremal Perturbation) show where a model looks but average over channels, while spectral/channel attribution methods (mutual information, attention weights, SHAP-style scores) rank bands globally but average over space. This is a poor fit for multispectral data, where a band can be decisive in one small region and irrelevant elsewhere — an assumption the authors describe as forcing a rank-1 structure A(g, c) = s(g)q(c). The paper's goal is a single joint attribution matrix that says, at each location, which channels support the prediction and by how much.

Key Contributions

1. Joint position–channel player formulation. The authors define attribution over players (g, c) ∈ 𝒫 = 𝒢 × 𝒞, targeting localized channel evidence rather than only a heatmap or a global channel ranking.
2. The observed-context SSI estimator. For each position g, SSI fixes the surrounding image context B_g and applies the standard Shapley value to a conditional game w_g over only the channels at that position, producing Φ(g, c) for the full grid.
3. Theoretical characterization. Theorem 1 gives a local Shapley characterization (local efficiency, within-position symmetry, within-position null-player, additivity); Theorem 2 proves contextual aggregation consistency, Σ_c Φ(g, c) = Δ(g); Proposition 1 shows that contextual aggregation and global efficiency cannot both hold across all games.
4. An explicit distinction from Owen-style accounting. SSI uses the same position–channel grouping as the Owen value but fixes the observed external context instead of averaging over external spatial coalitions, and is evaluated against an MC-Owen estimator.

Main Findings

• Best joint faithfulness across three datasets. Under both per-channel mean and zero baselines, SSI achieves the lowest joint deletion AUC and highest grid-sum insertion AUC on CUB-200, EuroSAT, and HLS Burn Scar. Under the mean baseline, CUB-200 joint deletion is .0088 ± .001 for SSI versus .0112 for both IG and DeepSHAP, .0178 for KernelSHAP, and .0183 for RISE.
• Strong insertion despite a much smaller budget. On CUB-200 (mean baseline) SSI reaches the highest insertion AUC (.7132 ± .019), surpassing RISE (.6886 ± .017) even though RISE uses 8,000 spatial masks versus 2048 for SSI.
• Larger gaps on multispectral data. On EuroSAT (13 bands) SSI records joint deletion .1397 ± .020 and insertion .7319 ± .023; on HLS Burn Scar (6 bands) .1545 ± .008 and .6638 ± .008, versus .2160 and .6291 for IG on HLS.
• Rank-1 factorization is insufficient. Rank-1 deletion AUC stays worse than joint deletion for every method, and the gap is smallest on CUB-200 (C = 3) and largest on EuroSAT (C = 13), suggesting the rank-1 approximation degrades as channel diversity grows.
• Best spatial localization. On HLS Burn Scar, SSI reaches .389 mIoU at Top-10% coverage compared to .277 for DeepSHAP and .191 for RISE; on CUB-200 it has the strongest mIoU across all coverage levels. KernelSHAP, the flat-Shapley baseline on the same player granularity, lags substantially on both.
• Best spectral summaries. SSI achieves the highest Hit@K at every reported K and the lowest channel deletion AUC on both EuroSAT and HLS Burn Scar.
• Beats MC-Owen at roughly 6% of the budget. With |𝒫| = 384 on HLS Burn Scar, MC-Owen with M = 200 block-respecting permutations uses up to 76,800 evaluations versus 4,480 for SSI (G·K(C+1)). SSI wins on all retained metrics: joint deletion .155 versus .173, channel deletion .218 versus .261, insertion .664 versus .627, mIoU@15% .387 versus .254, and Hit@5 .887 versus .730.
• Wins against stronger gradient baselines. On HLS Burn Scar, SSI is best on all ten reported metrics against Expected Gradients, Guided IG, and FullGrad (e.g. joint deletion .155 versus .185, .185, and .278).
• Complementary ROAD evaluation. SSI achieves the highest LeRF AUC (.657) and lowest MoRF AUC (.1143) among nine evaluated methods on HLS Burn Scar.
• Qualitative patterns match physics. On EuroSAT, SSI gives high attribution to red-edge and NIR bands with low red-band attribution for vegetation classes; on HLS Burn Scar it emphasizes Red and SWIR near the burn region; on CUB-200, plumage regions receive higher attribution in their dominant color channel.

Methodology in Plain English

SSI treats an image as a grid of positions, each holding a set of spectral channels. To explain a prediction, it visits one grid position at a time and keeps the entire rest of the image at its real, observed values. Within that single position it plays a small cooperative game: which subsets of channels, when kept versus replaced by a baseline, change the target-class logit? The change in logit relative to a fully masked input defines the value function v, and the Shapley value of this small game assigns credit to each channel at that position. Doing this for every position yields a full matrix Φ(g, c) rather than a single heatmap or a channel ranking.

Two consequences follow directly from the construction. Within each position the scores obey the standard Shapley fairness axioms, and summing the channels at a position exactly recovers the full-context contrast Δ(g) obtained by replacing that whole position with the baseline — no monotonicity or architectural assumption needed. The authors also prove that no rule can simultaneously deliver this per-position aggregation and allocate the total prediction value globally, so SSI deliberately targets local explanation rather than global accounting.

Practically, the game is estimated by permutation sampling over the channels at each position with K = min(2^C, 10) permutations, or by exhaustive enumeration of all 2^C subsets when 2^C ≤ 10. CUB-200 (C = 3) is enumerated exactly; for EuroSAT (C = 13) and HLS (C = 6) the empirical concentration bounds stay below 10^-2. The same full position–channel player space is used for every baseline, on an 8×8 grid for EuroSAT and HLS Burn Scar and a 16×16 grid for CUB-200, with a per-channel mean baseline as the primary masking value and zero baseline as a robustness check.

Why This Matters

Impact on research. The paper reframes explanation targets for multi-channel imagery from "a heatmap plus a global band ranking" to a joint position–channel object, and shows empirically that the rank-1 factorization underlying common two-stage pipelines loses faithfulness — with the degradation growing as channel count rises (largest at C = 13 on EuroSAT). Theorems 1 and 2 give the new object axiomatic grounding, and Proposition 1 clarifies a genuine trade-off between contextual and global efficiency that any hierarchical attribution method must confront.

Real-world applications:

• Burn-scar mapping and wildfire damage assessment from Harmonized Landsat–Sentinel-2 imagery, where Red and SWIR evidence is localized within the scar.
• Land-use and land-cover classification with Sentinel-2, where vegetation, water, and built-up areas align with different bands (B3, B8A, B12 in the paper's example) and vegetation signatures appear in red-edge and NIR.
• Fine-grained visual recognition such as bird species classification, where differently colored plumage regions map to different RGB channels.
• Model auditing and trust-building for any deployed multi-sensor or multi-band perception system where stakeholders need to know both where and on what physical evidence a prediction rests.

Industry relevance. SSI is model-agnostic and post-hoc, so it can be applied to already-trained models without retraining; cost grows only linearly in channel count and estimation error only as sqrt(ln C). The roughly 6%-of-budget result against MC-Owen matters for operational pipelines where explanation cost must stay near inference cost.

Future Directions

• Scaling to hundreds of bands. The authors note that applying SSI to hyperspectral sensors with hundreds of correlated bands may benefit from channel screening or adaptive grouping to reduce spectral redundancy before local Shapley allocation, and flag validation on operational hyperspectral missions as future work.
• Imaging spectroscopy missions. SSI is proposed as potentially useful for sensors such as EnMAP, where spectral channels carry spatially localized physical information.
• Cross-domain transfer. The observed-context formulation is suggested to extend to other multi-channel imaging domains with localized physical meaning, including biomedical microscopy and industrial materials inspection.
• Closing the sampling gap in hierarchical accounting. The paper reports the MC-Owen comparison only on HLS Burn Scar, the smallest player setting studied (|𝒫| = 384), and notes that tightening MC-Owen noise (permutation estimators converge at O(M^(-1/2))) would widen rather than narrow the cost gap — leaving the broader hierarchical comparison open.

Target Audience

This paper is most useful to explainable-AI researchers working on attribution methods and game-theoretic feature credit allocation; to remote sensing and Earth-observation scientists who need band-level, location-aware explanations from multispectral or hyperspectral classifiers and segmentation models; and to practitioners in regulated or safety-critical imaging domains (environmental monitoring, biomedical imaging, industrial inspection) who must justify model predictions to non-ML stakeholders. Readers should be comfortable with Shapley values and perturbation-based faithfulness metrics; the empirical sections are otherwise self-contained.

Authors’ abstract

Post-hoc image explanations usually show where a model looks using spatial heatmaps or indicate which channels matter using global channel scores. These separate views are limiting for multispectral data because channel relevance is often spatially localized and physically meaningful. We introduce Spectral-Spatial Interpretation (SSI), a full-grid attribution method over position-channel players (g,c). For each spatial position, SSI fixes the observed surrounding context, defines a conditional game over the channels at that position, and applies the Shapley value to this local game. The result is a full spectral-spatial attribution matrix rather than a single heatmap or a global channel ranking. SSI has a clear local interpretation. It satisfies Shapley fairness within each position, and aggregating its channel attributions exactly recovers the position-level contextual contrast obtained by replacing that position with the baseline. SSI also differs from context-averaged Owen-style hierarchical accounting, which uses the same position-channel grouping but averages over external spatial coalitions. On HLS Burn Scar, SSI outperforms an MC-Owen estimator across retained attribution metrics while using only about 6% of its evaluation budget. Across CUB-200, EuroSAT, and HLS Burn Scar under both per-channel mean and zero baselines, SSI achieves the best joint deletion and insertion AUC over dense, flat, and rank-1 baselines, yields the strongest spectral hit rates and channel deletion, and leads spatial localization on both annotated datasets. Code is available at https://github.com/AtheavenMa/Spectral-Spatial-Interpretation.