H-SPLID: HSIC-based Saliency Preserving Latent Information Decomposition

AI summary

Overview

Research area: Representation learning and robust machine learning — specifically saliency-aware latent space decomposition, kernel-based statistical independence (HSIC) regularization, and adversarial robustness in image classification.

Technical level: Intermediate to Advanced. The empirical results are accessible, but the core argument rests on a theorem involving RKHS kernel bounds, Hirschfeld-Gebelein-Rényi style dependence measures, and truncated multivariate normal distributions.

Scope: The paper introduces H-SPLID, an algorithm that splits a neural network's latent space into a low-dimensional task-relevant ("salient") subspace and a complementary "non-salient" subspace, proves a robustness bound tying prediction sensitivity to salient-space dimensionality and HSIC, and validates the approach across five image benchmarks spanning synthetic, natural, medical, and transfer-learning settings.

What This Paper Is About

Neural networks trained with standard cross-entropy often mix task-relevant signal with redundant information inside a single entangled latent space, which makes them fragile: a classifier that looks accurate can nonetheless depend on irrelevant input regions such as an image's background. The paper asks whether explicitly separating the latent space into a salient part (used for classification) and a non-salient part (which absorbs the rest of the input variability) can produce representations that are both accurate and insensitive to perturbations of irrelevant input features. The authors propose H-SPLID to do exactly this, and additionally prove that the expected change in predictions under input perturbations is bounded by the dimensionality of the salient subspace and the HSIC between inputs and salient representations.

Key Contributions

1. A new decomposition algorithm: H-SPLID, which learns salient features by partitioning the network's latent space into salient and non-salient subspaces using a learnable diagonal mask, combined with masked clustering losses and two HSIC penalties.

2. A robustness theorem: The paper proves that the dimensionality reduction of the salient subspace together with the HSIC between inputs and salient latent representations upper-bounds the expected change in predictions under input perturbations (Theorem 3.2).

3. A volume bound: A corollary (Corollary 3.3) shows the same two quantities also bound the volume of the input domain that is vulnerable to perturbations, connecting robustness to the size of the "salient-active" input region.

4. Empirical demonstration of saliency: Experiments on C-MNIST, a four-class COCO subset, ISIC-2017 skin lesion images, ImageNet-9, and CounterAnimal show that H-SPLID relies primarily on salient input components, measured by attacks and corruptions applied to non-salient regions, without adversarial training or prior knowledge of the redundant region.

Main Findings

• Diagnostic test on C-MNIST (Figure 1): A model trained with cross-entropy on the left digit of concatenated MNIST digits reaches 96.86% test accuracy, but accuracy drops to 31.76% under a high-magnitude PGD attack with ε = 1.0 applied to the right (irrelevant) digit. H-SPLID separates the latent space so that the salient subspace captures class-discriminative structure with 97.14% accuracy, the non-salient subspace contains no class-relevant information (8.57% accuracy), and robustness accuracy under the irrelevant-digit perturbation rises to 87.46%.

• COCO adversarial robustness (Table 2): Under background-only PGD attacks at ε = 3/255, H-SPLID reaches 57.5%, with the closest competitor at 43.0%. Under full-image PGD at ε = 2/255, H-SPLID sustains 58.9% versus 34.2% for the vanilla network and 41.9% for the best-performing competitor. Under AutoAttack, H-SPLID "consistently outperforms all baselines." All models are trained without adversarial data.

• Largest gains come from background-targeted attacks: The paper reports that the most pronounced improvements occur under background-only perturbations, which it interprets as evidence that H-SPLID effectively isolates redundant information.

• Medical imaging robustness (ISIC-2017, three classes: nevus, melanoma, seborrheic keratosis): Perturbing only non-lesion pixels with real-world corruptions averaged over 10 random seeds, H-SPLID achieves 76.78 ± 0.778 without perturbation, 70.00 ± 1.619 under brightness, 68.38 ± 1.376 under defocus blur, and 69.50 under occlusion (the standard deviation for this cell is cut off in the provided content). For comparison, the vanilla model scores 75.45 ± 0.986, 66.43 ± 2.527, 63.77 ± 2.388, and 62.87 ± 3.081 respectively, and HBaR scores 75.90 ± 0.844, 68.70 ± 1.942, 65.62 ± 2.058, and 66.18 ± 3.013.

• Transfer learning: The paper states it uses ImageNet-9 and CounterAnimal for transfer learning experiments, and that H-SPLID improves transfer accuracy of ResNet-based classifiers trained on ImageNet under real-world perturbations. The specific ImageNet-9 and CounterAnimal numbers are referenced as Table 4 but are not reproduced in the provided content.

• Theoretical tightening over prior work: The authors state their theorem differs from Wang et al. [56] in two ways: it sharpens the dependence of the upper bound on the power of the perturbation, explicitly linking it to the dimension of the salient mask s, and it extends their binary classification framework (k = 1) to multi-class classification with arbitrary k.

• Continuous masks work in practice: The mask optimization yields a continuous β ∈ [0,1]^m with a closed-form per-coordinate solution; the paper notes that binarizing by thresholding at 0.5 yields results similar to using the continuous mask directly, since learned values typically concentrate near 0 or 1.

Methodology in Plain English

The model is a standard encoder plus linear output layer, trained with a softmax cross-entropy loss. The novel part is what happens to the latent vector z produced by the encoder.

Splitting the latent space. A learnable diagonal mask M_s = diag(β) with β ∈ {0,1}^m selects a subset of latent dimensions as "salient." The complementary mask M_n = I − M_s selects the rest. Only the salient part is fed to the classifier; the non-salient part is kept out of the prediction path.

Two clustering-style regularizers. The salient loss L_s pulls each class's salient representations toward that class's centroid, encouraging compact, discriminative, unimodal clusters. The non-salient loss L_n pulls all samples' non-salient representations toward the global centroid, letting that subspace capture shared, task-irrelevant variation such as background.

Two HSIC penalties. The Hilbert-Schmidt Independence Criterion measures statistical dependence between two sets of variables. One penalty reduces dependence between the raw inputs X and the salient representations Z_s (forcing the salient space to drop redundant input information). The other reduces dependence between the labels Y and the non-salient representations Z_n (forcing the non-salient space to carry no label information). The unbiased empirical HSIC estimator is used, and the Normalized Cross Covariance Operator is used to make the penalty scale-insensitive.

Alternating optimization. Training alternates between (a) updating network parameters θ = {ψ, W} by minibatch SGD on the combined loss with the mask fixed, and (b) updating the mask with the network fixed. Step (b) has a closed-form per-coordinate solution that compares the weighted within-class variance in the salient space against the global variance in the non-salient space; a moving average is used across iterations to improve convergence.

Evaluation strategy. The key empirical idea is indirect: rather than checking whether the model "looks at" the right pixels, the authors attack or corrupt the parts of the input that should be irrelevant (the second digit, the background, non-lesion skin pixels) and measure how much accuracy moves. Methods are trained only on clean data, with no adversarial training and no access to saliency masks; saliency masks are used only at test time to restrict where attacks are applied. Comparisons include weight decay (L2), L1 regularization, Group-Lasso, two activation-sparsity variants (L1 and Group-Lasso penalties on penultimate-layer activations), HBaR (the HSIC bottleneck), and a vanilla cross-entropy baseline.

Why This Matters

The paper offers a route to robustness that does not require adversarial training — which is computationally demanding and tailored to specific attacks — and that does not require saliency annotations, pretrained saliency generators, or external background datasets (in contrast to Contrastive Analysis methods). It also provides a formal, checkable link between two quantities a practitioner can control (how many salient dimensions are kept, and how strongly HSIC is penalized) and a measurable property (sensitivity to input perturbations).

Real-world applications:

• Medical imaging: The ISIC-2017 experiments target skin lesion classification, where robustness to lighting, blur, and occlusion matters and where the relevant region is a small part of the image.
• Safety-critical vision systems: Any deployment where a model could be manipulated by altering background context rather than the object of interest.
• Wildlife and ecological monitoring: The CounterAnimal benchmark involves animals photographed against atypical backgrounds, a setting where spurious background correlations can badly mislead classifiers.
• Distillation and efficient deployment: The learned mask explicitly reveals which latent dimensions matter, which could inform model compression or feature-dimension selection.

Industry relevance: Because H-SPLID runs on top of a standard encoder-plus-linear-head architecture (LeNet-3, ResNet-18, and ResNet-50 in these experiments) and trains on clean data only, it is a drop-in regularization scheme rather than a new architecture. The public code release at github.com/neu-spiral/H-SPLID lowers the barrier to adoption.

Future Directions

The paper does not include an explicit future-work section in the provided content; the following are open questions the work raises.

• Scaling the guarantee: The robustness bound depends on the salient mask support s, the Lipschitz constant L, the truncation radius R, the variance σ², and the kernel sup-norm bounds K_F and K_G. How the bound behaves for very large encoders, and whether the constants are loose in practice, is not established in the provided content.

• Beyond image classification: All five evaluated datasets are image classification tasks. Whether the same decomposition helps in domains where the salient/non-salient split is not spatially meaningful (text, tabular data, time series) is not reported.

• Choosing the salient dimensionality: The mask is learned rather than specified, but how the resulting number of salient dimensions relates to the intrinsic difficulty of a task, and whether it transfers across datasets, is not addressed in the provided content.

• Interaction with adversarial training: H-SPLID's gains are achieved without adversarial training, and it is untested in combination with it. Combining the two — or testing whether H-SPLID's subspace separation makes adversarial training cheaper — is a natural open question, as is robustness to adaptive attacks that explicitly target the salient subspace.

• Reporting gaps: The provided content references ImageNet-9 and CounterAnimal results in Table 4 and a CounterAnimal footnote about its Common/Counter split taxonomy, but those numbers are not reproduced here, so the transfer-learning conclusions cannot be verified from this content alone.

Target Audience

Researchers and graduate students in representation learning, kernel methods, and adversarial robustness will get the most from this paper, particularly those already familiar with the information bottleneck and the HSIC bottleneck line of work (Wang et al. [56] is the direct predecessor). Practitioners building computer vision systems in medical imaging or other domains where background context is a known failure mode will find the empirical results and the released code directly actionable. Readers without a kernel-methods background can still follow the methodology and experiments, but the theoretical guarantees in Sections 3.5 and the appendices require comfort with RKHS theory.

Authors’ abstract

We introduce H-SPLID, a novel algorithm for learning salient feature representations through the explicit decomposition of salient and non-salient features into separate spaces. We show that H-SPLID promotes learning low-dimensional, task-relevant features. We prove that the expected prediction deviation under input perturbations is upper-bounded by the dimension of the salient subspace and the Hilbert-Schmidt Independence Criterion (HSIC) between inputs and representations. This establishes a link between robustness and latent representation compression in terms of the dimensionality and information preserved. Empirical evaluations on image classification tasks show that models trained with H-SPLID primarily rely on salient input components, as indicated by reduced sensitivity to perturbations affecting non-salient features, such as image backgrounds. Our code is available at https://github.com/neu-spiral/H-SPLID.