SALVE: Sparse Autoencoder-Latent Vector Editing for Mechanistic Control of Neural Networks

AI summary

Overview

Research area: Mechanistic interpretability and model editing for deep neural networks, specifically sparse autoencoder (SAE)-based feature discovery and permanent weight-space intervention in vision models.

Technical level: Intermediate to Advanced. The paper assumes familiarity with sparse autoencoders, ℓ1 regularization, saliency methods such as Grad-CAM, and the idea of mechanistic interpretability, though the core pipeline is described in accessible terms.

Scope (one sentence): The paper introduces SALVE, a "discover, validate, control" pipeline that trains a sparse autoencoder on internal model activations, validates the resulting features with visualization methods, and uses them to make permanent, continuous edits to a model's final-layer weights, validated on ResNet-18 and ViT-B/16.

What This Paper Is About

Deep neural networks perform well but are hard to interpret and control. Most existing work either explains a model's behavior after the fact (interpretation) or temporarily nudges its activations during inference (steering), leaving a gap between understanding and durable control. SALVE addresses this gap by learning an unsupervised, sparse feature basis from a model's own activations and then using the structure of that basis to permanently suppress or enhance specific concepts directly in the model's weights.

Key Contributions

1. A unified "discover, validate, control" framework. SALVE trains an ℓ1-regularized linear sparse autoencoder on internal activations (the final average pooling layer for ResNet-18, the [CLS] token after the final transformer encoder block for ViT) to obtain a sparse, model-native feature basis without supervision.

2. Grad-FAM, a feature-level saliency method. The paper proposes Gradient-weighted Feature Activation Mapping, which repurposes Grad-CAM logic to visualize which input regions activate a specific latent feature rather than a class prediction, and pairs it with activation maximization to visualize a feature's abstract concept.

3. Permanent, continuous weight-space editing guided by SAE structure. Using the SAE decoder matrix, the method multiplicatively rescales final-layer weights as w′ij = wij · max(0, 1 ± α|cj|), where α ≥ 0 controls intervention strength and enhancement (+) or suppression (−). Unlike inference-time steering, this edit is permanent and requires no auxiliary modules at inference.

4. The critical suppression threshold, α_crit. The paper derives both an analytical estimate (a linear-approximation lower bound, α_crit^(n) ≈ z_i^(n) / R_i(x^(n))) and a numerical computation of the intervention strength needed to drive a class's feature contribution to the logit to zero, providing a per-sample measure of how much a class relies on its dominant feature.

Main Findings

• SAE latents align with class and sub-concept semantics. On ResNet-18/Imagenette, mean feature activations show a sparse structure in which a single feature is strongly associated with a single class. Activation maximization of the "golf ball" feature reveals golf-ball-like texture, and Grad-FAM grounds a sample golf ball image as follows: the dominant "Feature 1" gives a high-level concept map, "Feature 2" activates on the ball's surface texture, and "Features 3 and 4" highlight parts of the golf club.

• Class-targeted edits flip and disable predictions. On an ambiguous out-of-distribution image containing both a golf ball and a church (the model originally predicts "Church"), suppressing the dominant "Church" feature flips the prediction to "Golf ball", and enhancing the "Golf ball" feature produces the same flip. On the Imagenette test set, suppressing the "Church" feature reduces that class's accuracy to near zero while enhancing the "Golf ball" feature maintains its near-perfect accuracy, in both cases with minimal off-target effects.

• Cross-class features can be edited, and reveal entanglement. A "Tower Feature" is activated by both churches (Class 4) and petrol pumps (Class 7). Suppressing it reduces "Petrol Pump" accuracy while leaving "Church" largely unaffected; enhancing it increases "Petrol Pump" accuracy. The intervention also produces a consistent inverse effect on "Chain Saw" (Class 3), which the authors interpret as a learned spurious negative correlation where the Tower Feature inhibits the Chain Saw classification.

• Suppression exhibits a sharp threshold and confidence reallocation. For "Church", accuracy stays stable as α increases, then drops sharply past a critical threshold, while other classes remain high. The drop corresponds to the model's confidence being redistributed among other classes. Suppression curves computed for the dominant feature of all classes show similar behavior with different thresholds, indicating classes rely on their dominant features to different degrees.

• α_crit behaves as expected relative to empirical thresholds. For ResNet-18, the analytical estimate is a lower bound on the critical intervention strength; the numerical α_crit aligns well with both the lower bound and the empirical α_50% (the population-level point where class accuracy falls to 50%), and captures a wider range of values where the linear approximation is less valid.

• Architecture matters for the threshold gap. On ViT-B/16, the analytical α_crit is a much more conservative lower bound, and there is a larger discrepancy between numerical α_crit (total evidence loss) and empirical α_50% (a typical prediction flip). The authors hypothesize this reflects the ViT's "curved", non-linear representation space and its tendency to produce less confident, more distributed logits, so a smaller intervention is needed to be surpassed by a competitor.

• Dataset diversity increases feature sharing. On CIFAR-100, a structured and relatively sparse representation still emerges, but dominant features show more cross-class sharing than in Imagenette, and off-target effects remain limited though higher than in Imagenette.

• Baselines match outcomes but differ in mechanism. ROME (rank-one weight editing adapted from language models to vision) and SAE-based Activation Steering (inference-time) both reduce target class accuracy to near zero with minimal off-target effects. The authors argue SALVE's distinctive benefits are permanent edits without inference overhead, systematic control over multiple latent concepts, and quantitative diagnostics such as α_crit.

• Results are robust to SAE randomness. Intervention effects were consistent across multiple SAE initializations and training runs even though the specific learned basis varied; the shaded regions in the "Church" suppression figure indicate standard deviation across 10 SAE initializations.

Methodology in Plain English

The authors work in three stages.

Discover. They take a pretrained ResNet-18 or ViT-B/16, replace the final classification layer to match the target number of classes, and fine-tune on Imagenette (10 classes) or, for ResNet-18 only, CIFAR-100 (100 classes). They then extract internal activations, a 512-dimensional vector from ResNet-18's final average pooling layer and a 768-dimensional [CLS] token representation from the ViT after the final transformer encoder block. A simple linear autoencoder with an encoder and decoder is trained to reconstruct these activations while an ℓ1 penalty on the latent code encourages sparsity. To find class-specific features, they average the latent activations per class; because latents are signed and sparse, this averaging suppresses inconsistent features, and they rank features by the magnitude of the class-conditional mean.

Validate. Two visualization tools check that the features mean something. Activation maximization synthesizes an image from random noise by gradient ascent on a chosen latent feature's activation, with total-variation and L2 penalties plus stochastic padding, jittering, scaling, rotation and center cropping (final 224×224 pixels) to keep the image visually coherent. Grad-FAM changes the target of the standard Grad-CAM gradient from a class to a specific latent feature, showing which image regions activate it.

Control. Using the SAE decoder, each final-layer weight is multiplied by max(0, 1 ± α|cj|), where cj is the selected latent feature's contribution to activation coordinate j. That gives a continuous knob: positive α enhances the feature, negative α suppresses it. Because the edit multiplies learned weights rather than adding to activations, it interacts with each sample's activation pattern, which is what allows per-sample diagnostics. The critical suppression threshold α_crit is then defined as the smallest α at which the feature-perturbed, bias-free logit equals zero, computed both by a linear approximation and numerically.

Why This Matters

The paper's contribution is less about beating existing benchmarks than about closing the loop from interpretation to control: the same sparse basis used to explain a model is used to permanently edit it, with a quantitative measure (α_crit) of how much each class depends on each feature.

Impact on research: It positions permanent, feature-driven weight edits as an alternative to the dominant inference-time steering paradigm used with SAEs, and offers a diagnostic that could be adapted into robustness auditing. It also provides a convolutional-versus-transformer comparison of how representation geometry (linear vs. "curved") affects intervention sensitivity.

Real-world applications (as motivated by the paper):

• Compliance-sensitive or deployment-constrained settings where a fixed, verifiable model state is preferable to carrying steering vectors or auxiliary modules at inference.
• Robustness auditing of vision models, using per-class and per-sample α_crit to pinpoint fragile, feature-dependent representations.
• Diagnosing and quantifying spurious cross-class correlations, such as the Tower Feature's inhibitory effect on the Chain Saw class.
• Targeted removal or reduction of a specific class or concept's influence in a deployed model without retraining or architectural change.

Industry relevance: Because the method is fully post-hoc, requires no retraining or architectural modification, and was run on a single local workstation reproducible on a standard modern laptop with GPU acceleration, it is comparatively cheap to apply to existing trained models. The paper notes that intervention effectiveness depends on both the backbone's training dynamics (e.g., batch size) and the SAE's training parameters, which links training procedures to post-hoc controllability.

Future Directions

• Scale to more models, datasets, and modalities. The study is framed as a mechanistic case analysis on basic datasets and well-understood architectures; the authors call for extending to larger models, more complex datasets, and other modalities such as natural language processing.
• Try newer SAE variants. The simple linear ℓ1 SAE shows limits in high-diversity settings such as CIFAR-100, motivating evaluation of Gated, Top-k, and JumpReLU SAEs to improve reconstruction fidelity and feature separation.
• Co-design training and intervention. Since intervention effectiveness depends on backbone training dynamics and SAE hyperparameters, the authors suggest designing training procedures that produce models that are inherently more editable.
• Push edits deeper and add rigor. Adapting the weight modulation to deeper layers could change how features are formed rather than how they are combined, and formal disentanglement metrics could better evaluate cases of feature entanglement such as the Tower Feature.
• Link feature reliance to adversarial vulnerability. The authors propose investigating whether low-α_crit features and samples correlate with known adversarial vulnerabilities.

Target Audience

Mechanistic interpretability researchers working with sparse autoencoders; model-editing and model-steering researchers interested in permanent versus inference-time interventions; robustness, safety, and reliability engineers who need to audit or constrain the behavior of deployed vision models; and practitioners in compliance-sensitive domains who require fixed, verifiable model states without inference-time overhead. The paper is also useful to readers comparing convolutional and transformer architectures from an interpretability standpoint, since it runs the same pipeline on both ResNet-18 and ViT-B/16.

Authors’ abstract

Deep neural networks achieve impressive performance but remain difficult to interpret and control. We present SALVE (Sparse Autoencoder-Latent Vector Editing), a unified "discover, validate, and control" framework that bridges mechanistic interpretability and model editing. Using an $\ell_1$-regularized autoencoder, we learn a sparse, model-native feature basis without supervision. We validate these features with Grad-FAM, a feature-level saliency mapping method that visually grounds latent features in input data. Leveraging the autoencoder's structure, we perform precise and permanent weight-space interventions, enabling continuous modulation of both class-defining and cross-class features. We further derive a critical suppression threshold, $α_{crit}$, quantifying each class's reliance on its dominant feature, supporting fine-grained robustness diagnostics. Our approach is validated on both convolutional (ResNet-18) and transformer-based (ViT-B/16) models, demonstrating consistent, interpretable control over their behavior. This work contributes a principled methodology for turning feature discovery into actionable model edits, advancing the development of transparent and controllable AI systems.