Causal Interpretation of Neural Network Computations with Contribution Decomposition

AI summary

Overview

Research area: Mechanistic interpretability of neural networks, with cross-over into computational neuroscience (retina models).

Technical level: Intermediate. Familiarity with convolutional networks, attribution/gradient methods, and sparse autoencoders helps, but the paper's framing around receptive fields and projective fields is accessible.

Scope: The paper introduces CODEC (Contribution Decomposition), a framework that quantifies how hidden neurons causally drive network outputs and then decomposes those contributions into sparse, coordinated modes, demonstrated on ResNet-50 image classifiers, a Vision Transformer (ViT-B), and a three-layer CNN model of the vertebrate retina.

What This Paper Is About

Most interpretability work studies what hidden units activate on, which only reflects a unit's sensitivity to its input and says nothing about whether that unit actually changes the network's output. The authors argue that the right unit of analysis is a neuron's contribution — the composition of its receptive field (sensitivity to input) and its projective field (effect on output) — and that groups of neurons act together in coordinated patterns. The goal is a general method that reveals how hidden neurons combine to construct a specific output, and that allows that output to be manipulated.

Key Contributions

1. A four-stage framework (CODEC) consisting of: choosing a scalar contribution target, computing per-hidden-unit contributions with attribution methods, decomposing those contributions into sparse modes via a sparse autoencoder, and visualizing the input features driving those modes.
2. Extension of attribution to hidden layers with a scalar target, which avoids an intractable three-dimensional decomposition of shape n_inputs × n_neurons × n_logits per layer. Supported targets are the top logit (the standard used here), the sum of top-k logits, and the entropy of the output distribution.
3. Empirical characterization of how contributions evolve across network depth in ResNet-50, including sparsity, dimensionality, and the relationship between positive and negative contributions.
4. Demonstrations that contribution modes support causal control and interpretation, via ablation/preservation experiments in ResNet-50, application to retina model interneurons, and initial results on ViT-B.

Main Findings

• Contributions are sparser than activations. Measured with the Hoyer sparsity index (ratio of L1 to L2 norms, 0 to 1), contributions were consistently sparser than activations at all layers, indicating that only a small subset of channels is functionally relevant for each classification decision.
• Sparsity increases with depth. Contributions grew more sparse throughout ResNet-50, consistent with feature selectivity emerging hierarchically.
• Sign matters in a way activations hide. ReLU constraints keep activations positive, but contributions can be positive or negative, indicating whether a spatial position increases or decreases the likelihood of the target output.
• Positive and negative contributions decorrelate with depth. In earlier layers, positive and negative contributions were highly correlated within each channel; through the network they became progressively decorrelated.
• Contributions are higher-dimensional than activations. Using PCA on spatially summed, class-averaged quantities, both increased in dimensionality with depth, but contributions required more components to reach 95 percent explained variance.
• Contribution modes align better with ImageNet classes than activation modes. Correlating mode loadings against binary class indicators over 50,000 validation images produced a k × 1000 matrix; contribution modes were more correlated with classes than activation modes, especially at intermediate layers, and more correlated with classes than individual channels despite never seeing labels during optimization.
• Reconstruction is accurate. Average R² was 0.85 for contributions and 0.84 for activations across all layers.
• Modes enable targeted control. Ablating 2 percent of salient channels identified from the top 2 most correlated modes for the "black widow" class greatly reduced target-class accuracy while leaving off-target performance largely unaffected; preservation of those channels yielded networks that could accurately classify only the targeted class.
• A possible representational shift between blocks 6 and 7. Ablation efficacy rose sharply between these blocks in ResNet-50.
• Modes generalize beyond the 1000 labeled classes. The authors ablated and preserved specific taxonomic categories.
• Modes map back to interpretable image features. Contribution maps (the InputGrad method) mask input pixels used by channels within a mode; example stimuli from different classes passing through two violin-correlated modes at layer 7 revealed consistent lower-level features such as shiny wood and hands.
• Retina: modes generate dynamic receptive fields. In a 3-layer CNN with 8 channels in the first two layers predicting 4–17 ganglion cells, contribution modes combined to drive cells at different times. Single modes produced similar instantaneous receptive fields across different ganglion cells, while simultaneous activation of multiple modes produced IRFs ranging from center-surround to oriented or textured. Clustering cells by their mode patterns with k-means (three clusters, chosen by silhouette value) gave consistent results in layers 1 and 2.
• Vision Transformer results are weaker. CODEC applied to token features, MLPs, and attention layers in ViT-B found contributions sparser than activations and ablated more effectively than activation-based modes, but overall ablation performance was inferior to CNNs, which the authors attribute to ViTs lacking an explicit spatial equivariant bias.
• The framework is robust to most hyperparameters. SAE reconstruction was little affected by L1 regularization, random seed, or the non-negativity constraint. Performance degraded when the threshold was too high (0.9 vs 0.5), dictionary size was equal to or smaller than the number of channels, or MLP size was not considerably greater than the number of classes.

Methodology in Plain English

The authors treat the question "how does this output get built?" as a causal question about each hidden unit, not just a descriptive question about what it responds to.

First, they pick a single scalar number representing the behavior they want to explain. For image classification this is normally the top class logit, but they also support the sum of the top-k logits and the entropy of the output distribution. For the retina model they use surprisal, I(x) = −log₂P(x), which reflects how unexpected a response is and varies with the Mahalanobis distance of the population response from its mean.

Second, they compute each hidden unit's contribution to that scalar using gradient-based attribution. They compare ActGrad (the element-wise product of activation and gradient), Integrated Gradients, and SmoothGrad, and settle on Integrated Gradients to the top logit with 10 integration steps as the standard. Integrated Gradients has a completeness property: contributions sum to the scalar target, so spatially summing a channel's contributions yields its net effect on the prediction — one number per channel.

Third, they stack these channel contributions into a matrix of channels by images (50,000 ImageNet validation images for ResNet-50) and decompose it with a sparse autoencoder. The autoencoder has an encoder mapping contributions into k dimensions and a non-negative dictionary of k modes of dimension d, where k is typically an expansion factor N times d. Sparsity comes from hard thresholding at a cutoff, and training minimizes the squared reconstruction error, optionally with L1 regularization. Each mode is a pattern of channel weightings — a coordinated way that a group of channels acts together — and each image gets a loading value per mode.

Fourth, to see what input features a mode is using, they trace gradients from output back to input but only through the channels belonging to that mode, producing a sensitivity map A, which they multiply element-wise by the input to get a contribution map C. Summing that map recovers the mode's contribution.

They validate the approach by perturbing networks — removing (ablation) or retaining (preservation) the top-weighted channels of a mode — and measuring classification accuracy for the target class versus off-target classes across all 1000 ImageNet classes.

Why This Matters

The paper reframes interpretability around causal influence rather than correlation with internal activations, and it argues that coordinated groups of units, not single units, are the natural unit of analysis. It provides a method that works on any trained feedforward model without access to training data or labels, and it connects artificial network interpretability to neuroscience through the shared receptive-field/projective-field framing.

Real-world applications:

• Safety and auditing of AI systems. Locating the specific internal pathways that produce a given output supports more precise inspection and intervention than activation-based analysis.
• Model debugging and repair. Ablation and preservation of contribution modes can identify which channels are necessary or sufficient for a behavior, and can restrict a model to a targeted class.
• Neuroscience hypothesis generation. Because the retina model's hidden units correlate with actual interneuron recordings, the modes CODEC identifies translate into experimentally testable predictions about which presynaptic cell types drive ganglion cell activity.
• Efficient architecture and transfer learning design. The authors suggest the sparse, interpretable motifs could serve as building blocks for more efficient architectures or transfer-learning approaches.

Industry relevance: Teams deploying vision or multimodal models can use contribution modes as a lightweight post-hoc tool for causal attribution and targeted model editing, with the reported runtimes indicating practical scale — contribution computation for ResNet-50 across all 16 layers over 50,000 images took about 4.1 hours with Integrated Gradients at 10 interpolation steps (4.7 seconds per batch of 16 images, 293 ms per image), or 1.3 hours with activation times normalized gradient (1.5 seconds per batch, 93 ms per image), on an NVIDIA RTX 6000 GPU.

Future Directions

• Improved spatial reduction for Vision Transformers. The authors note their initial approach of treating tokens as space yields weaker ablation performance than CNNs, and that the optimal strategy remains to be determined.
• Incorporating negative contributions into the mode analysis. The reported mode analyses focus exclusively on positive contributions, with extending to negative components left as future work.
• Application beyond image classification. The authors state CODEC is architecturally agnostic and could be applied to more complex models such as LLMs, but report that they have at this point only analyzed scaling with LLMs in supplemental material.
• Coverage of full networks. Some ResNet-50 analyses were limited to specific blocks rather than the entire network, which the authors list as a limitation.

Target Audience

Interpretability researchers and machine learning engineers who need causal, manipulable explanations of hidden-layer computation; computational neuroscientists studying retinal and early visual circuits who want to link model units to recorded cell types; and researchers interested in sparse-autoencoder-based analysis of neural networks, who will find the contribution-versus-activation comparison and the hyperparameter robustness sweeps most directly relevant. Readers seeking a beginner-level tutorial on neural network interpretability would need supplementary background.

Authors’ abstract

Understanding how neural networks transform inputs into outputs is crucial for interpreting and manipulating their behavior. Most existing approaches analyze internal representations by identifying hidden-layer activation patterns correlated with human-interpretable concepts. Here we take a direct approach to examine how hidden neurons act to drive network outputs. We introduce CODEC (Contribution Decomposition), a method that uses sparse autoencoders to decompose network behavior into sparse motifs of hidden-neuron contributions, revealing causal processes that cannot be determined by analyzing activations alone. Applying CODEC to benchmark image-classification networks, we find that contributions grow in sparsity and dimensionality across layers and, unexpectedly, that they progressively decorrelate positive and negative effects on network outputs. We further show that decomposing contributions into sparse modes enables greater control and interpretation of intermediate layers, supporting both causal manipulations of network output and human-interpretable visualizations of distinct image components that combine to drive that output. Finally, by analyzing state-of-the-art models of neural activity in the vertebrate retina, we demonstrate that CODEC uncovers combinatorial actions of model interneurons and identifies the sources of dynamic receptive fields. Overall, CODEC provides a rich and interpretable framework for understanding how nonlinear computations evolve across hierarchical layers, establishing contribution modes as an informative unit of analysis for mechanistic insights into artificial neural networks.