Superposition as Lossy Compression: Measure with Sparse Autoencoders and Connect to Adversarial Vulnerability

AI summary

Overview

Research area: Mechanistic interpretability and neural representation learning, combining information theory, sparse autoencoders, and adversarial robustness.

Technical level: Intermediate. The core idea (entropy of sparse-autoencoder activations) is simple, but comfortable reading requires familiarity with sparse autoencoders, the superposition hypothesis, and adversarial training.

Scope: The paper proposes an information-theoretic metric for measuring superposition in neural networks, validates it on toy models, and applies it to dropout, algorithmic tasks, grokking, a language model, and adversarially trained classifiers.

Note: the provided paper content is truncated partway through Section 7 (it ends mid-sentence in the description of the statistical methodology). Findings from the remainder of Section 7 and from later sections are therefore not reported here.

What This Paper Is About

Neural networks often represent more features than they have neurons by storing features as overlapping directions in activation space, a phenomenon called superposition. Existing ways to measure superposition rely on knowing the ground-truth feature-to-neuron mapping, which is only available in toy settings. The paper's goal is to define a measurable quantity for superposition that works on real networks, and then to use it to test whether superposition actually causes adversarial vulnerability as previously hypothesized.

Key Contributions

1. An information-theoretic definition of superposition as lossy compression. The number of effective features is defined as F = e^{H(p)}, the exponential of the Shannon entropy of a sparse-autoencoder-derived feature usage distribution, where each feature's probability is its share of total activation magnitude across samples. Superposition is then ψ = F/N, effective features per physical neuron, with ψ = 1 as the interference-free (lossless) limit and ψ > 1 as lossy compression.
2. Validation on toy models and dictionary-scaling analysis. The activation-based measure recovers reference superposition levels through an SAE pipeline at r = 0.94 ± 0.02, whereas the weight-based Frobenius baseline used by Elhage et al. fails on SAE weights. A separate experiment shows the measure plateaus as SAE dictionaries grow, given sufficient ℓ1 regularization.
3. Exploratory applications across four settings: dropout, algorithmic tasks compiled with Tracr, grokking on modular arithmetic, and layer-wise analysis of Pythia-70M.
4. A test of the superposition-vulnerability hypothesis. Systematic PGD adversarial training across MLPs, CNNs, and ResNet-18 on MNIST, Fashion-MNIST, and CIFAR-10 shows adversarial training can either expand or reduce effective features, contradicting the prediction that it should universally reduce superposition.

Main Findings

• The metric tracks ground truth where superposition is observable. With 100 toy models (sparsity S ∈ [0.001, 0.999], 20 features compressed through a 5-neuron bottleneck, importance weights ω_i = 0.7^i, 10,000 activation samples, SAEs with 40-dimensional dictionaries at 8× expansion and ℓ1 coefficient 0.1), both the entropy measure and the Frobenius baseline correlate with toy weights at r = 0.99 ± 0.01. Only the activation-based measure survives the SAE pipeline (r = 0.94 ± 0.02); the Frobenius norm on SAE weights produces incorrect scales of 0.1–0.7 versus the expected 1–4.
• Stable across hyperparameters, with identifiable failure modes. Performance is stable across ℓ1 strength (10⁻³ to 10¹), model scale (8–32 input dimensions), and dictionary expansion (2× to 32×). Degradation occurs at extreme regularization (ℓ1 = 10), undersized dictionaries, toy models too small or too large to train reliably, and very large dictionaries that permit feature splitting.
• Measurements converge as dictionaries grow, under adequate regularization. On multi-task sparse parity (3 tasks, 4 bits each; 64 hidden neurons; dictionary scales 0.5× to 16× hidden dimension; ℓ1 from 0.01 to 10.0), feature counts plateau for ℓ1 ≥ 0.1, while weak regularization (ℓ1 = 0.01) shows unbounded growth through feature splitting, and ℓ1 = 10.0 suppresses features entirely.
• Dropout reduces effective features by up to 50% across hidden dimensions h ∈ {16, 32, 64, 128} and dropout rates from 0.0 to 0.9. The paper argues that dropout's redundant encoding consumes capacity, so the network prunes features rather than adding superposition—larger networks show reduced sensitivity, narrow networks show sharp reduction.
• Algorithmic tasks resist genuine superposition. Tracr-compiled sequence reversal ("123" → "321") and sorting ("213" → "123"), plus transformers trained from scratch, converge to approximately 12 features for reversal and approximately 10 for sorting, far below the original 45D compiled dimension for reversal. As compression is applied, ψ rises from about 0.3 toward ψ = 1, but never beyond: further compression causes performance degradation and then collapse rather than ψ > 1.
• Feature counts detect the grokking transition. On a two-path MLP trained on modular arithmetic (a + b) mod 53, feature count shows sharp consolidation at the generalization transition (epoch 60), while the Local Learning Coefficient decays more smoothly over training. Feature count correlates with LLC at r = 0.908, p < 0.001.
• Layer-wise organization in Pythia-70M is non-monotonic. Using pretrained SAEs from Marks et al., measurements saturate after 2×10⁴ samples. MLP layer 1 peaks at roughly 10,000 features (20× the neuron count); of the 21,000 SAE features that activate for MLP 1, the entropy measure yields 5,600 effective features. MLPs hold the most features, residual streams next, attention the least, and the trajectory peaks in early-middle layers before declining—mirroring intrinsic dimensionality studies.
• Adversarial training's effect depends on task complexity and capacity. Across PGD adversarial training with MLPs (hidden dimensions 8–512), CNNs (8–64 filters), and ResNet-18 (1/4× to 2× width), on MNIST, Fashion-MNIST, and CIFAR-10 with 2, 3, 5, and 10 classes (SAEs at 4× dictionary expansion, ℓ1 = 0.1, ε ∈ {0.0, 0.1, 0.2, 0.3}, three seeds), simple tasks with ample capacity show feature expansion (abundance), while complex tasks or limited capacity show reduction below baseline (scarcity). Fashion-MNIST amplifies reduction relative to MNIST. The paper states the formal statistical methodology beyond this point is cut off in the provided content.

Methodology in Plain English

The authors start from a simple accounting question: if a network's features were given their own dedicated neurons, how many neurons would it take to carry the same information without any interference? That number is the "effective feature" count.

To get it, they first run each layer's activations through a sparse autoencoder, which decomposes tangled neural activations into a larger set of sparse dictionary elements (candidate features). They then compute each feature's share of total activation magnitude across many samples, giving a probability distribution over features. The exponential of the Shannon entropy of that distribution is F, the number of interference-free channels needed to carry the same feature usage. Dividing by the layer's actual neuron count N gives ψ, the compression ratio.

They validate the approach in two stages on toy models where the true interference structure is visible in the weight matrix: first checking that both their metric and the existing weight-based metric track the reference values, then checking which metric still works when only SAE outputs are available. They then apply the metric unchanged across dropout experiments, Tracr-compiled algorithmic tasks with progressive dimensional compression, a grokking setup compared against the Local Learning Coefficient, a pretrained language model with off-the-shelf SAEs, and adversarially trained classifiers. For convolutional layers, spatial positions are treated as independent samples so that superposition is measured across channels.

Why This Matters

Impact on research. Superposition has been a central theoretical concept in mechanistic interpretability but has lacked a practical measurement that does not require known ground-truth features. This paper supplies one, and its adversarial robustness experiments undercut the widely repeated assumption that superposition is the cause of adversarial vulnerability. It also connects superposition measurement to other complexity measures (local learning coefficient, intrinsic dimensionality), suggesting a shared quantitative language for representation analysis.

Real-world applications:

• Auditing trained models for how much representational compression they perform, layer by layer, without access to internal ground truth.
• Monitoring training runs for phase transitions such as grokking, using feature consolidation as an early signal of algorithmic discovery.
• Informing architecture and regularization choices (width, dropout, dictionary size) with a measurable view of capacity pressure.
• Informing adversarial robustness practice by showing that robustness interventions do not map onto a single direction of representational change.

Industry relevance. The framework uses standard sparse autoencoders and applies to MLPs, CNNs, and ResNet-18 on common datasets, so it can be layered onto existing interpretability tooling. For teams deploying safety-critical or adversarially exposed models, the result that adversarial training can either expand or reduce features depending on task complexity and capacity means robustness evaluations cannot assume a fixed relationship between compression and vulnerability.

Future Directions

• Establish the true direction of the superposition-robustness relationship. The abundance-versus-scarcity bifurcation needs a fuller statistical account than the truncated Section 7 provides, and testing on larger datasets and architectures beyond those listed.
• Resolve absolute versus comparative measurement. Because absolute feature counts shift with SAE dictionary size, the authors recommend comparative interpretation; whether the F ≈ N tracking in constrained algorithmic settings can be extended to meaningful absolute counts elsewhere remains open.
• Connect the measure to other complexity metrics. The correlation with LLC (r = 0.908) and the parallel with intrinsic dimensionality suggest a broader theory linking feature count, geometric complexity, and learning dynamics.
• Extend beyond vanilla SAEs and current task families. The framework is stated to be architecture-agnostic and compatible with gated, TopK, and Matryoshka SAEs, but all reported experiments use vanilla SAEs, and generalization to other model scales and modalities is not reported.

Target Audience

Researchers and graduate students in mechanistic interpretability, representation learning, and adversarial robustness who want a quantitative handle on superposition; practitioners applying sparse autoencoders to real models; and readers interested in information-theoretic accounts of how networks allocate limited representational capacity.

Authors’ abstract

Neural networks achieve remarkable performance through superposition: encoding multiple features as overlapping directions in activation space rather than dedicating individual neurons to each feature. This challenges interpretability, yet we lack principled methods to measure superposition. We present an information-theoretic framework measuring a neural representation's effective degrees of freedom. We apply Shannon entropy to sparse autoencoder activations to compute the number of effective features as the minimum neurons needed for interference-free encoding. Equivalently, this measures how many "virtual neurons" the network simulates through superposition. When networks encode more effective features than actual neurons, they must accept interference as the price of compression. Our metric strongly correlates with ground truth in toy models, detects minimal superposition in algorithmic tasks, and reveals systematic reduction under dropout. Layer-wise patterns mirror intrinsic dimensionality studies on Pythia-70M. The metric also captures developmental dynamics, detecting sharp feature consolidation during grokking. Surprisingly, adversarial training can increase effective features while improving robustness, contradicting the hypothesis that superposition causes vulnerability. Instead, the effect depends on task complexity and network capacity: simple tasks with ample capacity allow feature expansion (abundance regime), while complex tasks or limited capacity force reduction (scarcity regime). By defining superposition as lossy compression, this work enables principled measurement of how neural networks organize information under computational constraints, connecting superposition to adversarial robustness.