Complete or Sparse: A Tale of Two Identifiabilities

AI summary

Overview

Research area: Identifiability theory for neural representations, bridging nonlinear Independent Component Analysis (ICA), sparse coding and compressed sensing, mechanistic interpretability, and neuroscience. Technical level: Advanced (formal theorems and proofs, though each result is accompanied by an intuitive summary in the paper). Scope: A theoretical and empirical account of when a supervised neural encoder provably recovers the true latent variables underlying data — first up to a linear transformation, then up to permutation — and when it provably cannot.

What This Paper Is About

Neuroscience and interpretability research keep finding individual neurons that respond to single, clean concepts ("grandmother cells", the "Jennifer Aniston neuron"), yet identifiability theory for nonlinear ICA predicts only that representations recover latents up to an arbitrary linear transformation — which implies mixed selectivity (polysemanticity) rather than clean single-concept neurons. The paper asks what ingredients decide which of these two outcomes occurs, and answers with two data generating processes that differ in causal direction (labels cause latents, or latents cause labels) plus a capacity argument based on whether the number of neurons m is at least the effective latent dimension n' = rank(W*).

Key Contributions

1. Two data generating processes with opposite causal arrows. In the generative setting (Y → Z) latents are non-negative with factorized exponential conditionals and class-dependent rate parameters λ_y (with w*_y = -λ_y < 0), constant conditional entropy across classes, a uniform class prior, and rank(W*) = n requiring |Y| > n, so n' = n. In the discriminative setting (Z → Y) the true weight matrix factorizes as W* = U* A* with a measurement matrix A* ∈ R^{n'×n} and readout U*, allowing n' < n.
2. A capacity dichotomy for linear identifiability. Theorem 4.1 proves that for m ≥ n' = rank(W*) every global cross-entropy minimizer has the form h(z) = Az + ν(z) with UA = W*, rank(A) = n', and ν(z) in the null space of U; Theorem 4.2 proves that for m < n' the global optimum is generically nonlinear, so linear encoders are provably suboptimal.
3. Two routes from linear to axis-aligned codes. Theorem 4.3 shows that in the generative setting at exact capacity (m = n = n'), with non-negative activations and an energy constraint, the minimum-energy cross-entropy optimum is h(z) = D P_σ z — a permutation composed with positive component-wise scaling, i.e. one latent per neuron. Theorem 4.4 shows that in the discriminative setting, with k-sparse latents and spark(A*) > 2k, the learned encoder satisfies N(A) = N(A*) and spark(A) = spark(A*) > 2k, so latents can be recovered by sparse decoding. The authors describe this as, to their knowledge, the first identifiability result for undercomplete nonlinear ICA.
4. Empirical validation across synthetic benchmarks, vision models, language models, and macaque cortex, testing each theoretical prediction with its own metric (linear alignment, MCC-based axis alignment, KL divergence, R², row-space cosine, MIS and MMU scores).

Main Findings

• Linearity is the default when capacity suffices. In the full-capacity experiments, all configurations reached near-perfect linear alignment on both the hyperspherical and the proposed sparse DGP, confirming Theorem 4.1 regardless of DGP or output nonlinearity.
• Axis alignment needs both sparsity and non-negativity. In the stricter axis-alignment metric, only the combination of the sparse DGP with a non-negative (ReLU) output converged toward a permutation-aligned representation (slowly); the hyperspherical DGP and the linear head both retained mixed selectivity, confirming Theorem 4.3. The supplement reports that weight decay and alternative activation functions substantially accelerate axis alignment.
• A visible phase transition at m = n'. With n' = 8 and encoder dimension swept over m ∈ {2, 4, 6, 8, 10} across three latent distributions, the MLP encoder achieved lower final KL(p*‖q) than the linear encoder at every setting below the threshold, while both converged to near-zero KL at and above it — matching Theorems 4.2 and 4.1.
• Compressed sensing structure is inherited at the optimum. Training a linear encoder at full capacity while sweeping three sparsity levels, the row-space alignment cos(θ_max) between learned A and true A* undergoes a sharp transition to 1 as the KL approaches zero, confirming N(A) = N(A*).
• Only sparse decoding recovers the full latent. Supervised Lasso achieved near-perfect R² across all sparsity levels, whereas OLS plateaued at R² ≈ n'/n. Unsupervised sparse dictionary learning reached an MCC of about 0.86 under extreme (1-sparse) conditions and declined notably as k grew.
• Disentangled units are findable in other systems. Using MIS as a proxy for monosemanticity and MMU as cross-system preservation, MIS predicted MMU positively for all twelve directed pairs among four ImageNet-trained models (VGG-19, ResNet-50, ViT-b/16, DINOv2 ViT-S/14); the same pattern held, though weaker, for GPT-2, Pythia-70m, and TinyLlama on the 100,000 most frequent Wikipedia unigrams; and in macaque IT cortex (macaque "Chabo": 58 neurons; macaque "Tito": 110 neurons; 3,200 natural images) correlations were positive but weaker (Tito → Chabo r = 0.158, p = 0.0986; Chabo → Tito r = 0.135, p = 0.312).
• Limited robustness picture. Ablations find linear identifiability and axis alignment robust to moderate violations of constant conditional entropy (particularly with an affine readout) and to moderate departures from conditional independence and the exponential conditional form, but violating the non-negative latent-support assumption substantially degrades axis alignment.

Methodology in Plain English

The authors set up a standard supervised classification problem: a fixed invertible nonlinear map turns latents into inputs, a learned encoder turns inputs into m activations, and a linear readout produces class probabilities trained with cross-entropy against a log-linear true posterior. They then vary two things — which direction the causal arrow runs, and whether m is smaller than, equal to, or larger than the rank of the true weight matrix — and derive what any global optimum must look like. Extra conditions (non-negative activations plus energy minimization in one setting; sparse latents plus a spark condition in the other) are layered on to strengthen "recovered up to a linear mixture" into "recovered up to permutation." Predictions are then tested: on synthetic data with known ground-truth latents (using 10 random seeds and reporting means with standard error), and on real activations from vision models, language models, and two macaque monkeys, where ground-truth latents are unavailable and MIS serves as a proxy for disentanglement.

Why This Matters

The paper gives a single statistical account that explains three apparently conflicting empirical observations at once: dedicated monosemantic units, polysemantic superposition, and the empirical success of sparse autoencoders. It also reframes efficient-coding debates — selectivity is not the opposite of efficiency but the efficient solution under sparse, non-negative latents with sufficient capacity.

• Mechanistic interpretability: licenses sparse dictionaries (SAEs) as a principled decoding tool by showing the encoder's geometry is a compressed-sensing projector, not an arbitrary distortion — with the caveat that recovery quality depends on sparsity level.
• Neuroscience: offers a possible mechanism for grandmother-cell-like responses and a way to predict which recorded neurons should have counterparts in other animals or models.
• Model comparison and auditing: cross-system unit matching suggests that well-disentangled features carry across architectures, which is directly usable for model fingerprinting and transfer analysis.
• Efficient coding theory: reconciles dedicated units such as the frog's fly detector and the mouse's bird detector with distributed-coding intuitions.

Industry relevance: sparse-autoencoder pipelines for LLM interpretability, feature steering and auditing, model-to-model feature transfer, and evaluation of whether a representation is recoverable at all given a fixed neuron budget.

Future Directions

• Relax the most restrictive assumptions: non-negativity, constant conditional entropy across classes, and the uniform class prior; extend to signed (Laplace-distributed) latents and manifold-valued latents, where Theorem 4.3 does not apply directly.
• Systematic robustness study: the authors note that moderate entropy deviations merely blurred axis alignment in preliminary simulations but leave a full study for future work, and warn that extreme class imbalance could interact with optimization in ways the theory does not capture.
• Close the gap to finite data and finite optimization: the theory assumes global minimizers of the population loss, which real training does not guarantee.
• The existence question: whether a "true" sparse data generating process exists at all is flagged by the authors as perhaps a philosophical question that no amount of data or theory will settle.

Target Audience

Researchers in mechanistic interpretability and representation learning who want theoretical grounding for sparse autoencoders; theoretical neuroscientists working on neural coding and selectivity; and graduate students or advanced practitioners comfortable with identifiability arguments, linear algebra, and softmax/log-linear models. Readers seeking implementation guidance will find the theory more useful than the experimental protocol. Code is available at https://github.com/klindtlab/complete-or-sparse.

Authors’ abstract

Understanding when neural networks learn interpretable representations is central to mechanistic interpretability. We study this through identifiability: when can supervised learning recover the true latent variables underlying the data? We introduce two data generating processes distinguished by their causal direction where either labels cause latents (generative) or latents cause labels (discriminative). When the representation has at least as many dimensions as the latent space, we prove that the optimal encoders are linear in the latents. Conversely, when dimensions are insufficient, identifiability generically fails and encoders become nonlinear. We then show that additional constraints yield recovery of individual latent variables, not just linear mixtures. In the generative setting, non-negative activations and energy efficiency force each neuron to encode exactly one latent, providing a precise account of selective grandmother cell-like responses. In the discriminative setting, when latents are sparse and the true measurement matrix satisfies the spark condition, the learned encoder inherits compressed sensing structure, enabling sparse dictionary learning to perfectly recover individual latents. This constitutes, to our knowledge, the first identifiability result for undercomplete nonlinear independent component analysis. Our framework unifies nonlinear ICA, compressed sensing, and neural network interpretability, delineating exactly when representations transition from nonlinear distortions to linear representations and, under additional constraints or steps, to clean, axis-aligned codes. Theoretical predictions are validated on synthetic benchmarks and on activations drawn from vision models, large language models, and macaque inferotemporal cortex.