From Isolation to Entanglement: When Do Interpretability Methods Identify and Disentangle Known Concepts?

AI summary

Overview

Research area: Mechanistic interpretability and causal representation learning (CRL), specifically the evaluation of feature-discovery ("featurization") methods such as sparse autoencoders (SAEs) and linear probes.

Technical level: Advanced. The paper assumes familiarity with identifiability theory, sparse autoencoders, and correlational disentanglement metrics (MCC, DCI-ES).

One-sentence scope: The paper builds a multi-concept dataset and evaluation framework to test whether interpretability methods that look disentangled by correlational metrics actually allow selective, independent control of individual concepts when features are steered.

What This Paper Is About

Interpretability research assumes that if a featurizer (e.g., an SAE or probe) assigns a distinct feature to each concept, then manipulating that feature should change only that concept. Most prior evaluations test one concept at a time and rely on correlational metrics, leaving open whether concepts are truly disentangled from one another. This paper asks whether representational disentanglement actually guarantees independent, selective manipulability, using a dataset where every sentence carries four ground-truth labels (voice, tense, sentiment, domain) and where correlations between those labels can be tuned.

Key Contributions

1. A multi-concept evaluation framework and dataset. The authors use a probabilistic context-free grammar (PCFG) to generate a training set of 382,459 sentences and a test set of 1,007 sentences, each labeled with four concepts (voice, tense, sentiment, domain), with adjustable correlation between concept values (e.g., positive sentiment and the science domain).
2. Correlational evidence that common featurizers look disentangled but do not predict steering selectivity. ReLU SAEs, Top-K SAEs, SpADE SAEs, SSAEs, and sparse probes score highly on MCC and on the Disentanglement, Informativeness, and Explicitness components of DCI-ES, yet this does not predict how selectively their features can be steered.
3. New metrics for causal entanglement in steering. The authors introduce concept independence (whether a concept is influenced only by its own top feature) and feature selectivity (whether a feature influences only its own concept), computed from change in log-odds under steering.
4. A distinction between disjointness and selective steerability. They show that features can be effectively disjoint (their effects are additive, with almost no interaction) even though steering one feature still moves many unrelated concepts.

Main Findings

• Probes outperform SAEs on identification. On MCC under varying correlations, linear probes significantly outperform SAEs up to correlations of 0.9, and the margin is generally substantial. The authors note that if the target concepts are known in advance, supervised methods should be preferred.
• SSAEs and Top-K SAEs are the strongest unsupervised featurizers. SSAEs perform best or close to best among SAEs, as hypothesized, because sparse representations of shifts between concepts are provably identifiable; Top-K SAEs also perform well but underperform for sentiment=positive.
• Synthetic data can match large natural-language corpora for locating concepts. SAEs trained on the paper's generated data perform comparably to publicly available SAEs trained on much larger natural-language data (Marks et al. (2025) ReLU SAEs for Pythia-70M, GemmaScope JumpReLU SAEs of Lieberum et al. (2024) for Gemma-2-2B), with SSAEs outperforming both for Pythia-70M but not for Gemma-2-2B. The SAE architecture matters significantly.
• Correlations degrade identification for most methods, except SSAEs. Probes and SpADE maintain consistent MCCs up to correlations of 0.5 between concept pairs; beyond that, representations degrade. SSAEs remain consistent up to complete correlations of 1.0, as their theory predicts. The overall implication is that with an optimal architecture, one may not need to worry about spurious correlates unless the correlation is near-complete.
• Disentanglement and informativeness are high; completeness is low. Across all SAE architectures, Disentanglement, Informativeness and Explicitness are high (but not for the original model representation space), while Completeness is low. This indicates most features are sensitive to only one concept, but each concept is spread across many features — i.e., feature splitting, not concept entanglement.
• Feature splitting is quantifiable. Using k-sparse probes to find how many features are needed before probe performance saturates, the authors report this generally requires 10 features.
• Steering is not selective, even in idealized settings. Heatmaps of ΔLogOdds show the expected diagonal (steering a concept's feature raises that concept's log-odds) but also significant effects on many unrelated concepts. Increasing dataset correlations up to 0.9 produces qualitatively similar entanglement patterns. The authors use a default steering coefficient α of 5.
• Mean independence and selectivity are low, but best cases are high. In Table 1, mean concept independence and feature selectivity are generally low; for example, ReLU SAEs at ρ=0.1 yield independence 0.32 and selectivity 0.29 for Pythia-70M, and 0.24 and 0.25 for Gemma-2-2B. Maxima are much higher, indicating that at least one concept is selectively recovered by each architecture.
• Features are nearly perfectly disjoint. Stepping two features in the same forward pass gives a change in ΔLogOdds almost exactly equal to the sum of steering each separately. R² between predicted and actual ΔLogOdds is near 1.00 in Table 2 for ReLU, Top-K, and SSAE architectures on both models, at both ρ=0.1 and ρ=0.9.
• Disjointness does not imply independence. Because interaction effects are nearly absent, feature interactions do not explain the non-selective steering. The authors conclude that demonstrating two features operate in separate spaces is insufficient to claim they are selective for one concept, and that correlational metrics are insufficient to establish steering selectivity.

Methodology in Plain English

The researchers wanted a test bed where the "right answer" is known, so they used a probabilistic context-free grammar to synthesize sentences and label every one with four concepts: voice, tense, sentiment, and domain. By upsampling examples where specific concept values co-occur, they can dial the correlation between concepts up or down while training.

They then take two frozen language models — Pythia-70M and Gemma-2-2B — and apply several featurizers to their activations: supervised linear probes, plus SAEs of four kinds (ReLU, Top-K, SpADE, SSAE), plus baselines including a randomly initialized SAE, the raw activations ("Neuron"), and SAEs pretrained on natural language.

They first score these featurizers with standard correlational metrics from causal representation learning — the mean correlation coefficient (MCC) and DCI-ES — which ask how well each concept can be recovered from the features, whether a feature is selective for a single concept, and whether a concept is spread across many features.

Then they move from correlation to intervention. For each concept, they pick the feature most correlated with it and set that feature's activation to a multiple of its maximum value on the training data, then decode back into the model (adding back the reconstruction error so changes are attributable to steering and not reconstruction artifacts). They measure the change in log-odds of every concept using a multinomial probe on the final layer. From this they define independence (does steering other concepts' features affect this concept?) and selectivity (does steering this concept's feature affect other concepts?). Finally, they test disjointness by comparing the effect of steering two features in one forward pass against the sum of steering them separately, using steering coefficients of 0.1, 0.5, 1.0, 2.0 and 5.0.

Why This Matters

Impact on research. The paper argues that single-concept evaluations and correlational disentanglement metrics can create a false sense of confidence: a method can score high on MCC and DCI-ES while its features are not selectively steerable. It also sharpens a conceptual distinction — disjointness versus independence — that the interpretability literature often conflates, and recommends separating correlational from interventional evidence in featurizer evaluation.

Real-world applications:

• Model editing and behavioral control. Anyone editing a model's tone, domain, or stylistic properties via feature steering needs to know whether the edit will leak into unrelated behaviors.
• Safety and alignment auditing. Detecting a harmful property is not the same as being able to suppress it selectively; this work gives a method for measuring that gap.
• Reliability of interpretability tooling. Teams choosing between SAEs and simple probes now have evidence that architecture choice, not just dataset size, drives concept recovery.
• Benchmark design. The PCFG-based dataset with tunable concept correlations offers a template for controllable evaluation sets where ground-truth factors are known.

Industry relevance. Organizations deploying SAE-based monitoring or steering in products depend on claims of "one feature, one concept." The paper's finding that mean independence and selectivity are low — despite near-perfect disjointness — is directly relevant to whether such controls are safe to compose, and the recommendation that sufficiently large natural-language SAEs match task-specific SAEs speaks to practical training-cost tradeoffs.

Future Directions

• Closing the selectivity gap. Mean independence and selectivity are low but maxima are high; understanding why a subset of concepts is selectively manipulable while others are not is an open question.
• Locating steering-relevant features. The authors cite Arad et al. (2025), who find that features that detect a concept and features that control it are nearly disjoint; they test gradient attributions to locate features as an alternative, but this line of work is not settled.
• Better objectives for disentanglement. The results suggest current featurization objectives may be optimizing the wrong notion of concept separation, raising the question of what training objective would produce non-interacting features that are also selective.
• Extending multi-concept evaluation. The paper explicitly calls for multi-concept evaluation settings to become standard, which implies porting this framework to more concepts, other modalities, and larger models.

Target Audience

Interpretability and causal representation learning researchers; practitioners who use SAEs or probes for model monitoring, editing, or steering; and evaluation/benchmark designers who need ground-truth concept labels with controllable correlations. Readers unfamiliar with identifiability theory or metrics like MCC and DCI-ES will need background reading, since the paper is written at an advanced technical level.

Authors’ abstract

A goal of interpretability is to recover disentangled representations of latent concepts (features) from the activations of neural networks. The quality of features is typically evaluated in isolation, and under implicit independence assumptions that may not hold in practice. Thus, it is unclear to what extent common featurization methods such as sparse autoencoders (SAEs) and probes disentangle one concept from another. We propose a multi-concept evaluation setting using concepts including sentiment, domain, voice, and tense. We evaluate how well featurizers produce disentangled representations of each concept, observing that features are typically sensitive to only one concept, but also that concepts are distributed across many features. Then, we steer these features, measuring whether each concept is independently manipulable, and whether features interact. Even in idealized settings, steering a feature often affects many concepts, despite a near absence of interaction effects. These results suggest that correlational metrics are insufficient to establish steering selectivity, and that demonstrating that two features operate in separate spaces is insufficient to claim that they will be selective for one concept. These results underscore the importance of multi-concept evaluations in interpretability research.