Behavioral Compositionality of Sparse Autoencoder Representations

AI summary

Overview

Research area: Mechanistic interpretability and representation engineering for large language models, specifically the evaluation of sparse autoencoder (SAE) features.

Technical level: Advanced. The core idea is intuitive, but the paper formalizes it with an orthogonal (Walsh) decomposition of set functions, projection-based variance residuals, and a proof bounding coalition-ordering error.

Scope: The paper introduces SAE Additivity Residuals (SAR), a framework for measuring whether the downstream behavioral effects of individual SAE features add up when those features are combined, and applies it across Gemma-3 4B, Mistral-7B, and Llama-3.1 8B on six behavioral readouts.

What This Paper Is About

Sparse autoencoders break a model's internal activations into sparse, linearly decoded features, and practitioners routinely assume that knowing what each feature does tells them what a group of features does together. The paper tests that assumption directly: it turns selected SAE features on and off in controlled subsets, pushes the patched activation through the frozen upper layers, and measures how much of the resulting behavioral variation cannot be explained by individual feature effects alone. The goal is to quantify how "composable" SAE coordinates actually are, and whether adding pairwise interactions fixes what first-order explanations miss.

Key Contributions

1. A formal definition of behavioral compositionality for SAE representations. The authors define SAE Additivity Residuals (SAR), the fraction of behavioral variance unexplained by the best approximation of a given interaction order, and prove that under independent feature-subset sampling, population SAR is exactly the normalized energy of interaction terms above that order.

2. A coordinate-choice control using sum-preserving rotations. By comparing learned SAE coordinates against orthogonal rotations Q of the same selected subspace (with Q𝟏 = 𝟏, so the subspace and total contribution are unchanged), the paper isolates whether compositionality depends on the SAE basis rather than only on the subspace and the downstream computation.

3. Evidence that pairwise terms recover missing structure. Pairwise models improve held-out coalition ordering in all 18 model-and-behavior settings, beat parameter-matched controls that swap in randomly selected third-order terms, and improve coalition-ordering predictions across all 20 repeated fitting/evaluation splits.

4. Extension to steering and to reconstruction quality. The analysis is carried to a combined SAE steering experiment on Llama-3.1 8B, and a matched six-SAE comparison shows that reconstruction quality (FVU) and SAR capture different variation in joint behavioral fidelity.

Main Findings

• First-order additivity is never exact. The prompt-median estimated SAR₁ is positive in all 57 model, layer, and behavior combinations examined, while affine controls remain at numerical zero. Magnitude varies by behavior: simple copying and subject-verb agreement have lower residuals than CounterFact in all three main pipelines.

• Learned SAE coordinates are more compositional than rotated alternatives. Learned coordinates show lower median SAR₁ than sum-preserving random rotations in all 18 model-and-behavior comparisons. For example, Mistral subject-verb agreement medians are 0.0218 versus 0.1196. Since the subspace, total contribution, and downstream continuation are held fixed, the gap is attributable to coordinate choice.

• Estimated SAR tracks first-order explanation quality. Moving from the lowest to the highest SAR₁ decile, rank fidelity decreases and selection regret increases in all three models. Selection regret approximately doubles in Gemma and Llama and rises roughly fivefold in Mistral. Across the 18 combinations, the prompt-level correlation between SAR₁ and rank fidelity is negative in 17 and at most −0.6 in 14.

• Pairs substantially close the gap. Pairwise models improve rank fidelity in all 18 settings, raising the average from 0.9782 to 0.9959, and satisfy ℰ̂₂ < ℰ̂₁ in all 18. Average selection regret falls from 0.0272 to 0.0051, and sign error decreases in every combination.

• Depth trends are partly coordinate-independent. Across nine early-to-late comparisons, SAR₁ decreases in eight and ρ̂₂ increases in all nine. Random rotations show the same first-order decrease in eight of nine, so part of the trend comes from shorter remaining downstream computation; nonetheless SAE coordinates have lower SAR₁ than rotations in all 33 model, behavior, and layer comparisons. Llama profiles are nonmonotonic near the final sampled layer.

• Pairwise modeling predicts steering better. With 16 published CorrSteer features in Llama-3.1 8B and 1,024 steering combinations, pairwise modeling on the 1,400-question MMLU-Pro response raises rank fidelity from 0.5874 to 0.7790, reduces ordering error from 0.1444 to 0.0606, and lowers the held-out residual estimate from 0.6424 to 0.3891.

• Reconstruction quality is not a proxy for behavioral compositionality. Pooling prompts over six SAEs at a fixed Gemma-3 1B site (n = 1,800 per behavior), SAR is positively related to sign error and selection regret in all three behaviors, while FVU is negative or near zero for sign error and near zero for selection regret.

Methodology in Plain English

The intervention. For a given prompt, the researchers take the model's activation at one layer, pick the top K = 16 active SAE features, and write each feature's contribution as its activation times its decoder direction. They then construct subsets of those contributions: keep some, delete the rest, and leave all other activation content (including the reconstruction residual) untouched. The full coalition recovers the original activation exactly. The patched activation is pushed through the frozen upper layers, and a scalar behavior score is read out. This produces a "set function" mapping feature subsets to behavior.

Measured through variance, not reconstruction. SAR_r is the squared distance between this behavior function and its best order-r approximation, divided by the behavioral variance. SAR₁ is what individual feature effects fail to explain; SAR₂ also allows pairs. Their difference Δ₂, divided by SAR₁, gives ρ₂, the pairwise share of the first-order residual. Under independent subset sampling with inclusion probability p, the authors prove this residual equals the normalized sum of squared interaction coefficients above order r, and that it upper-bounds a behavior-weighted coalition-ordering error for the optimal low-order approximation. A corollary shows SAR₁ = 0 whenever the downstream continuation is affine over the tested states.

The rotation control. To check whether the result is about the SAE basis or just the subspace, they replace the decoder matrix D_x with D_x Q for orthogonal Q satisfying Q𝟏 = 𝟏. This preserves the span and the total contribution, so any systematic SAR gap is attributable to the coordinates.

Estimation. Because exhaustive enumeration scales as 2^K, they sample coalitions: 384 per prompt, with 256 used for fitting and 128 held out for evaluation, using unregularized least squares. Held-out metrics include rank fidelity (Spearman correlation), a behavior-weighted ordering error, sign error, and selection regret. For the main analyses p = 1/2 (uniform over subsets); milder interventions at p = 0.75 and halved-contribution settings are reported as controls. Diagnostics on 2,700 prompts show every design is full rank, with a median second-order Gram condition number of 41.02 (95th percentile 49.88), and median relative split variation of 2.29% (first order) and 2.70% (second order).

Scale. Main comparisons use Gemma-3 4B, Mistral-7B, and Llama-3.1 8B across simple copying, subject-verb agreement, gendered pronoun preference, CounterFact, IOI, and greater-than comparison. Prompt counts are 1,000 (fixed-layer landscape), 100 (rotation), 800 (fidelity), and 150 (pairwise), with additional analyses spanning seven pipelines.

Why This Matters

Impact on research. SAE-based interpretability has largely been evaluated on reconstruction, sparsity, and human-judged interpretability. This paper argues that a fourth property — behavioral compositionality — is separate and must be measured, because circuits, attribution graphs, joint explanations, and steering interventions all assume that individually characterized features can be meaningfully combined. The finding that learned coordinates beat matched rotations shows compositionality is not purely a property of the subspace or the downstream network.

Real-world applications:

• Auditing and safety review: Before acting on a multi-feature explanation of unsafe or undesirable behavior, reviewers need to know whether that explanation's predictions hold when features are jointly manipulated.
• Steering and behavior control: Applications that add several SAE directions at once (e.g., to shift tone, refusal, or style) can use SAR-style measurements to predict which combinations will behave as expected.
• Model editing and debugging: Practitioners localizing a capability to a set of features can check whether the identified set actually composes, rather than assuming superposition of effects.
• Evaluation tooling: SAR provides a model-level diagnostic that can flag when behavioral effects of a representation are strongly interacting, complementing reconstruction metrics.

Industry relevance. Teams deploying SAE-based monitoring or steering in production need reliability guarantees for combinations of interventions, not just single features. The paper's steering result — rank fidelity rising from 0.5874 to 0.7790 when pairwise terms are included — indicates that first-order predictions would materially mis-rank interventions in a realistic downstream setting.

Future Directions

• Extending beyond second order. The framework is defined for any order r, but the empirical work stops at pairs; whether third- and higher-order terms carry practically important structure is not reported.
• Closing the gap between population SAR and the fitted estimate. The authors note that the comparative conclusions are stable by 224–256 fitting coalitions, but the fitted second-order residual continues to change (about 21% between 224 and 256), so a high-precision estimate of population SAR₂ remains open.
• Explaining the depth trend. Because random rotations show much of the same first-order decrease across depth, disentangling how much of the trend comes from the SAE basis versus the shrinking downstream computation is unresolved; Llama's nonmonotonic profile near the final sampled layer is also unexplained.
• Broadening models, behaviors, and intervention distributions. The authors state their conclusions depend on the selected contributions, behavioral readout, and intervention distribution rather than being a global property of an SAE dictionary, so generalization to other pipelines, readouts, and sampling schemes (e.g., p = 0.75 and halved-contribution variants) needs further study.

Target Audience

Interpretability researchers and SAE practitioners who build multi-feature explanations, circuits, or steering interventions and need to know when individual feature characterizations can be safely combined. It is also relevant to ML engineers deploying SAE-based monitoring or control, and to theoretically inclined readers interested in interaction decompositions and coalition-ordering guarantees. A background in linear algebra and basic interpretability methods is assumed; the proofs are self-contained but mathematically dense.

Authors’ abstract

Sparse autoencoders (SAEs) represent model activations with sparse, linearly decoded features, but the downstream effects of those features need not compose additively. We introduce SAE Additivity Residuals (SAR), which measure the behavioral variation left unexplained above a chosen interaction order. Under independent feature subset sampling, population SAR is exactly the normalized energy of interactions above that order and bounds coalition ordering error for the optimal low order approximation. To distinguish compositionality associated with the SAE representation from that induced by downstream computation, we compare learned SAE coordinates with sum preserving random rotations of the same selected subspace and find lower first order residuals for the learned coordinates across all 18 model and behavior settings. The coordinates are nevertheless not fully additive: pairwise terms improve held out coalition ordering in every setting and outperform parameter matched models using randomly selected third order terms. In a combined steering experiment, pairwise modeling raises rank fidelity from 0.5874 to 0.7790 and reduces ordering error from 0.1444 to 0.0606. Together, these results show that SAE coordinates support comparatively compositional behavioral descriptions while faithful joint explanations and interventions still require interactions beyond individual feature effects. Code is available at https://github.com/OpenSAE/SAE-Behavioral-Compositionality.