When a Conceptor Complement Becomes Gain: Operator Geometry and Behavioral Attribution in Language Models

AI summary

Overview

Research area: Mechanistic interpretability and activation steering in large language models — specifically, how to attribute a behavioral change to the individual components of a composed steering operator.

Technical level: Advanced. The paper relies on covariance eigendecompositions, symmetric conceptor operators, operator gain bounds, invertibility arguments, and bootstrap-based behavioral statistics.

Scope: One-sentence scope: The paper re-derives the complete scaled conceptor-complement transformation used for activation steering, shows analytically that at the tested operating point (α = .05, β = 2) every fitted map expands every direction, and then tests empirically whether the fitted covariance matrix contributes anything behavioral beyond the target-specific center and gain it is composed with.

What This Paper Is About

Activation steering modifies a language model's hidden states at inference time without changing weights. When a steering intervention works, it is tempting to credit the learned matrix inside it. But the deployed transformation also contains a target-specific reference point (a mean, m_t) and a global gain (β), so a whole-intervention comparison against the unmodified model cannot say which piece deserves credit. This paper isolates the fitted conceptor matrix C_t by removing it while holding the center, gain, layer, prompts, seeds, decoding, and response budget fixed, then measures both the activation-space geometry and the generated behavior that remain.

Key Contributions

1. A component-attribution protocol for composed steering operators. The authors define a matched matrix-removed control, F_0,t(h) = m_t + β(h − m_t), which sets only C_t to zero and preserves the target-specific center. The activation-space difference between the deployed map and this control is exactly −βC_t(h − m_t), isolating the matrix term at a common input rather than comparing two independently evolving generation trajectories.

2. A complete-map spectral analysis showing the tested regime is purely expansive. Expanding Eq. (1) yields a complete per-direction gain g_i(α, β) = β(1 − c_i) = β/(1 + α²λ_i), with attenuation requiring c_i > 1 − 1/β, i.e. c_i > 0.5 at β = 2. At α = .05 and β = 2, all 21 generation-layer maps (3 models × 7 targets) expand every direction around their target centers.

3. A matched behavioral panel testing the matrix's incremental output effect. A 1,920-response study crossing three models, four targets, twenty questions, two replicates, and four conditions yields 480 paired complete-versus-control comparisons — the first direct test of whether retaining C_t adds target suppression once the center and gain are already present.

4. A separation of soft filtering from linear erasure. The paper proves that a finite-aperture complement is invertible and that an unrestricted affine readout can preserve its scores exactly after the edit, then contrasts this with LEACE and INLP, which target linear accessibility directly.

Main Findings

• Every tested map expands, not shrinks. At α = .05 and β = 2, the largest conceptor eigenvalue across the seven-target panel is 0.170124 (Evil in Qwen2.5-1.5B-Instruct, which is outside the four-target generation panel), and the minimum complete gain, computed before rounding, is 1.659751. All twelve matched maps also satisfy the expansion bound.

• The matrix correction is small on held-out activations. On question-held-out target-present extraction activations, the fitted matrix correction spans 0.093% to 7.815% of the matched-control displacement across twelve matched settings.

• The regime is parameter-dependent, not a property of representation geometry alone. A sweep over α ∈ {.025, .05, .1, .2, .5, 1, 2} at β = 2 leaves all 21 maps expansive at α = .1; at α = .2 four maps have at least one attenuated direction; at α = 2 all 21 do. Held-out correction ranges rise to 1.459%–55.470% at α = .2 and 52.998%–129.875% at α = 2. Because the denominator is the matrix-removed displacement, values above 100% are possible, and the ratio is bounded by 200%.

• Retaining the matrix changes many texts but does not establish a consistent benefit. Target-expression contrasts on the four-target panel range from −4.75 to 5.25 points, and all twelve 95% question-cluster bootstrap intervals include zero. Corrected-coherence contrasts range from −2.125 to 1.875 points, also with all twelve intervals including zero. Only 195 of 480 paired texts are identical; 285 of 480 differ.

• Coherence measurement is ceiling-saturated. 872 of 960 corrected-coherence scores reach the top anchor, and seven zero-width coherence intervals reflect all-100 scores. The authors state explicitly that zero does not imply identical responses and that the estimates do not establish equivalence.

• Erasure reduces freshly fitted probe accuracy, soft filtering does not. Averaged over three models and seven targets under question-grouped splits, raw and finite-complement balanced accuracy/AUROC are both .893/.928. Rank-1 LEACE reduces this to .663/.730 versus .893/.927 for a random rank-1 projection; rank-20 INLP reaches .669/.752 versus .891/.927 for random rank 20. Held-out-template splits preserve the ordering.

• Invertibility is the mechanism behind the soft-filter result. At finite positive aperture, A = β(I − C) has all positive eigenvalues and is invertible, so every affine score wᵀh + b has an equivalent form in the transformed state. Finite-sample accessibility still depends on the fitted probe, regularization, and data split.

• Refusal and length effects are negligible in the matched panel. The complete map produces 11 refusals and the control 12; ten pairs refuse in both arms, one only under the complete map, two only under the control, and 467 in neither (pooled exact McNemar p = 1). Four of the 1,920 responses reach the 1,024-token cap; three occur in the matched β = 2 arms across two Qwen2.5-1.5B-Instruct Hallucination pairs. Removing the two affected pairs leaves the target-expression contrast at 0.00 and changes corrected coherence from 1.875 to 1.375 points.

• Evaluator sensitivity is real but does not rescue the attribution. Across 227 jointly numeric target-expression responses, primary–secondary agreement is Spearman ρ = .891 and absolute-agreement ICC = .900, with mean absolute difference 10.29 points; across 240 coherence responses these are .644, .737, and 6.81, with refusal agreement 240/240. Some model–target effect directions change with the evaluator: Qwen-1.5B Sycophancy moves from +18 with Gemini to −10 with DeepSeek, though Qwen-7B stays negative (−10 versus −6).

• Statistical tests are uniformly non-significant. All archived full-response Holm-adjusted values equal 1.000 in a 24-test family per model, and the separately frozen family of twelve full-response corrected-coherence contrasts also yields all adjusted values of 1.000. For Qwen-1.5B Hallucination, all 35 eligible paired target-expression differences are zero.

• A descriptive geometry mismatch between layers. At the separate diagnostic layers 26/26/30, paired-difference conceptors have higher Shannon effective rank and more directions accounting for 90% of conceptor trace in all 21 comparisons. The authors present this as motivation for testing contrast-based interventions, not as evidence that such interventions work, and not as an explanation of the matched generation result at layers 9/9/10.

Methodology in Plain English

The researchers start from a standard steering recipe. They fit a conceptor C_t to the covariance of activations collected while a model exhibits a target behavior, then deploy the map F(h) = m_t + β(I − C_t)(h − m_t), where m_t is the average target-present activation and β is a scale factor.

Their key move is to build a control that removes only C_t. The control keeps the same target mean, the same gain, the same layer, the same prompts, the same random seeds, and the same decoding settings. Because both maps are evaluated at the same input activation, the difference between them is exactly the matrix term — no confound from comparing two separately generated trajectories. They then estimate how large that term is on activations from questions the fit never saw, using five folds: each fold fits the center and matrix on 80 rows and evaluates on 20 held-out rows.

Next they check the mathematics of the complete map. Since C_t's eigenvalues lie between 0 and 1, the deployed map multiplies each covariance direction by β(1 − c_i). At β = 2, this only shrinks a direction when c_i exceeds 0.5. They compute the actual spectra for all 21 model–target combinations and find the largest c_i is 0.170124, well below that threshold — so everything expands. Repeating the calculation across seven aperture values shows when that conclusion changes.

Finally they run a large generation experiment: three models, four targets, twenty questions, two sampling replicates, and four conditions, giving 1,920 responses and 480 paired comparisons. Two automated judges score the outputs, and a separate study tests whether LEACE and INLP can remove the target from an activation while a freshly fitted probe tries to recover it.

Why This Matters

Impact on research. The paper makes a methodological point with teeth: showing that a composed intervention changes behavior is not evidence that its learned matrix caused the change. This matters for any interpretability claim built on "we steered the model, therefore we found the mechanism." The paper also elevates a mathematical observation — that a conceptor complement composed with gain β = 2 is an expanding map at the inherited aperture — into a concrete operating-regime warning, since the α = .05, β = 2 setting was inherited from a GPT-J configuration in Postmus and Abreu (2024) rather than optimized for these models and targets.

Real-world applications:

• Safety and behavior steering: Practitioners who adjust model traits such as sycophancy or hallucination tendency can use the matched-control design to check whether a fitted matrix contributes anything, instead of assuming the composed pipeline's effect belongs to it.
• Interpretability audits: Teams validating claims that a specific direction or subspace encodes a behavior get a template for holding shared components fixed and measuring output consequences directly.
• Privacy and fairness erasure: The LEACE/INLP comparison shows why invertible soft filters and dedicated erasure methods should be evaluated differently — reduced measured accessibility from a fresh probe is not the same claim as information removal.
• Evaluation design: The coherence ceiling (872 of 960 scores at the top anchor) and the evaluator disagreement results illustrate why behavioral endpoints need headroom and sensitivity checks before conclusions are drawn.

Industry relevance. Teams shipping steerable or controllable language models need to know which knob does the work. Reporting the complete transformation delivered to the model, removing the component whose contribution is claimed while preserving the others, and measuring the resulting behavior is a cheap and reproducible audit step, and the paper shows it can be run at model scales of 1.5B to 8B without retraining.

Future Directions

1. Matched behavioral comparisons at larger apertures. The aperture sweep changes the geometric regime substantially — at α = .2 the held-out correction rises to 1.459%–55.470%, and at α = 2 to 52.998%–129.875% — but the paper states that matched generations at larger apertures remain unperformed. This is the most direct open experiment.

2. Testing contrast-based interventions. The paired-difference conceptors at layers 26/26/30 show higher Shannon effective rank and more trace-covering directions in all 21 comparisons, but the paper explicitly says this is not evidence that a contrast-based intervention works. Whether it does is untested.

3. Extending beyond the exploratory four-target panel. The matched panel covers Sycophancy, Hallucination, Optimistic, and Apathetic, chosen from the earlier seven-target 200-token study; the paper notes it is not a representative or preregistered sample of all targets. Efficient is also excluded from the generation panel despite having the largest fitted eigenvalue.

4. Human validation and evaluator robustness. Neither automated evaluator has human validation, the corrected coherence evaluation shows ceiling saturation and occasional disagreement on identical inputs, and the coherence rescore changes both judge and rubric relative to Gemini, so their differences do not isolate a rubric effect. A prospectively designed human study, and a comparison that varies judge and rubric independently, would address both gaps.

5. Component matching for other operator families. The paper suggests the same requirement applies when rotations, transport maps, or learned subspaces are composed with other operations — a natural extension of the attribution protocol beyond conceptor complements.

Target Audience

This paper is most valuable to interpretability and representation-engineering researchers who design or evaluate activation-steering interventions, and to methodologists interested in causal attribution for composed machine-learning operators. It also speaks directly to safety and alignment engineers who deploy steering as a practical control mechanism and need to know which component is responsible for the behavioral change they observe. Readers will need comfort with eigendecompositions, operator norms, and bootstrap confidence intervals; the paper is not an introductory treatment, but its central message — remove the component you want to credit, keep everything else fixed, and measure the output — is legible without the linear algebra.

Authors’ abstract

When a steering intervention changes a language model's behavior, which part of the intervention deserves the credit? We test this question for a target-centered, scaled conceptor complement by removing its fitted covariance filter while preserving the same target-specific center and gain. At the original aperture $α=.05$ and gain $β=2$, all 21 fitted maps across three models and seven targets expand every direction at the generation intervention layers. The matrix correction is $0.093\%$--$7.815\%$ of the matched-control displacement on held-out extraction activations. In 480 paired comparisons within a 1,920-response study, retaining the matrix changes many individual texts but does not establish a consistent additional target-suppression benefit on the four-target panel. A geometric aperture sweep shows that the operating regime changes with the filter setting. Separately, an invertible soft filter preserves what an unrestricted affine readout can recover, whereas dedicated erasure methods reduce prediction by newly fitted probes. Together, the results show why a whole-intervention comparison cannot by itself attribute generated behavior to the fitted matrix: analyze the complete transformation, hold shared components fixed, and test its output consequences directly.