Sparse Internal Control of Language Models

AI summary

Overview

Research area: Natural language processing / mechanistic interpretability — specifically inference-time activation steering of frozen Transformer language models.

Technical level: Advanced. The paper combines Jacobian-based local sensitivity analysis, sparse support selection (an orthogonal-matching-pursuit variant), an energy-constrained refit, and a formal control certificate with a six-axis evaluation protocol.

Scope (one sentence): The paper proposes a framework, two selection algorithms, and a six-axis evaluation protocol for steering task-defined behaviors of frozen language models by additively patching a small number of internal activation sites, tested in exploratory experiments on IOI and Sentiment (GPT-2 small), MMLU Humanities/STEM multiple-choice question answering (Qwen3-4B), and Refusal (Qwen3-1.7B).

What This Paper Is About

Activation steering has shown that you can nudge a frozen model's behavior by adding directions to its internal activations, but there is little guidance on which internal sites and directions to pick, how strong the intervention should be, and how to measure whether the resulting control is any good beyond a single before/after check. This paper formalizes that choice as a sparse control problem: pick a small, costly-to-select set of site–direction pairs ("driver nodes") that move a task-specific readout toward a target while bounding collateral change on unrelated prompts and bounding the total perturbation energy. It then supplies two greedy selectors for that problem and a standardized six-axis protocol for characterizing what the resulting intervention can and cannot do.

Key Contributions

1. A formal sparse internal control problem. Selected internal activations are treated as temporary, adjustable control inputs to a frozen model, with explicit specifications for behavioral targets (via a task-specific readout), weighted sparsity (selection cost), off-target preservation (a divergence on a disjoint prompt set), and intervention energy, plus a nonlinear control certificate with tolerances (ε, δ, κ) and numerical stabilizer ξ = 10⁻⁶.
2. A local target-controllability formulation and two complementary selectors. Driver-OMP greedily adds the candidate with the largest regularized, cost-normalized residual-alignment score for reconstructing a desired behavioral displacement; Coverage-Driver instead credits candidates for first-order contribution across target-readout rows with saturating coverage, favoring stable prompt-level coverage. Both share a three-stage procedure: local linear proposal, joint strength refit on the selected support, then evaluation in the original nonlinear Transformer.
3. A six-axis control-evaluation protocol. Reachability, signed reversal, dose response, feedback controllability, off-target preservation, and held-out reuse, each with operational definitions and quantitative readouts, extending evaluation beyond one-shot target restoration.
4. Exploratory empirical study across four behaviors and three models, with code and experimental records released (github.com/zxzok/sparse-internal-control).

Main Findings

• Reachability depends heavily on direction information. At ε = 0.4, IOI Driver-OMP passes in 10/10 same-panel seeds (0.201 ± 0.058), and module-level Coverage-Driver passes in 3/3 fitting-panel seeds per category (Humanities 0.152 ± 0.154, STEM 0.061 ± 0.033). Held-out head-level MCQA and paired Sentiment share an 8/8 P (evaluation-pair directions) versus 0/8 F (training-mean directions) pattern for Coverage-Driver and Top-k in each task. Refusal passes in 0/8 seeds for all three selectors in both modes.
• Sparse supports beat random supports on nominal recovery. In the same-panel table, Random scores 0.747 ± 0.240 (IOI), 1.209 ± 0.949 (Humanities), 1.167 ± 1.141 (STEM), versus Driver-OMP at 0.201 ± 0.058, 0.279 ± 0.032, 0.175 ± 0.035 and Coverage-Driver at 0.294 ± 0.029, 0.152 ± 0.154, 0.061 ± 0.033. Nonlinear greedy, which uses additional model evaluations, reaches 0.151 ± 0.037, 0.073 ± 0.093, and 0.020 ± 0.012.
• Signed reversal tracks task and direction access, not just target movement. With a pass criterion of Reversal(β) ≥ 2: all three selectors pass in each of ten IOI P seeds; Sentiment P counts are 4/8 (Driver-OMP), 8/8 (Coverage-Driver), 8/8 (Top-k) with all F counts 0/8; across selectors, 21/24 on Humanities and 23/24 on STEM under P versus 0/24 in either F condition; Refusal has 0/24 passing method–seed cells in each mode.
• Dose response and reversal are distinct properties. Under the criterion signed Spearman ρ ≥ 0.95, Driver-OMP, Coverage-Driver, and Top-k pass 3/8, 5/8, and 2/8 P seeds on Humanities and 4/8, 6/8, and 5/8 on STEM. Under F, each STEM selector shows four strongly positive (ρ ≥ 0.95) and four strongly negative (ρ ≤ −0.95) seed curves. All 48 Refusal curves have ρ = 1, yet none passes target-side signed reversal.
• Feedback controllability separates P from F more than it separates selectors. With τ_ctrl = 0.3 for IOI and MCQA and 0.1 for Refusal, Coverage-Driver reaches 46/48 Humanities and 47/48 STEM endpoints, Top-k 45/48 and 47/48, Driver-OMP 38/48 and 39/48 under P; under F only 7–8 of 48 MCQA endpoints are reached. IOI reaches 24/24 on its same fitting/evaluation panel. Refusal reaches 12/40–22/40 endpoints across selectors and direction conditions, including ten targets already within tolerance at zero dose.
• Off-target change is substantial on most tasks. At a KL threshold of 0.001 on the shared 32-prompt panel, mean KL ranges 0.110 to 0.497 (Humanities), 0.119 to 0.582 (STEM), and 0.00553 to 0.01157 (Sentiment); none of the eight seeds satisfies either the mean or the all-prompt criterion under any selector for these tasks. Refusal Driver-OMP yields mean KL 0.001080, with 4/8 seeds meeting the mean-KL criterion and 2/8 the all-prompt criterion; Coverage-Driver and Top-k have no passing seeds.
• Reusable sites do not imply a transferable frozen intervention. In the four-seed module-level MCQA study, Driver-OMP recovers 0.992 ± 0.009 (Humanities) and 0.987 ± 0.034 (STEM) when sites are kept and coefficients refit on held-out targets, but only 0.021 ± 0.026 and 0.024 ± 0.066 when the whole intervention is frozen. Sentiment at k = 8 shows the opposite-looking pattern: F-direction interventions reach mean-margin recovery of 0.942 ± 0.535 (Coverage-Driver) and 0.937 ± 0.526 (Top-k) across eight seeds, yet none of the three selectors passes signed reversal in those F seeds.

Methodology in Plain English

The authors treat a frozen model as a fixed circuit and ask which few internal "dials" to turn. Each candidate dial is a site (a layer component, token position, and tensor slice) paired with a direction and a selection cost; turning a dial adds a multiple of that direction to the activation at that site during the forward pass.

Direct search over dials and strengths is combinatorially hard and the model's response is nonlinear, so the paper uses a three-stage workaround. First, a local proposal step: a first-order (Jacobian-based) estimate predicts how much each candidate would move the behavioral readout, producing a target dictionary and a matching preservation dictionary, and a sparse regression over those dictionaries picks a small support under a selection-cost budget and an energy budget. Second, a strength refit: once the support is fixed, coefficients are jointly re-solved on that support with an energy-constrained refit. Third, nonlinear validation: the patch is actually applied to the original Transformer and the normalized target residual, off-target loss, and energy are measured against the certificate.

The two selectors differ in what they greedily reward: Driver-OMP rewards alignment with the remaining reconstruction residual (it reduces to ordinary orthogonal matching pursuit under unit normalization, unit costs, no ridge, and no off-target rows), which suits per-prompt targets; Coverage-Driver rewards marginal increase in saturating prompt-level coverage, which favors sites that work consistently across many prompts. Experiments compare these against Top-k and, in supplementary comparisons, random supports and nonlinear greedy, under a maximum support budget of eight candidates, with two direction regimes: P (directions computed from each evaluation pair's activation contrast) and F (directions averaged over the fitting split and shared across inputs). All comparisons are run with model parameters frozen; MCQA and Refusal use plain-ridge refitting on raw activation contrasts without an independent energy constraint or off-target penalty. Each of the six evaluation axes is then measured with its own readout — normalized residual, reversal ratio, progress curves and signed Spearman ρ over a ten-dose grid, closed-loop dose adjustment against batch-mean targets, full-vocabulary KL on a 32-prompt panel, and refit-β versus frozen-β reuse.

Why This Matters

Impact on research. The paper reframes "does steering work?" as a multi-property control question. Its clearest methodological point is that nominal reachability, monotone dose response, and signed reversal can dissociate — Refusal shows perfectly ordered dose curves (ρ = 1 in all 48 curves) with zero reversal — so single-number evaluations of steering methods can be misleading. The distinction between reusable sites and transferable frozen interventions (0.992 versus 0.021 recovery on held-out MCQA) is a concrete caution for anyone reporting steering results on held-out data.

Potential real-world applications (as motivations for controllable generation, not demonstrated deployments in the paper):

• Safety-relevant behavior control, such as refusal/compliance steering, which the paper probes on Qwen3-1.7B with Arditi-inspired instruction pairs.
• Reliable decision support and multiple-choice answering, where the paper's MMLU Humanities/STEM experiments target answer-margin control.
• Sentiment and tone regulation in generated text, tested with template-based sentiment pairs on GPT-2 small.
• Auditing and interpretability tooling that needs localized, reversible interventions on deployed frozen models rather than retraining.

Industry relevance. Because the approach keeps weights fixed and acts only at inference time, it is compatible with serving a single frozen checkpoint while adjusting behavior per deployment or per request. The paper's negative results are equally relevant operationally: off-target KL exceeded the 0.001 threshold for every seed on Humanities, STEM, and Sentiment, which is a direct obstacle to shipping such interventions without a preservation audit.

Future Directions

• Reduce reliance on evaluation-pair direction information. The dominant empirical pattern is P versus F (for example 8/8 versus 0/8 held-out reachability and 7–8 of 48 versus 46–48 of 48 feedback endpoints), so methods that derive good directions without per-example contrasts are the obvious next target.
• Make off-target preservation actually hold. No seed met the KL criteria on Humanities, STEM, or Sentiment despite the certificate including an off-target tolerance; tighter coupling between support selection and the preservation dictionary, or explicit constrained refitting, is unresolved.
• Improve frozen-intervention transfer. Held-out MCQA recovery collapses from 0.992 ± 0.009 to 0.021 ± 0.026 when coefficients are frozen; understanding whether this is a direction-stability problem or a support problem is open.
• Fill the acknowledged evaluation gaps. Multi-target feedback tracking is not evaluated, Refusal fails reachability and reversal in every configuration, and the paper describes its experiments as exploratory — larger seed counts, additional models, and standardized task thresholds would test whether the six-axis profiles are stable.

Target Audience

Researchers and engineers working on mechanistic interpretability, activation steering, and controllable generation who already understand Transformer internals and sparse optimization; safety and alignment practitioners who need localized, reversible inference-time control with an auditable evaluation protocol; and evaluation-focused readers who want a template for measuring steering methods beyond a single target-recovery number. Beginners will find the formal sections (control certificate, energy-constrained refit, coverage objective) heavy going, though the six-axis protocol and the results tables are readable on their own.

Authors’ abstract

Language models are increasingly used in text generation, decision support, and automated interaction, creating a need for localized, reversible, and selective control of their behavior. Activation steering demonstrates that internal interventions can alter the behavior of frozen models, but how to select and evaluate compact, multi-site control interfaces remains an open question. I introduce sparse internal control, a framework for steering target behavior by applying inference-time interventions to a small set of internal model nodes. A local target-controllability formulation characterizes candidate site--direction pairs by their behavioral effects and specifies constraints on target recovery, off-target preservation, and intervention energy. Two complementary selectors instantiate this formulation: Driver-OMP selects nodes for sparse reconstruction of a desired behavioral displacement, while Coverage-Driver favors stable coverage across prompts. I propose a six-axis control-evaluation protocol measuring reachability, signed reversal, dose response, feedback controllability, off-target preservation, and held-out reuse. Exploratory experiments on indirect object identification, MMLU multiple-choice question answering, sentiment steering, and refusal steering provide evidence for the feasibility of steering task-defined behavior through sparse internal interventions. The six-axis evaluation characterizes these interventions beyond target movement, revealing distinct control properties that depend on the task and available direction information. Together, the formulation, reference algorithms, and evaluation protocol provide a foundation for the systematic study and development of internal control methods for language models.