The Pulse
Apple Researchers Make LLM Activation Steering More Selective
Apple researchers introduce Dynamically Scaled Activation Steering, a method that adjusts intervention strength token by token instead of applying one global setting. Tests on Qwen 2.5 and Gemma 2 show lower toxicity with smaller losses in
AI.info Team ·
Apple researchers are targeting a central weakness in activation steering: methods that suppress unwanted model behavior can also alter answers that never needed intervention. Their proposed solution, Dynamically Scaled Activation Steering, applies the steering transformation selectively, adjusting its strength for individual tokens and model layers rather than treating every input the same.
The work, published on Apple’s machine-learning research site in September 2026 and described in a paper dated August 24, introduces DSAS as a framework that can sit on top of several existing steering methods. The authors report lower toxicity and better retention of general language-model capabilities across Qwen 2.5 and Gemma 2 models. The paper is available from Apple Machine Learning Research, with the full technical version hosted on arXiv.
Global steering creates collateral changes
Activation steering changes a model’s internal representations during generation without updating its weights. Researchers have used the technique to reduce toxic language, induce concepts, improve factuality and alter image-generation behavior. Existing methods generally expose a global strength parameter, usually written as λ, that applies the same intervention across an input sequence or image.
That uniform treatment creates a trade-off. Stronger steering can suppress an unwanted behavior more effectively, but it can also damage fluency, factual performance or unrelated generations. Apple’s paper describes the problem with a simple example: a model steered away from a concept such as fruit may alter outputs even when the prompt has nothing to do with fruit.
DSAS separates two decisions that earlier methods often combine: when a model should be steered and how the steering transformation should work. The steering map still moves activations toward a desired target, but a separate gate estimates how much intervention each token or spatial feature requires.
DSAS uses a soft gate instead of a binary trigger
At each intervened layer, DSAS assigns a value between zero and one to an activation. A value near zero leaves the activation almost unchanged; a value near one applies the full steering transformation. The gate is produced by a logistic regressor trained to distinguish source examples that exhibit the unwanted behavior from a control set that should remain unchanged.
Apple’s researchers use average activations from each training example to fit the regressors, then apply the resulting classifier to individual token representations during inference. Principal component analysis reduces the activation dimensions before training. Afterward, the classifier can be folded back into the original activation space so that inference does not require a separate projection step.
The method needs a third data group in addition to source and target examples: controls. Source data represents behavior the model should move away from, target data represents the desired behavior, and control data represents neutral inputs that should retain the model’s original behavior. The paper’s toxicity experiments use 32 toxic source sentences, 32 non-toxic target sentences and 32 additional non-toxic control sentences.
Lower toxicity with smaller capability losses
The main evaluation combines DSAS with three steering techniques: Chinese Astron. Astrophys., ITI and LinEAS. Tests cover Qwen 2.5 in 1.5-billion- and 7-billion-parameter versions, along with Gemma 2 at 2 billion parameters. The researchers measure toxicity on the TET benchmark, perplexity on 20,000 Wikipedia sentences and five-shot accuracy on MMLU.
At one operating point, LinEAS combined with DSAS records toxicity scores of 3.98% on Qwen 2.5 1.5B, 2.26% on Gemma 2 2B and 12.80% on Qwen 2.5 7B. The corresponding LinEAS-only scores are 9.78%, 6.50% and 15.57%. On the same comparison, DSAS keeps MMLU accuracy at 59.49%, 51.60% and 73.81%, respectively.
Results also improve when DSAS is paired with the other steering methods. Chinese Astron. Astrophys. plus DSAS reaches toxicity scores of 8.64% on Qwen 2.5 1.5B, 3.48% on Gemma 2 2B and 9.79% on Qwen 2.5 7B, compared with 13.67%, 4.51% and 14.19% for the unmodified steering method. ITI plus DSAS produces lower toxicity than ITI alone on all three models.
The authors compare DSAS with CAST, MERA and AlphaSteer, three methods that also attempt conditional intervention. Their evaluation selects configurations that keep Wikipedia perplexity within a 5% increase and MMLU within a 3% decrease of the original model. Apple reports that DSAS produces the strongest toxicity-performance trade-off across the tested model and method combinations.
End-to-end training extends the approach
Apple also presents E2E-DSAS, which trains the gate and the steering transformation together. The end-to-end version builds on LinEAS and adds a control-preservation term to the training loss. Without that term, the paper says, the learned gate tends to apply strong steering broadly and approaches ordinary LinEAS behavior.
In the reported table, LinEAS with end-to-end DSAS using a ReLU gate reaches a toxicity score of 5.18% on Qwen 2.5 1.5B, 1.64% on Gemma 2 2B and 12.89% on Qwen 2.5 7B. The corresponding MMLU scores are 60.00%, 52.10% and 74.16%. The sigmoid version performs better on some Gemma 2 tests but does not improve Qwen 2.5 7B in the same way.
Selective control also works for image generation
DSAS is not limited to autoregressive language models. The researchers apply it to the single-step DMD2 text-to-image model, using activation steering inside U-Net normalization layers. They train the system to blur images containing one of six placeholder concepts—bananas, phones, castles, apples, astronauts and elephants—while leaving unrelated images close to their unmodified versions.
In the paper’s examples, ordinary steering introduces blurring across both relevant and irrelevant prompts. DSAS concentrates the effect on images associated with the selected concept. The experiment excludes CAST and MERA because the former assumes autoregressive generation and the latter cannot use the control data required by the image setup.
The cost is small, but the assumptions remain
DSAS adds computation at inference, although the measured overhead is modest relative to the steering transformation itself. Processing 100 tokens takes 0.0269 seconds for unmodified Qwen 2.5 1.5B, 0.290 seconds with the baseline Chinese Astron. Astrophys. intervention and 0.0316 seconds with Chinese Astron. Astrophys. plus DSAS. On Gemma 2 2B, the same figures are 0.0489, 0.501 and 0.0520 seconds.
The authors report a decode-throughput penalty of about 5% to 6% relative to the baseline steering method on Qwen 2.5 1.5B, with a smaller increase on Qwen 2.5 7B. They also acknowledge that DSAS assumes the relevant behavior is sufficiently separable in activation space for a linear classifier to detect it. When that assumption fails, a nonlinear gate may be needed.
DSAS therefore does not remove the judgment involved in choosing source, target and control data. It adds a selective control mechanism around existing steering systems, allowing them to intervene strongly on detected problem content and lightly—or not at all—on everything else. The paper’s code is scheduled for release on GitHub, but the Apple page does not yet provide a repository link.