AntiPaSTO: Self-Supervised Honesty Steering via Anti-Parallel Representations

AI summary

Overview

Research area: Machine learning alignment and interpretability — specifically representation-level (activation) steering of large language models, sitting at the intersection of representation engineering, singular value decomposition (SVD)-based weight editing, and self-supervised alignment.

Technical level: Advanced. The paper assumes familiarity with transformer internals (residual stream, residual-writer matrices, attention output projection, MLP down-projection), SVD decomposition, LoRA-style adapters, and loss-function design.

Scope (one sentence): The paper introduces and evaluates AntiPaSTO, a gradient-based, self-supervised method that learns to steer a language model's value-relevant behavior by training on internal representation differences elicited by contrastive persona prompts, evaluated primarily on out-of-distribution moral dilemmas.

What This Paper Is About

The paper argues that as models become more capable, humans cannot reliably verify what they say, so alignment methods must be internal (operating on representations rather than outputs), self-supervised (avoiding preference labels that become optimization targets for deception), and able to transfer out-of-distribution. The author observes that existing steering methods satisfy only some of these three requirements — supervised gradient methods (ReFT, BiPO, CAA) need human-labeled preference pairs, arithmetic self-supervised methods (ActAdd, RepE) lack gradient optimization, and probing (CCS) cannot intervene — and proposes AntiPaSTO to fill the empty gradient + self-supervised cell of that taxonomy.

Key Contributions

1. A new gradient-based internal steering method trained without preference labels beyond naming an axis. AntiPaSTO is described by its author as, to his knowledge, the first gradient-based internal steering method trained without preference labels beyond naming an axis, with value-level out-of-distribution transfer from persona pairs to moral dilemmas.

2. A loss design that separates representations along an anti-parallel axis. Steering with α = +1 and α = −1 must produce opposite hidden-state shifts along a reference direction, with coherence (total variation) and monotonicity constraints acting as barriers that prevent collapse and enforce ordered behavioral change.

3. Empirical demonstration of large out-of-distribution gains over prompting. On Gemma-3-1B, AntiPaSTO is reported to beat prompting by 6.9× Steering F1 on DailyDilemmas and to be competitive with LLM-engineered prompting, while arithmetic steering (ActAdd) fails entirely.

4. Preliminary evidence of suppression bypass. The paper reports anecdotal/preliminary evidence that internal steering can maintain bidirectional control in cases where persona prompting collapses into refusal or meta-commentary.

Main Findings

• Out-of-distribution value transfer on Gemma-3-1B: On honesty (800 persona pairs → 1,360 DailyDilemmas dilemmas), AntiPaSTO scores Steering F1 = 31.2 ± 5.3, versus 13.0 for engineered prompting (AxBench-style, GPT-4o-mini generated) and 4.5 for simple prompting. ActAdd scores 0.0.

• Raw flip metrics alongside F1: AntiPaSTO records Tgt% = 29.9, Wrong% = 1.9, Arb% = 47.0, Pmass = 0.95; simple prompting records Tgt% = 10.0, Wrong% = 1.3, Arb% = 13.4, Pmass = 0.99.

• Multi-axis transfer: Across 3 seeds each on Gemma-3-1B, AntiPaSTO wins on 5 of 6 tested value axes — Honesty (31.2 ± 5.3 vs 13.0 engineered prompt vs 4.5 prompting), Ambitious (8.9 ± 5.3 vs 4.1 vs 6.2), Friendly (12.8 ± 15.1 vs 4.1 vs 6.9), Responsible (14.4 ± 3.0 vs 12.4 vs 8.4), Traditional (8.3 ± 4.4 vs 3.4 vs 1.4). Courageous is the exception, where engineered prompting dominates (13.0 vs 7.3 ± 9.5 vs 4.6).

• Cross-model generalization at default hyperparameters (≤ 4B): Gemma-3-270M (0.27B) 38.7 vs prompting 0.0; Gemma-3-1B 31.2 vs 4.5; Qwen3-0.6B 11.2 vs 0.0; Qwen3-4B 9.3 vs 2.6; Gemma-3-4B 5.5 vs 0.6.

• Larger models need exploration: Gemma-3-12B reaches F1 = 43.9 (2.5× prompting) and Qwen3-14B reaches F1 = 25.7 (vs 0 for prompting) with hyperparameter exploration, but larger models show higher initialization variance and NaN failures or overfitting with bad seeds. Only Llama-3.1-8B resisted steering even with exploration.

• Suppression bypass (OLMo-3-7B-Think): On Value/Honesty items, AntiPaSTO steers bidirectionally (−0.2 at α = −1, 0.0 baseline, +0.6 at α = +1), while prompting produces −0.4 for both α = ±1, indicating output-level resistance. Control categories (Preference/A, Math/Correct) show the two methods behaving differently as well.

• Component ablations on Gemma-3-1B (n=3 except rotation n=2): Full AntiPaSTO reaches F1 = 21.4 ± 5.5. Removing V rotation drops performance by 99%; replacing WANDA dimension selection with random dimensions drops 92%; replacing the SVD adapter with LoRA drops 96%; removing the coherence region (no TV bound) drops 76%; random loss subspace drops 61%; fixing S scaling drops 50%; removing Fisher weighting (dot product instead) drops 34%; removing the monotonicity barrier drops 18% (to 17.5 ± 4.5).

• Which values compete with honesty: Of the 145 items where honesty conflicts with another value, the main opponents are self-interest (52 dilemmas), patience (27), loyalty (18), and empathy-related values (peacekeeping, protection, avoidance). The author interprets the model as prioritizing competing values rather than being dishonest.

• Asymmetric resistance is mostly seed-dependent: Across 500 runs on multiple models, only 8% of questions show a consistent asymmetry direction across random seeds. Qwen3 shows a consistent aggregate bias toward easier dishonesty-direction steering (+0.9–2.3 nats, p < 0.001).

• Post-training affects steerability: On seven Olmo-3 models, steerability correlates with post-training stages; the author reports that safety-focused training reduces steerability while reasoning-focused training preserves it.

• Seed variance is a stated limitation: Typical standard deviation is approximately 5–7 across 3 seeds, and the paper reports no in-distribution evaluation.

Methodology in Plain English

The method trains a small adapter that changes how a model processes information, rather than injecting a fixed vector into its activations.

Training data is deliberately minimal. Instead of human-labeled preference pairs, the method uses 800 synthetic pairs built from a template sentence with two contrasting words swapped in — "You are honest…" versus "You are dishonest…" — with no completions generated and no labels indicating which output is preferred. These are called incomplete contrast pairs because they stop before the model writes an answer. The author notes the two prefixes produce representations that are roughly 95% identical; the difference Δh = h_cho − h_rej is the training signal, extracted at the final token position.

The loss pushes the two steering directions apart, but in opposite directions. The projection loss rewards anti-parallel separation: it multiplies an axis-alignment term (cosine similarity of each shift with a reference direction) by a subspace-concentration term that penalizes the adapter for moving representations outside the relevant subspace. A symlog compression bounds gradients and a quadratic term penalizes same-side shifts.

Two constraints act as guardrails rather than objectives. A coherence constraint bounds the total variation distance between the steered and reference next-token distributions per token, using an entropy-adaptive threshold (κ = 0.3, β = 0.1) and a log-barrier penalty aggregated with LogSumExp. The paper argues TV is preferable to KL because it is bounded in [0, 1], interpretable, and cannot be reward-hacked by pushing rare token probabilities to extremes. A monotonicity constraint penalizes violations of the expected ordering of preference gaps across α values.

The adapter edits the model's own transformation basis. Rather than adding to the activation stream, it modifies residual-writer weight matrices (attention output projection W_O and MLP down-projection W_down) in their SVD coordinates, following the PiSSA decomposition. It learns a coefficient-dependent weight W′(α) = U(S + αΔS)R_v(α)V^T + W_res for α ∈ {−1, 0, +1}, where R_v(α) is a Cayley-parameterized rotation in V-space (following SSVD) and ΔS is a learnable singular-value perturbation. Cayley parameterization ensures exact orthogonality and reversibility. Learnable SVD dimensions are initialized with a WANDA variant.

Calibration is required post-hoc. Because the learned direction is unsupervised, α = +1 may correspond to honest or to dishonest depending on the random seed; a small validation set determines which direction maps to which behavior.

Evaluation. Training uses AdamW on 800 persona pairs for 30 epochs (about 1 hour on an A100). Evaluation is on DailyDilemmas — an external benchmark of 1,360 moral dilemmas across 9 value dimensions developed independently — using the "self" subset, extended with 40 control questions (math correctness, arbitrary preferences like favorite color). The model outputs log-odds y(α) = log(P(Yes|α)/P(No|α)) at α ∈ {−1, 0, +1}. The new Steering F1 metric treats intended flips as true positives, reverse flips as false positives that cancel correct flips, and arbitrary flips as additional false positives, using net_correct = max(0, correct − wrong) so that methods breaking more than they fix score zero, with a pmass_ratio term penalizing weak probability shifts and z-weighting by baseline confidence (|y_0|/σ). The author reports 2400+ runs on sub-1B models.

Why This Matters

Impact on research. The paper targets a specific gap: gradient-based steering that does not require human preference labels. It also pushes back on the idea that probing accuracy demonstrates control — the author cites the argument that probing is correlational, whereas intervening on a representation gives causal evidence. The introduction of a bidirectional metric (Steering F1, with net_correct and pmass_ratio) responds to the paper's claim that no existing metric captures bidirectional value flipping. The finding that only 8% of questions show consistent asymmetry direction across seeds is framed as encouraging, because it suggests resistance comes from random local minima rather than stable model properties — a testable claim for follow-up work.

Real-world applications. The paper does not enumerate deployment applications, but several plausible uses follow from the properties the method claims:

• Auditing and red-teaming safety-trained models, since internal steering appears in the reported examples to keep working where persona prompting triggers refusal or meta-commentary.
• Studying models whose stated values conflict with other values (self-interest, loyalty, patience, empathy) in morally charged decision scenarios.
• Investigating how training stages (base, SFT, DPO, thinking) change how steerable a model is, using Olmo-3's public checkpoints.
• Building controllable assistants where behavioral changes must be applied without verbose system prompts that are visible or that the model may reject.

Industry relevance. Steering adapters are cheap to train by the paper's numbers (about 1 hour on a single A100 per value dimension for a 4B model), which is broadly relevant to teams that want behavioral control without collecting human preference data. The finding that default hyperparameters work across Gemma-3 and Qwen3 model families up to 4B parameters, but that 12B and 14B models required exploration, is directly relevant to practitioners deciding whether to budget for hyperparameter search. The paper also notes that safety-focused post-training reduces steerability, which matters to anyone deploying safety-tuned models.

Future Directions

1. In-distribution evaluation. The paper explicitly states it does not evaluate in-distribution steering performance because training data consists of incomplete prompt prefixes with no ground-truth labels; whether OOD gains come at a cost to in-distribution behavior is unmeasured.

2. Reducing initialization sensitivity. Larger models concentrate gradient pressure on fewer layers, causing NaN failures or overfitting with bad seeds; the paper calls seed variance (std ≈ 5–7) an engineering problem and suggests better initialization or optimization may resolve both seed variance and asymmetric resistance.

3. Using engineered prompts as inputs. Replacing the simple persona pair ("honest"/"dishonest") with AxBench-style engineered prompts as the source of the contrastive axis is explicitly named as future work, as is direct comparison against task-specific prompting.

4. Extending beyond the final token and beyond honesty. The author notes that extracting only from the final token may miss steering-relevant information distributed across earlier tokens' key-value states or other attention pathways, and that reliable generalization to further value dimensions (fairness, harm, deception) requires further work.

Target Audience

This paper is most useful to alignment and interpretability researchers working on activation steering, representation engineering, and model editing, particularly those already familiar with CAA, RepE, ActAdd, ReFT, and PiSSA-style SVD adapters. It is also relevant to safety engineers evaluating whether steering can substitute for or supplement prompting when models refuse, to researchers studying how post-training stages (SFT, DPO, reasoning training) affect steerability using the Olmo-3 family, and to practitioners who need behavioral control over small- to mid-sized models without collecting preference labels. It is less suited to readers without a working understanding of transformer internals and loss design.

Authors’ abstract

As models grow more capable, humans cannot reliably verify what they say. Scalable steering requires methods that are internal, self-supervised, and transfer out-of-distribution; existing methods satisfy some but not all three. We introduce AntiPaSTO, which separates representations along an antiparallel axis (+1/-1 produce opposite shifts), with coherence constraints preventing collapse. Training uses only two contrasting words inserted into template sentences, with no preference labels. When we use 800 such synthetic pairs on Gemma-3-1B, AntiPaSTO beats prompting baselines by 6.9x Steering F1 on DailyDilemmas and wins on 5 of 6 tested value axes. We also find preliminary evidence that it maintains bidirectional control where prompting triggers refusal.