Label-free steering: Compressing test-time reinforcement learning into bias-only subspaces

AI summary

Overview

Research area: Test-time reinforcement learning (TTRL) and parameter-efficient adaptation of large language models, extended to vision-language and audio reasoning models.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning objectives (GRPO, policy gradients), low-rank adaptation, and subspace arguments from optimization theory.

Scope: A single set of experiments and two theoretical propositions testing whether test-time RL can work when both the reward signal (majority-vote pseudo-labels, no ground truth) and the trainable parameter space (≈100K additive bias vectors inside a frozen 7B-scale backbone) are severely restricted.

What This Paper Is About

Prior test-time reinforcement learning methods let a model improve itself on unlabeled data using its own generations as rewards, but they normally update most or all of the model's parameters. This paper asks how much optimization capacity is actually needed: it freezes a 7B-scale pretrained backbone, trains only a small set of additive bias vectors (≈100K parameters), and rewards those vectors with majority-vote pseudo-labels derived from the model's own rollouts. The goal is to show that useful self-improvement survives this extreme bottleneck, and to explain when and why it does.

Key Contributions

1. A label-free bias-only TTRL pipeline. The paper combines TTRL's majority-vote pseudo-label reward, GRPO's group-normalized advantage, and bias-only steering (one additive bias vector per targeted decoder layer), so that the backbone θ receives no gradient and only the bias parameters φ are updated. No ground-truth labels are used at any point in the loop.

2. Cross-modality validation of the same procedure. The identical ≈100K-parameter training recipe — changing only the prompt and the reward/grading function — is evaluated on text (MATH-500), vision-language (AI2D, LogicVista, MathVista), and audio (MMAU) reasoning.

3. Held-out transfer evidence. Bias vectors trained on MATH-500 are frozen and applied without further optimization to 4,500 verified-disjoint MATH problems.

4. A theoretical and empirical account of why a tiny subspace works. Proposition 1 bounds majority-vote pseudo-label error as decaying exponentially in the number of rollouts for positive-margin problems; Theorem 1 bounds guaranteed local improvement by "accessible gradient energy" — the squared norm of the full RL gradient projected into the trainable subspace — rather than by subspace dimensionality. Both are tested empirically.

Main Findings

• Headline MATH-500 results. Label-free bias-only TTRL reaches 79.50% ± 0.70 with Qwen2.5-Math-7B and 76.67% ± 1.57 with Qwen2.5-7B, against untrained baselines of 56.0 and 45.9 respectively (Δ +23.50 and +30.77). This updates approximately 76,000× fewer parameters than full-model fine-tuning.

• Bias-only beats parameter-matched LoRA. Under an identical 100,352-parameter budget, a LoRA (exact-match) adapter (r=1 restricted to q_proj on 14 layers) scores 78.80 ± 1.25 on Qwen2.5-Math-7B/MATH-500 versus bias-only's 79.50 ± 0.70, and falls below the untrained baseline on LogicVista (35.86% ± 0.68 vs. 37.50%). Bias-only TTRL outperforms both LoRA (exact-match) and LoRA r=16 on all six benchmarks, despite LoRA r=16 training roughly 50× more parameters, and outperforms LoRA r=64 on five of six benchmarks — losing only on AI2D.

• Full fine-tuning is not uniformly better. FullFT (≈7.6B trainable parameters) exceeds bias-only TTRL on five of six benchmarks, but on MathVista bias-only TTRL is best overall (65.40% ± 1.77) ahead of FullFT (62.57% ± 0.40).

• Self-consistency alone explains part, not all, of the gain. Inference-time majority voting with no training already recovers substantial ground on MATH-500 (e.g., 65.53% ± 1.27 with Qwen2.5-Math-7B), but bias-only TTRL improves further on every benchmark.

• Label-free can match or slightly exceed labeled bias-only adaptation. In a 2×2 ablation on MATH-500 crossing reward source with parameterization, pseudo-label bias-only reaches 76.67% ± 1.57 vs. labeled bias-only 75.47% ± 1.21 on Qwen2.5-7B (a 1.2-point gap) and 79.50% ± 0.70 vs. 77.27% ± 0.70 on Qwen2.5-Math-7B (a 2.2-point gap). Full fine-tuning shows the opposite pattern: labeled FullFT is 87.93% ± 0.99 vs. pseudo-label FullFT 77.00% ± 0.53 on Qwen2.5-7B (10.9 points), and 85.13% ± 1.33 vs. 83.80% ± 0.28 on Qwen2.5-Math-7B (1.3 points).

• Large compute reduction. Bias-only training uses a single GH200 GPU at 4.45–4.95 GPU-hours/run, versus four GH200 GPUs under FSDP at 79.24–82.12 GPU-hours/run for full fine-tuning — a 16.0–18.5× reduction in GPU-hours on top of the parameter reduction.

• The learned vectors transfer. Applying frozen step-200 checkpoints to 4,500 verified-disjoint MATH problems improves Qwen2.5-7B from 46.3% to 70.9% (+24.6) and Qwen2.5-Math-7B from 52.5% to 75.4% (+22.9), with no further training.

• Pseudo-label reliability tracks rollout count and consensus. On an N=64 batch of MATH-500 problems with G_max=64 rollouts, majority-vote pseudo-label accuracy increases monotonically with the number of rollouts G, and higher rollout consensus (proxy m = (N₁−N₂)/64) corresponds to higher pseudo-label accuracy — consistent with Proposition 1's exponential decay bound.

• Equal-sized subspaces are not equally trainable. Across nine single-layer bias subspaces each containing exactly 3,584 parameters, accessible gradient energy correlates strongly with final training accuracy: Spearman ρ = 0.917, exact two-tailed permutation test, p = 0.0013, n = 9.

• Multimodal audits find no single dominant confound, but narrow task-specific effects. On AI2D, training substantially reduces the official parser's failure-to-extract rate, so part of the gain comes from output-format compliance. On LogicVista, an exhaustive check of all 448 problems finds 77 flipped from wrong to correct and 47 from correct to wrong. On MathVista, the steered model answers exactly 0 on all 49 age-gap questions, contributing only one net problem to the gain. On MMAU, a single phoneme-counting template (15.3% of the set) accounts for roughly two-thirds of the improvement: +15.1 points on that template versus +1.4 on the remaining 847 problems, which is within seed variation.

Methodology in Plain English

The method keeps the pretrained model completely frozen and adds a single learned vector to the output of each targeted MLP block. Concretely, for each of the 28 MLP layers in the 7B models, the layer output becomes the usual down-projection of the up-projection plus a bias vector b_l; only these bias vectors are ever updated (28 × 3584 = 100,352 parameters).

Training is a self-contained loop. For each unlabeled problem the frozen model generates G rollouts (G = 32–64). The most common final answer across those rollouts becomes a pseudo-label. Each rollout gets reward +1 if its answer matches the pseudo-label and −1 otherwise. GRPO converts these rewards into a group-normalized advantage, applied identically at every token, and gradients are back-propagated only into the bias vectors — the backbone receives none. Optimization uses AdamW at bfloat16, with a fixed budget of 200 steps and the final checkpoint reported. Hyperparameters (learning rate, G, schedule) are swept per task; text uses a cosine schedule, vision-language a delayed-cosine schedule (60 flat + 140 decay steps), and audio a constant schedule, all at 1e-3.

Evaluation is greedy decoding, pass@1, single generation per problem. The same procedure is applied unchanged across modalities, with only the prompt and the reward/grading function adapted. To test whether adaptation is specific to the test problems, the trained bias vectors are frozen and applied to 4,500 verified-disjoint MATH problems. To test whether the choice of subspace matters, the authors compare against a parameter-matched LoRA adapter and run a single-layer sweep across nine layers, computing each layer's accessible gradient energy from a shared backward pass on a fixed 16-problem batch and correlating it with the accuracy achieved after training that layer alone.

Why This Matters

The paper reframes parameter efficiency as an experimental probe: by squeezing test-time RL into ~100K parameters, it isolates which properties of a trainable subspace actually enable adaptation. The answer offered is that dimensionality is the wrong variable — what matters is how much of the full RL gradient the subspace can access, plus how reliable the self-generated reward is. If that generalizes, it suggests that self-improvement under label-free rewards is primarily a question of signal quality and gradient alignment, not of scale of trainable capacity.

Real-world applications (inferred from the method's setting, not claims made by the paper):

• Adapting deployed reasoning models to new domains at inference time using only unlabeled inputs, without collecting annotation.
• Low-resource or privacy-constrained settings where training labels cannot be gathered or shared.
• On-device or edge adaptation, given the single-GPU, few-GPU-hour training cost relative to full fine-tuning.
• Multimodal assistants that need to adjust to a specific deployment distribution (charts, diagrams, audio queries) where verifiable reward functions are hard to write.

Industry relevance: The compute footprint matters. A method that trains 100,352 parameters on one GH200 in 4.45–4.95 GPU-hours, versus 79.24–82.12 GPU-hours on four GH200s for full fine-tuning, changes the economics of shipping per-deployment or per-customer adaptation. The paper's own caveat is also relevant to production: because rewards come from the model's own outputs, the procedure can reinforce existing errors rather than correct them, and the authors treat auditing for that as a necessary deployment step.

Future Directions

• Beyond majority-vote rewards. The paper adopts TTRL's majority-vote signal unchanged and notes that an incorrect-but-self-consistent rollout group still yields a pseudo-label. Whether better label-free reward estimators exist for the bias-only regime is left open.

• Understanding the accessible-gradient-energy mechanism more deeply. The correlation (ρ = 0.917, p = 0.0013, n = 9) is measured over nine single-layer subspaces on one pipeline; whether accessible gradient energy can be used predictively to select layers or subspaces before training, rather than only to explain results afterward, is not established.

• Closing the gap to labeled full fine-tuning. Under label-free rewards, bias-only adaptation stays within 0.3–4.3 points of full fine-tuning while updating ~76,000× fewer parameters, but the labeled FullFT result on Qwen2.5-7B (87.93% ± 0.99) remains far above the label-free bias-only result (76.67% ± 1.57). What the gap consists of — capacity, reward quality, or both — is unresolved.

• Auditing and mitigating reinforcement of existing errors. The paper identifies narrow task-specific confounds (AI2D parser failures, MathVista age-gap questions, the MMAU phoneme-counting template covering 15.3% of the set) but no single dominant confound; general methods for detecting and preventing such reward exploitation in label-free loops are not proposed.

Target Audience

Researchers working on test-time adaptation, self-training and self-rewarding models, parameter-efficient fine-tuning, and reinforcement learning for language and multimodal models. It is also useful to practitioners evaluating whether any form of test-time RL is affordable for their deployment, though the theory in Section 4.2 and the ablation design require a background in RL objectives and optimization that makes the full paper inaccessible without graduate-level machine learning training.

Authors’ abstract

Test-time reinforcement learning (TTRL) enables models to improve their reasoning without relying on labeled training data, but existing approaches typically optimize a large fraction of the model parameters. This raises a natural question: can effective test-time adaptation emerge when both the reward signal and the optimization space are severely restricted? We answer this question with label-free bias-only TTRL, which uses majority-vote pseudolabels as rewards and optimizes only ~100K bias parameters while keeping the pretrained backbone frozen. On MATH-500, our approach reaches 76.67% accuracy with Qwen2.5-7B, slightly exceeding our own labeled bias-steering reproduction while optimizing 76,000x fewer parameters than full-parameter TTRL. The same training procedure improves performance across vision-language and audio reasoning tasks, including MathVista, AI2D, LogicVista, and MMAU. We further show that the learned steering vectors transfer to 4,500 held-out MATH problems, indicating that the adaptation is not limited to the problems used during test-time optimization. Finally, we analyze why this highly restricted adaptation can work, showing that majority-vote reliability improves with rollout consensus and that bias subspaces with greater accessible gradient energy exhibit stronger downstream trainability. These results demonstrate that substantial test-time adaptation can emerge from optimizing a tiny bias-only subspace using entirely label-free rewards.