Self-Improvement with Auxiliary Rewards: Learning from Verified Traces and Evaluating Transfer

AI summary

Overview

• Research area: Machine learning / large language model post-training — specifically, self-improvement of small reasoning models using reward signals derived from the model's own predictive distributions, plus evaluation of transfer to non-mathematical tasks.
• Technical level: Advanced. The paper assumes familiarity with reinforcement learning from verifiable rewards (RLVR), test-time reinforcement learning (TTRL), GRPO, DPO, supervised fine-tuning, and token-level entropy/confidence statistics.
• Scope: A controlled study of whether auxiliary trajectory scores (Self-Certainty vs. a UID-based conjunction score) add learning signal beyond answer-level correctness when adapting Qwen3-1.7B (with supplementary Qwen3-4B experiments) on only 30 AIME2026 problems, evaluated on math, science, critique, scientific-reasoning, and instruction-following benchmarks.

What This Paper Is About

Answer-level rewards — the checkable right/wrong signals used in RL with verifiable rewards — treat every response with the same final answer as equally good, even though those responses differ in reasoning soundness, completeness, and redundancy. The authors ask whether auxiliary preferences computed from the model's own token-level predictive statistics can extract additional training value from these equally-correct traces, without new problems or step-level annotations. They test this both online (as a reward augmentation inside TTRL, with no reference answers) and offline (as DPO preferences or SFT weights among verified-correct responses), and then measure whether the resulting improvements transfer to tasks beyond math answer generation.

Key Contributions

1. An independent assessment of whether auxiliary signals distinguish reasoning quality among correct responses — comparing Self-Certainty (SC) and UID-Conj. as selectors against majority voting and against random selection from verifier-correct pools, plus a blinded external-judge evaluation on validity, sufficiency, consistency, and relevance/non-redundancy.
2. A controlled study of auxiliary trajectory preferences for small-model self-improvement under limited supervision, spanning online TTRL reward augmentation and offline DPO/weighted-SFT, with matched controls within each training objective (no-auxiliary TTRL, a random-auxiliary-score TTRL control, random correct–correct DPO pairs, and uniform SFT on the same response pool).
3. An evaluation of transfer from mathematical self-improvement to critique-verdict prediction and research-level scientific reasoning (CriticBench, LabBench2) without task-specific training, plus instruction-following benchmarks (IHEval, IFBench) as a contrasting probe.
4. A data-scale demonstration: adaptation uses only 30 AIME2026 problems without reference answers for 2 epochs, and offline extensions use historical AIME problems gathered until 2025, showing gains under deliberately limited supervision.

Main Findings

• Auxiliary scores help most when combined with answer-level agreement. Used as direct selectors, SC and UID-Conj. do not beat majority voting: for Qwen3-1.7B, SC reaches 25.4% and UID-Conj. 25.7% versus 25.9% for majority voting; for Qwen3-4B, 33.8% and 34.2% versus 33.9% (averaged over rollout samples n ∈ {2, 4, 8} on 300 pooled questions per model). Adding Borda voting changes this: UID-Conj. + Borda reaches 27.0% for Qwen3-1.7B (the best in that setup) and SC + Borda reaches 34.7% for Qwen3-4B.

• Self-Certainty picks better-judged traces; UID-Conj. does not. Against random selection among verifier-correct responses, SC shows net win rates of 13.2 and 11.7 percentage points for Qwen3-1.7B and Qwen3-4B, while UID-Conj. shows −7.7 and −0.5 points. SC holds a mean total-score advantage of 0.584 and 0.338 points. Ties account for more than half of outcomes. SC wins on validity and consistency; UID-Conj. scores higher on relevance/non-redundancy.

• Online adaptation improves with auxiliary rewards, with a caveat about the reported magnitude. In Table 3, adding SC or UID-Conj. to TTRL raises Qwen3-1.7B's four-benchmark macro-average from 25.51% (TTRL) to 27.54% and 27.59% respectively, i.e. +1.63 and +1.68 points over the base model's 25.91%. The abstract reports the UID-Conj. gain as 2.08 percentage points over unmodified TTRL, and the introduction reports gains of 2.03 and 2.08 points for SC and UID-Conj.; these framings differ from the base-model deltas tabulated in Table 3.

• Auxiliary rewards can mitigate transfer degradation on the larger model. For Qwen3-4B, unmodified TTRL raises AIME2026 accuracy from 55.33% to 60.00% but lowers HMMT2026 and MinervaMath performance. UID-Conj. instead preserves AIME2026 at 55.33% while improving HMMT2026 to 13.94% and MinervaMath to 31.40%, for a macro-average of 39.59% (+0.98). SC-TTRL on Qwen3-4B falls to 37.81% (−0.81).

• Offline weighted SFT gives the strongest and most statistically reliable evidence. UID-Conj. reaches 29.49% macro-average with weighted SFT (+3.58 over the 25.91% base) versus 27.56% for uniform SFT (+1.65) — a 1.93-point gap with a paired prompt-bootstrap 95% CI of [0.44, 3.44]. UID-Conj. DPO reaches 29.03% (+3.12), 1.25 points above Random-DPO's 27.78%, but that interval [−0.90, 3.47] includes zero. SC-SFT reaches 28.14% (+2.23); SC-DPO reaches 26.82% (+0.91), below Random-DPO.

• Transfer to critique-verdict prediction occurs without task-specific training. UID-Conj. TTRL achieves the highest CriticBench F1 (63.23%) and accuracy (68.00%), averaging 65.62% — 0.36 points above the base model, 1.41 above unmodified TTRL, and 0.93 above SC. The CriticBench accuracy gain over TTRL (+1.24, CI [0.20, 2.32]) and F1 gain (+1.58, CI [0.18, 3.01]) have intervals excluding zero; comparisons against Random-TTRL and the LabBench2 comparison (+0.76, CI [−0.80, 2.32]) include zero.

• Transfer is task-dependent and not uniformly favorable. UID-Conj. raises LabBench2 accuracy to 42.00% from the 41.44% base, while SC's 41.80% is the smaller gain. On instruction following, SC achieves a higher macro-average (39.16%) than UID-Conj. (38.51%) and the base model (38.44%).

• Judged trace quality does not predict training utility. SC selects higher-rated traces, but UID-Conj. produces larger offline training gains and stronger selected transfer results — a contrast the authors emphasize as motivation for evaluating auxiliary preferences by downstream learning outcomes rather than by selection quality alone. The authors note this is observed across separate diagnostic and training cohorts and does not establish that lower-judge-rated traces are individually more useful for learning.

• Benefits are not tied to one training dataset. When Qwen3-1.7B is trained on the S1K dataset instead, UID-Conj. outperforms standard TTRL on five of the six reported benchmarks (AIME 37.33 vs 35.33, HMMT 13.94 vs 13.33, MinervaMath 26.10 vs 25.07, GPQA-D 41.21 vs 40.71, LabBench2 42.00 vs 41.20), with CriticBench lower (63.19 vs 62.46 for TTRL — actually higher; the exception is AIME where SC leads at 41.33).

Methodology in Plain English

The two signals. Both scores are computed from the model's own next-token probability distributions, restricted to the top-25 highest-logit tokens at each position and renormalized.

• Self-Certainty (SC) averages, over response tokens, the KL divergence between a uniform distribution over those top-25 tokens and the model's predictive distribution — a confidence-style score.
• UID-Conj. splits the token-entropy trajectory into 32 relative-position bins, then multiplies two calibrated features: a global-variation term (population variance of bin entropies, so globally flat entropy is not rewarded) and a local-smoothness term (negative mean absolute difference between adjacent bins). Each feature is mapped to a quantile (midrank) and multiplied, so a response must score well on both.

Online setting (majority-gated RL). Following TTRL, the model samples responses to each prompt and the most frequent extracted answer becomes a pseudo-label. Responses agreeing with it get reward +1, disagreeing responses −1. Auxiliary scores are then used only within the agreeing group: a centered, clipped score difference (λ = 0.1, δ = 0.5, bounding the adjustment by 0.05) is added to the reward. This preserves the ordering between agreeing and disagreeing responses; groups with fewer than two agreeing responses get no adjustment. GRPO advantages are computed from the combined rewards, with a clipped policy objective and a separate reference KL penalty. Training runs for 2 epochs on 30 AIME2026 problems with no reference answers.

Offline setting (verified traces). A fixed pool of responses is scored by the frozen base model, keeping only complete, non-truncated, correct, score-eligible responses. Two uses:

• Auxiliary-score DPO: pair the highest-scoring and lowest-scoring correct response per prompt (requiring a minimum score gap), so the preference reflects trajectory quality, not correctness. Controls use randomly selected and oriented correct–correct pairs.
• Auxiliary-weighted SFT: weight each correct response by a softmax over its auxiliary score, normalized so weights average one within each prompt — keeping the prompt's total example weight fixed while emphasizing higher-scoring responses. Uniform SFT (all weights = 1) is the control on the identical pool. Offline training runs for 1 epoch.

Evaluation. Mathematical reasoning is measured on AIME2026, HMMT2026, and MinervaMath; science reasoning on GPQA's Diamond subset; research-level reasoning on CriticBench (verdict accuracy and incorrect-class F1) and LabBench2; instruction following on IHEval and IFBench. Verifiable benchmarks report mean accuracy over five sampled responses per question. Uncertainty is quantified with 95% paired prompt-cluster bootstrap intervals from 100,000 replicates, retaining all five responses within each resampled prompt, weighting benchmarks equally.

Why This Matters

Impact on research. The paper separates two things the literature often conflates: whether an auxiliary signal selects good reasoning, and whether it teaches well. SC wins on the first and UID-Conj. often wins on the second, which argues that intrinsic-signal research should be evaluated by downstream training gains rather than by best-of-N selection accuracy. The within-group reward augmentation design is also a clean way to add trajectory-level supervision without a reward model or step-level labels — a practical route to squeezing more signal out of a very small problem set (30 problems).

Real-world applications:

• Low-data domain adaptation: adapting a compact model to a new domain from a handful of unlabeled problems, using majority agreement plus an auxiliary score as the training signal.
• Reusing generated data: weighted SFT over an existing pool of verified-correct model outputs, extracting more from data already generated rather than collecting more.
• Cost-constrained deployment: improving 1.7B–4B class models where running large models or training separate reward models is infeasible.
• Transfer to judgment tasks: using math self-improvement to improve critique-verdict prediction and evidence-conditioned scientific reasoning without task-specific supervision.

Industry relevance. The methods need no separately trained reward model and no process annotations, and the offline variants operate on response pools that pipelines already produce. The paper is candid, however, that generating and scoring multiple trajectories still costs compute and that net energy or cost savings are not established.

Future Directions

1. Isolating trajectory-specific effects from generic reward variation. The authors note that online comparisons do not fully separate the benefit of trajectory-specific preferences from the effect of introducing reward variation into unanimous groups, and point to a control that permutes the realized auxiliary terms within majority-agreeing groups.
2. Building auxiliary preferences that predict learning contribution. The gap between judged trace quality (SC wins) and training utility (UID-Conj. often wins) motivates scores designed to predict a response's value as a training example rather than its quality as an answer.
3. Stronger evaluation of transfer. The current diagnostic uses one model judge on verifier-correct responses and measures critique-verdict accuracy, evidence-conditioned scientific responses, and programmatically verified instruction following rather than unrestricted open-ended reasoning; broader response cohorts and human validation are called for.
4. Open-ended and research-level reasoning with expert verification. The stated long-term goal is extending these improvements to tasks where reliable answer-level supervision is unavailable, such as research-level mathematics — with the caveat that confidence and smooth entropy trajectories do not guarantee factual or logical validity, and majority agreement can encode shared misconceptions.

Target Audience

Researchers and practitioners working on LLM post-training, reinforcement learning with verifiable rewards, and self-improvement methods — particularly those interested in intrinsic/intra-model reward signals, data-efficient adaptation of small models, and rigorous evaluation of transfer beyond the training distribution. It is also useful for evaluation-minded readers who want a template for reporting paired bootstrap confidence intervals alongside benchmark deltas, and for engineers deciding between DPO-style preference construction and score-weighted SFT on an existing correct-response pool.

Authors’ abstract

Answer-level rewards cannot distinguish reasoning traces with equally correct outcomes. We study whether auxiliary trajectory preferences improve small-language-model self-improvement and transfer under limited supervision. We compare confidence-based (SC) and entropy-based metrics (UID-Conj.), which combines global entropy variation and local smoothness, through online reward augmentation and offline training on correct responses. Online experiments adapt Qwen3-1.7B on 30 AIME problems without reference answers; offline experiments use historical AIME problems. UID-Conj. improves the online macro-average across the adaptation benchmark and three transfer benchmarks by 2.08 percentage points over unmodified TTRL. The strongest evidence comes from offline weighted SFT: UID-Conj. exceeds uniform weighting on the same response pool by 1.93 points, with a paired prompt-bootstrap 95\% CI of [0.44, 3.44]. Benefits are task-dependent: SC rewards judged trace quality and instruction following, while UID-Conj. rewards yield stronger training outcomes in several settings. We find that auxiliary preferences can expose learning value among equally correct traces not captured during inference time, motivating their evaluation through downstream gains.