TTPO: Test-Time Policy Optimization

AI summary

Overview

• Research area: Natural Language Processing, specifically post-training of large language models for mathematical reasoning, test-time training (TTT), on-policy self-distillation (OPSD), and reinforcement learning from verifiable rewards (RLVR).
• Technical level: Advanced. The paper assumes familiarity with GRPO-style group-relative reinforcement learning, forward KL distillation, token-level gradient weighting and masking, and the distinction between a teacher and student policy conditioned on different prompt prefixes.
• Scope: The paper introduces Test-Time Policy Optimization (TTPO), an asymmetric objective that combines distillation on pseudo-label-agreeing rollouts with RL penalties on disagreeing rollouts, and evaluates it on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B across five competition-level mathematics benchmarks.

What This Paper Is About

Reinforcement learning and on-policy self-distillation have driven recent progress in mathematical reasoning for large language models, but both depend on ground-truth answers: the reward needs them for verification and the teacher needs them as privileged context. In test-time training, where a model must improve on the very problems it is solving and labels never arrive, these methods do not apply, and the natural substitute — a majority-vote pseudo-label — is fragile because a corrupted teacher misleads at every token. The paper's goal is to build a label-free training objective that stays reliable even when the majority vote is frequently wrong.

Key Contributions

1. A structural observation about pseudo-label noise: The authors show that majority-vote pseudo-labels remain useful despite frequent errors. On AIME 2026 with Qwen3-1.7B, pseudo-labels are wrong for approximately 85% of prompts on average, yet approximately 79% of negative samples produce answers that are neither the pseudo-label nor the ground truth, so penalizing disagreement stays correct regardless of pseudo-label quality.
2. The TTPO method: An asymmetric objective that applies each signal where it stays correct — OPSD distillation on rollouts that agree with the pseudo-label, and GRPO penalties on rollouts that disagree — with token-level selection in both branches (down-weighting converged positions in distillation, masking to confident errors in RL).
3. Label-free results matching label-supervised methods: Trained without any labels, TTPO matches or exceeds label-supervised OPSD across Qwen3-1.7B/4B/8B on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in the pure TTT setting, and reaches gains of +25.2% to +36.4% with thinking mode disabled across scales.
4. Evidence of sustainable self-evolution and generalization: Tracking Avg@12 and Maj@12 shows that majority-vote routing creates a cycle in which better rollouts yield better pseudo-labels and a higher training ceiling, and training on any one benchmark improves performance on the other two.

Main Findings

• Pseudo-label errors are asymmetric: A corrupted reward misleads once per trajectory, but a corrupted teacher misleads at every token. GRPO on negative samples requires only "not in the majority cluster," which is independent of the pseudo-label's content, whereas pure forward KL on all samples propagates the error to all K trajectories (reported as K = 64 sampled trajectories per problem in the TTPO configuration).
• TTPO exceeds label-dependent OPSD on OpenThoughts training data: Averages are 40.1 vs. 39.7 (Qwen3-1.7B), 58.6 vs. 58.4 (Qwen3-4B), and 62.6 vs. 61.7 (Qwen3-8B), despite TTPO using only majority-vote pseudo-labels while OPSD and GRPO use ground truth.
• Strong label-free TTT performance (Table 2): Qwen3-1.7B reaches 45.2 average with TTPO versus 41.9 for OPSD-TTT and 40.2 for TTRL, a 7.2-point absolute gain over the 38.0 base. Qwen3-4B reaches 61.1 and Qwen3-8B reaches 65.3.
• Smaller models trained with TTPO match or surpass larger untrained ones: On OpenThoughts, TTPO on Qwen3-4B (58.6 avg) matches the Qwen3-8B base model (58.6 avg); in TTT, TTPO on Qwen3-4B (61.1 avg) surpasses the Qwen3-8B base (60.7 avg).
• Large non-thinking gains (Table 7): With thinking mode disabled, TTPO improves over base by +25.2 (1.7B), +30.6 (4B), and +36.4 (8B) points on average, while OPSD improves by only +7.1, +5.8, and +3.5 points.
• Both token-level mechanisms contribute (Table 3): On Qwen3-1.7B OpenThoughts, removing positive-sample weighting drops AIME26 from 46.5 to 43.3 and HMMT26 from 31.6 to 30.6; removing negative-sample masking drops HMMT26 from 31.6 to 29.5 and BRUMO25 from 54.7 to 50.0.
• Asymmetric routing beats alternatives (Figure 3, AIME26 TTT): The full TTPO (pos=FKL, neg=GRPO) scores 48.9, ahead of positive-only FKL (46.7), all-FKL (46.3), negative-only FKL (43.9), GRPO on positives (37.2), and the reversed assignment (37.2).
• Pseudo-labels can outperform ground truth: In Figure 5, TTPO with pseudo-labels outperforms both TTPO w/ GT and OPSD (Leakage). The authors attribute this to ground truth starving the FKL branch (few or zero positives per instance) and to AIME26 ground-truth answers being short numbers that barely shift the thinking teacher.
• Answer with a thinking-mode teacher is the best privileged-information setting (Table 4): On Qwen3-1.7B, AIME26, an answer-conditioned thinking teacher scores 46.5 (39.8 non-thinking) versus 45.8 (36.1) for no privilege, 41.1 (8.9) for a full trajectory, and 33.6 (6.7) with a non-thinking teacher using an answer.
• RL weight matters (Table 8): Performance peaks at λ = 0.1 (46.5 AIME26, 31.6 HMMT26, 54.7 BRUMO25); both λ ≤ 0.05 and λ ≥ 0.15 degrade results.
• Selection details: A fixed 50% positive / 50% negative split (46.5 / 31.6 / 54.7) beats random and dynamic fractions, and the "shortest" K_train selection strategy ties the full method's numbers.
• Cross-task generalization (Figure 4): Training on any single benchmark consistently improves performance on the other two, indicating generalizable reasoning rather than problem-specific overfitting.

Methodology in Plain English

For each unlabeled problem, the model samples a group of K rollouts and extracts their final answers. Answers are clustered by mathematical equivalence, and the largest cluster becomes the pseudo-label. Rollouts are then split into those that agree with it (positive samples) and those that disagree (negative samples).

The two groups receive different treatment. Positive rollouts are distilled using OPSD: the same model is conditioned on the pseudo-label as a "reference solution" to act as a teacher, and it re-scores the student's own tokens. Crucially, since an agreeing rollout already produced that answer, even a wrong pseudo-label reduces the update to thinking-to-non-thinking distillation rather than pushing the model toward an arbitrary error. Negative rollouts instead receive a GRPO penalty, which uses only the fact of disagreement and never the pseudo-label's content.

Token-level selection sharpens both branches. In distillation, each token is scored by student entropy and teacher-student divergence, each min-max normalized to [0,1], and combined with a Soft-OR so that weight approaches zero only when both signals are low — the student is already confident and aligned with the teacher. In the RL branch, tokens are scored by negative log-probability multiplied by normalized certainty, and only the top 50% by score are penalized, which keeps the penalty on confidently wrong tokens rather than on locally correct arithmetic or formatting. The two branches are combined with a weight λ = 0.1 on the GRPO loss and optimized jointly.

Experiments use LoRA (r = 64, α = 128) on all linear layers of Qwen3-1.7B, Qwen3-4B, and Qwen3-8B, with AdamW, bfloat16 precision, and Flash Attention 2. Evaluation covers AIME 2025, AIME 2026, HMMT 2025, HMMT 2026, and BRUMO 2025, reported as Avg@12 at temperature 1.0 with thinking mode enabled.

Why This Matters

• Research impact: The paper provides a principled reason for using disagreement-based penalties under noisy pseudo-labels and shows that dense supervision can be made robust by decoupling positive and negative treatment. It also challenges the intuition that perfect labels are always better, reporting that ground-truth routing can starve the distillation branch on hard problems.
• Real-world applications:
  • Deploying reasoning models on novel mathematics or competition-style problems where no answer key exists, adapting at inference time to the problems actually being served.
  • Improving model performance in non-thinking, low-latency inference modes, where TTPO reports gains of +25.2 to +36.4 points across scales.
  • Reducing dependence on expensive human- or verifier-generated answer labels for domain adaptation to new problem distributions.
  • Increasing the effective capability of smaller models so that they match the base performance of roughly 2× larger untrained models in the reported settings.
• Industry relevance: The method requires no annotation pipeline, uses LoRA adapters, and was run on 4× H20 GPUs for TTPO versus 8× H20 GPUs for the other methods, which lowers the infrastructure barrier for label-free adaptation. The released code at https://github.com/ZJU-REAL/TTPO makes the recipe directly reusable.

Future Directions

• Extending beyond competition mathematics: All five benchmarks (AIME 2025, AIME 2026, HMMT 2025, HMMT 2026, BRUMO 2025) are competition-level math problems; whether the asymmetric objective holds for other verifiable domains such as code, formal proof, or multi-step agentic tasks is not established in the reported experiments.
• Scaling and cost of majority voting: The configuration samples K = 64 trajectories per problem with a maximum generation length of 16,000 tokens to make voting reliable on hard problems, and the paper does not report a systematic study of how small K can become before the asymmetry argument weakens.
• Alternative pseudo-label construction: The paper compares majority voting against ground truth but does not explore consensus-quality filters such as the hierarchical reward shaping of Hi-TTRL or the selective pseudo-labeling of SCRL, which it cites as related TTT work.
• Richer privileged information for the teacher: Table 4 shows a short answer works best with a thinking-mode teacher and a full trajectory works best with a non-thinking one; finding privileged signals that guide strongly without crowding out teacher reasoning remains open.

Target Audience

This paper is most useful to machine learning researchers and practitioners working on LLM post-training, reinforcement learning for reasoning, self-distillation, and test-time adaptation — particularly those who need to improve models on unlabeled problem sets where verification is unavailable. It also benefits engineers selecting label-free training recipes for reasoning systems and readers following the combination of outcome-reward RL with dense token-level distillation losses.

Authors’ abstract

Recent prominent post-training methods, such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD), have driven rapid progress in mathematical reasoning for large language models, yet their reliance on ground-truth labels precludes test-time training (TTT). Replacing ground truth with majority-vote pseudo-labels is a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token. We observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, we propose Test-Time Policy Optimization (TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL. Token-level selection further refines both branches: distillation down-weights already-converged positions, while RL penalizes only confident errors. Both updates remain well-grounded even under frequent pseudo-label errors, and majority-vote routing yields tighter self-supervision as the model improves. Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization.