Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning

AI summary

Overview

Research area: Natural Language Processing / large language model reasoning, specifically test-time reinforcement learning (TTRL) and unsupervised reward-signal construction.

Technical level: Intermediate. The paper builds directly on Group Relative Policy Optimization (GRPO) and the TTRL framework, and assumes familiarity with pseudo-labeling, policy optimization, and token-level probability metrics.

Scope: The paper proposes SCOPE, a TTRL framework that replaces majority-voting pseudo-labels with step-wise confidence-weighted voting and dynamic subgroup partitioning, and evaluates it on four mathematical reasoning benchmarks across four models.

What This Paper Is About

Test-time reinforcement learning lets a language model improve its reasoning without any human-labeled answers by sampling many responses to the same question and treating the majority-voted answer as a fake "correct" label. This paper argues that majority voting is a poor label estimator: it treats every vote as equally trustworthy, which can lock in the model's own mistakes (confirmation bias) and produces only a binary, sparse reward signal.

The goal is to build a finer-grained and more reliable reward signal for unsupervised test-time RL by weighting votes with the model's own step-level confidence, and by splitting the response pool into subgroups that each get their own local consensus label.

Key Contributions

1. SCOPE framework. A test-time reinforcement learning framework that combines step-wise confidence weighting with subgroup-specific pseudo-label estimation to address confirmation bias and sparse rewards in unsupervised RL.

2. Step-wise confidence for label estimation. The authors introduce an average step confidence metric computed over newline-delimited reasoning steps, which is used as a weight in consensus voting so that high-certainty reasoning paths can override simple frequency counts and recover correct answers that appear in the minority.

3. Pareto-optimized subgroup partitioning. Responses are partitioned into subgroups of size m, each receiving its own bootstrap-sampled, confidence-weighted consensus label. The optimal subgroup size is selected automatically during training by optimizing a trade-off between a quality rate and an exploration rate over a Pareto front.

4. Empirical validation across model scales and benchmarks. Experiments on Qwen2.5-Math-1.5B, Qwen3-1.7B, Qwen3-8B, and LLaMA3.1-8B-Instruct on AIME 2024, AIME 2025, AMC, and MATH-500 show consistent gains over baselines, with the abstract reporting relative improvements of 13.1% on AIME 2025 and 8.1% on AMC. Code is released at https://github.com/szu-tera/SCOPE.

Main Findings

• Consistent gains over TTRL across all four models. On Qwen2.5-Math-1.5B, SCOPE reaches an average of 41.36 versus TTRL's 36.95 (absolute margin of 4.41, +11.9%). On Qwen3-1.7B, SCOPE reaches 44.02 versus TTRL's 41.91. On LLaMA3.1-8B-Instruct, SCOPE reaches 28.18 versus TTRL's 26.38. On Qwen3-8B, SCOPE reaches 62.20 versus TTRL's 58.21 (+6.9%).

• Largest gains on competition-level benchmarks. On Qwen3-8B, SCOPE improves AIME 2024 from 47.13 (TTRL) to 52.70 (+5.57, +11.8%), AIME 2025 from 27.40 to 31.00 (+3.60, +13.1%), and AMC from 68.55 to 74.09 (+5.54, +8.1%). The introduction describes the Qwen3-8B AIME 2024 improvement as 10.48% with an average gain of 6.85% across all benchmarks; Table 1 and Section 4.2 report +5.57 (+11.8%) on AIME 2024 and a 62.20 average (+6.9%).

• Small models benefit substantially. On Qwen2.5-Math-1.5B, SCOPE raises AIME 2024 from 16.48 (TTRL) to 22.50 (a 36.5% relative improvement) and AIME 2025 from 9.86 to 14.90 (51.1%). On LLaMA3.1-8B-Instruct, SCOPE improves AIME 2024 from TTRL's 9.56 to 14.37, a 50.3% relative gain.

• One regression on an easier benchmark. On LLaMA3.1-8B-Instruct MATH-500, SCOPE scores 61.67 versus TTRL's 62.93, a 2.0% relative decrease. The authors attribute this to SCOPE prioritizing complex reasoning over simpler tasks.

• Ablation confirms both components matter. Removing confidence weighting (w/o Conf) drops Qwen3-8B on AIME 2024 from 52.70 to 47.70 (-5.00) and Qwen2.5-Math-1.5B on AIME 2025 from 14.90 to 11.77 (-3.13). Removing subgroup partitioning (w/o Subgroup) drops Qwen2.5-Math-1.5B on AIME 2024 from 22.50 to 16.67 (-5.83) and Qwen3-8B on AIME 2025 from 31.00 to 26.92 (-4.08).

• Trade-off parameter behavior. On Qwen3-8B, setting λ = 1.0 (consensus quality only) still beats TTRL but saturates at 51.66% on AIME 2024. Peak performance occurs at λ = 0.5 with 53.75% on AIME 2024 and 31.0% on AIME 2025. Setting λ = 0 causes a noticeable decline from over-exploration. The experiments in Section 3.3 use λ = 0.7, with the parameter analysis in Section 5.1.

• Step-level confidence beats other confidence granularities. Compared with average trace confidence, bottom-10% confidence, and tail-10% confidence on Qwen2.5-Math-1.5B, step-wise confidence is best, achieving a 36.5% relative improvement on AIME 2024 and 51.1% on AIME 2025. The bottom-10% strategy collapses (+0.0%).

• Automatic subgroup selection outperforms fixed sizes. In training-dynamics analysis, m = 1 (atomized) and m = 8 subgroups grow quickly but saturate early at suboptimal levels, while m = 64 (global) is stable but converges more slowly. Automatic selection leads on both convergence speed and peak performance.

• Subgroup partitioning reduces consensus drift. Pseudo-label Accuracy (PLA) against ground-truth labels is 25.42 for TTRL, 26.75 for SCOPE with m = 1 (+5.24%), 26.84 with m = 8 (+5.59%), 29.57 with m = 64 (+16.33%), and 30.71 for automatic SCOPE (+20.81%).

Methodology in Plain English

The approach has four moving parts, applied at every training iteration:

1. Generate and score. For a given question, the policy model samples a pool of candidate responses. Each response is split into reasoning steps using the newline character as a delimiter. Each step gets a confidence score equal to the average negative log-probability of its tokens (the top-k formulation from prior work), and each response gets an average step confidence across its steps.

2. Weighted voting for labels. Instead of counting votes, the system sums the average step confidence of responses that reach each candidate answer and picks the answer with the highest total weight. This lets a high-confidence minority answer beat a low-confidence plurality.

3. Partition into subgroups. The pool of responses is divided into n subgroups of size m. For each subgroup, a separate consensus label is derived by bootstrap sampling from the global response pool and applying the confidence-weighted vote. Each response is then rewarded based on whether its answer matches its own subgroup's label, which produces denser and more varied reward signals than one global label.

4. Pick the subgroup size automatically. The authors evaluate multiple candidate subgroup sizes by two metrics: quality rate q (how often outputs match their subgroup consensus) and exploration rate e (the proportion of unique consensus labels across subgroups). Candidate (q, e) points form a Pareto front, normalized, and scored by a weighted trade-off distance with parameter λ. The subgroup size minimizing this distance is used for that iteration.

Training uses GRPO with these computed rewards. Implementation uses the Volcano Engine Reinforcement Learning for LLMs framework, AdamW with a cosine learning rate schedule peaking at 5 × 10⁻⁷, 64 sampled responses per prompt at temperature 0.6 (1.0 for Qwen2.5-Math), 32 bootstrap samples per subgroup, and a maximum generation length of 3,072 tokens. Evaluation follows the pass@1 protocol with 16 responses per question (4 for 32k-context models) at temperature 0.6 and top-p 0.95.

Why This Matters

Impact on research. The paper targets the weakest link in unsupervised RL for LLMs: the quality of the pseudo-label that stands in for a ground-truth answer. It shows that treating confidence as a vote weight, and treating reward computation as a granularity choice rather than a fixed global consensus, produces measurable gains on competition-level math. The Pseudo-label Accuracy analysis offers a way to measure consensus drift directly, which is useful for anyone studying self-training stability.

Real-world applications:

• Deploying reasoning models in domains with no labeled answers, such as internal code repositories or proprietary document collections, where a model must improve from its own outputs.
• Cost reduction in RL post-training pipelines, since the method removes the need for ground-truth labels that typically require expert annotators.
• Adapting small or open-weight models to hard reasoning tasks, where the paper demonstrates the largest relative gains (36.5% and 50.3% on AIME 2024).
• Test-time self-improvement on edge or on-premise deployments, where a model fine-tunes on unlabeled user queries rather than shipping data to a labeling pipeline.

Industry relevance. The method is built on GRPO and a public RL training framework, so it is a drop-in modification to existing post-training stacks rather than a new architecture. The reported ~10% computational overhead from Pareto-optimal subgroup calculation is modest relative to the accuracy and sample-efficiency gains the authors claim.

Future Directions

• Replacing heuristic step segmentation. The paper's limitations section notes that steps are defined by newline characters, which assumes a structured generation format. A more principled or learned segmentation would extend the method to outputs that do not follow the standard reasoning format.
• Reducing the Pareto optimization overhead. The dynamic subgroup size selection costs roughly 10% extra computation; more efficient test-time training dynamics are named as a direction.
• Extending to long-horizon and interactive agent settings. The authors propose applying the paradigm beyond single-question math reasoning to agentic tasks.
• Calibrating confidence further. The paper contrasts step-wise confidence against average trace, bottom-10%, and tail-10% aggregation, and shows the bottom-10% strategy collapses. Finding aggregation schemes that avoid both error masking and over-weighting weak steps remains open.

Target Audience

Researchers and engineers working on LLM post-training, RLVR, and unsupervised or self-supervised reinforcement learning. It is most useful to those already familiar with GRPO and test-time training pipelines who want to improve pseudo-label reliability without adding labeled data. Practitioners deploying reasoning models on math or competition-style problems will find the benchmark tables and the ablation on confidence granularity directly actionable. Readers without background in policy optimization will need to consult the cited GRPO and TTRL papers first.

Authors’ abstract

Test-time reinforcement learning mitigates the reliance on annotated data by using majority voting results as pseudo-labels, emerging as a complementary direction to reinforcement learning with verifiable rewards (RLVR) for improving reasoning ability. However, this voting strategy often induces confirmation bias and suffers from sparse rewards, limiting the overall performance. In this work, we propose subgroup-specific step-wise confidence-weighted pseudo-label estimation (SCOPE), a framework integrating model confidence and dynamic subgroup partitioning to address these issues. Specifically, SCOPE integrates the proposed step-wise confidence into pseudo label estimation, prioritizing high-quality reasoning paths over simple frequency count. Furthermore, it dynamically partitions the candidate outputs pool into independent subgroups by balancing reasoning quality against exploration diversity. By deriving local consensus via repeat sampling for each sub group, SCOPE provides diverse supervision targets to encourage broader exploration. We conduct experiments across various models and benchmarks, experimental results show that SCOPE consistently outperforms recent baselines. Notably, SCOPE achieving relative improvements of 13.1% on challenging AIME 2025 and 8.1% on AMC. The code is released at https://github.com/szu-tera/SCOPE.