How Far Can Unsupervised RLVR Scale LLM Training?

AI summary

Overview

• Research area: Reinforcement learning for large language models, specifically reinforcement learning with verifiable rewards (RLVR) and its unsupervised variant (URLVR), where rewards are derived without ground-truth labels.
• Technical level: Advanced. The paper combines a taxonomy of methods, a formal theoretical analysis using KL-regularized RL objectives and a convergence theorem, and a large empirical study spanning many model families and reward designs.
• Scope (one sentence): The paper asks whether unsupervised RLVR can scale LLM training, and answers by classifying unsupervised reward methods, proving that intrinsic rewards merely sharpen a model's initial distribution, and empirically demonstrating an inevitable rise-then-fall training pattern.

What This Paper Is About

Standard RLVR improves LLM reasoning by rewarding outputs that can be checked against ground truth, such as a final numerical answer in a math problem or a passing code execution. This creates a supervision bottleneck: as models surpass human expertise, obtaining reliable labels becomes infeasible. The paper investigates Unsupervised RLVR (URLVR), which derives rewards without ground-truth labels, and asks whether such rewards can actually scale training or whether they only produce early gains that later collapse.

Key Contributions

1. A taxonomy of URLVR methods. The paper classifies unsupervised RLVR into intrinsic rewards (derived from the model itself, split further into certainty-based and ensemble-based) and external rewards (derived from outside the model, split into unlabeled-data methods and generation-verification asymmetry methods).
2. A unified theoretical framework. Despite their diverse designs, all intrinsic reward methods are shown to converge toward sharpening the model's initial distribution, amplifying existing preferences rather than discovering new knowledge. Theorem 1 states that under stated assumptions the probability mass on the majority answer converges geometrically to 1 with rate ρ = e^(−1/β).
3. Systematic empirical characterization of failure. Across five intrinsic reward methods and tuned hyperparameters, intrinsic URLVR consistently follows a rise-then-fall pattern, with collapse timing determined by model prior rather than engineering choices. Three distinct failure modes are identified: gradual degradation, length collapse, and repetition collapse.
4. Practical and forward-looking tools. The paper proposes Model Collapse Step as an indicator of model prior and RL trainability, argues intrinsic rewards remain useful for test-time training on small, domain-specific datasets, and presents preliminary evidence that external reward methods based on computational asymmetries may escape the confidence-correctness ceiling.

Main Findings

• All intrinsic rewards sharpen the initial distribution: Certainty-based rewards (Self-Certainty, Trajectory-Level Entropy, Token-Level Entropy, Probability, Probability Disparity) and ensemble-based rewards (majority voting and its extensions) are all shown to be different formalizations of rewarding high-confidence outputs; the paper's unified framework reduces them to manipulating cross-entropy between carefully chosen distributions.
• Success depends on confidence-correctness alignment: The sharpening mechanism works as a beneficial amplifier when the model's initial confidence aligns with correctness, and systematically reinforces errors when it does not.
• A universal rise-then-fall pattern: Training with the TTRL majority voting reward on Qwen3-1.7B-Base matched or exceeded ground-truth training on AIME 2024, AIME 2025 and AMC 2023 during the early phase, but the proxy Majority Voting Reward kept rising while Reward Accuracy and validation performance declined, exhibiting reward hacking.
• Collapse is not an engineering artifact: After tuning four key hyperparameters (training temperature, mini-batch size, KL regularization and rollout number) across five intrinsic reward methods, nearly all settings eventually degrade. Even the most stable combined configuration collapsed around 1,000 steps, roughly 4 epochs.
• Different methods fail differently: Self-Certainty and Majority Voting degrade gradually while maintaining label accuracy; Probability collapses toward brevity because it multiplies token probabilities and favors shorter sequences; the entropy-based methods (Token-Level Entropy, Trajectory-Level Entropy) drive entropy down through repetition rather than correctness, since averaging entropy across tokens rewards padding with high-probability tokens.
• Training amplifies rather than corrects within a problem: In per-problem experiments training Qwen3-1.7B-Base on 25 randomly sampled individual problems from MATH500, training amplified the model's existing preferences rather than correcting errors on the same problem.
• Amplification can still generalize: Even when training amplifies errors on one problem, it can still correct an out-of-distribution problem, showing that confidence-correctness alignment varies across problems.
• Intrinsic rewards retain value in test-time training: On small and domain-specific datasets, intrinsic rewards avoid collapse even when all initial preferences are wrong, making them well-suited to test-time training.
• Model Collapse Step as a prior indicator: The timing of collapse in the rise-then-fall pattern can be used to measure model prior and predict RL trainability without expensive training runs.
• External rewards avoid the collapse pattern: Taking self-verification as an example, external methods that generate verifiable rewards through generation-verification asymmetries rather than internal model states show sustained improvement without the collapse inherent to intrinsic methods.
• External rewards scale through two mechanisms: Unlabeled-data methods derive rewards that grow with corpus size rather than model capability, and generation-verification asymmetry methods ground rewards in external computation (compilers, proof assistants, game engines, simulators) whose reliability does not degrade as the model improves.

Methodology in Plain English

The researchers first organize the sprawling literature on unsupervised RLVR into a taxonomy based on where the reward comes from: inside the model (intrinsic) or outside it (external). They then analyze the intrinsic case mathematically, using the standard KL-regularized reinforcement learning objective and its known closed-form optimal policy. Taking TTRL's majority voting reward as a representative example, they show that a single update amplifies the probability of trajectories leading to the majority answer by a factor of e^(1/β), leading to a "rich-get-richer" dynamic. They prove that under two assumptions — majority stability and effective learning — the policy converges geometrically to a deterministic policy concentrated on the initial majority answer.

To test this theory, they train Qwen3-1.7B-Base on DAPO-17k with the TTRL majority voting reward and compare it against training with ground-truth labels, evaluating on AIME 2024, AIME 2025 and AMC 2023, generating 32 solutions per problem at temperature 0.6 with top-p 0.95 and reporting average accuracy (avg@32). They track the majority voting reward, reward accuracy, actor entropy, label accuracy and mean response length. They then repeat this across five intrinsic reward methods with separately tuned hyperparameters, and finally zoom in to individual problems from MATH500, training on 25 randomly sampled problems with REINFORCE to observe per-problem behavior.

Why This Matters

• Impact on research: The paper reframes the debate about unsupervised RLVR from a patchwork of method comparisons to a single mechanistic explanation. It provides a theoretical reason why intrinsic rewards cannot keep improving a model indefinitely, and it offers a practical diagnostic (Model Collapse Step) that researchers can use to check whether a model is worth running RL on before spending compute.

• Real-world applications:

  • Test-time training for reasoning models, where a model is adapted on a small domain-specific set of problems at deployment time, a setting where the paper finds intrinsic rewards avoid collapse.
  • Mathematics and competition problem training, the primary evaluation setting in the paper, where verifiable rewards are already standard.
  • Formal theorem proving, where proof checkers in systems like Lean provide verification that does not hallucinate.
  • Code generation and software engineering, where executing a program against test cases is deterministic and instantaneous even when writing the program is hard.
  • Scientific simulation domains, where numerical simulators and puzzle rules can serve as automatic verifiers.

• Industry relevance: Supervised RLVR requires large volumes of human-labeled verifiable data, which is expensive and increasingly infeasible as models surpass human expertise. The paper's conclusion that external reward methods are the more promising direction for long-run scaling gives labs a concrete investment target: building diverse verifiable environments and extending verification asymmetries into new scientific domains, rather than continuing to tune intrinsic reward hyperparameters.

Future Directions

1. Building diverse verifiable environments: The paper highlights developing varied verification setups as a key open challenge, since external rewards depend on having a procedure that can check candidate solutions reliably.
2. Extending verification asymmetries to new scientific domains: Moving beyond mathematics, code and formal proofs to fields where a cheap checker has not yet been established.
3. Exploring self-supervised signals from vast unlabeled corpora: Unlabeled-data methods such as next-token prediction rewards and dual reconstruction objectives provide a reward landscape that expands with data rather than with model capability.
4. Determining whether external methods truly escape the confidence-correctness ceiling: The paper describes the evidence for external reward methods escaping collapse as preliminary, leaving a fuller scaling characterization open.

Target Audience

Researchers and engineers working on post-training and reinforcement learning for large language models, particularly those applying or building RLVR pipelines. It is also valuable for graduate students studying the theory of policy optimization, for practitioners deciding whether to invest in test-time training versus full RL training runs, and for research groups looking for a structured map of the URLVR literature and its open problems.

Authors’ abstract

Unsupervised reinforcement learning with verifiable rewards (URLVR) offers a pathway to scale LLM training beyond the supervision bottleneck by deriving rewards without ground truth labels. Recent works leverage model intrinsic signals, showing promising early gains, yet their potential and limitations remain unclear. In this work, we revisit URLVR and provide a comprehensive analysis spanning taxonomy, theory and extensive experiments. We first classify URLVR methods into intrinsic versus external based on reward sources, then establish a unified theoretical framework revealing that all intrinsic methods converge toward sharpening the model's initial distribution This sharpening mechanism succeeds when initial confidence aligns with correctness but fails catastrophically when misaligned. Through systematic experiments, we show intrinsic rewards consistently follow a rise-then-fall pattern across methods, with collapse timing determined by model prior rather than engineering choices. Despite these scaling limits, we find intrinsic rewards remain valuable in test-time training on small datasets, and propose Model Collapse Step to measure model prior, serving as a practical indicator for RL trainability. Finally, we explore external reward methods that ground verification in computational asymmetries, showing preliminary evidence they may escape the confidence-correctness ceiling. Our findings chart boundaries for intrinsic URLVR while motivating paths toward scalable alternatives.