You Need Reasoning to Learn Reasoning: The Limitations of Label-Free RL in Weak Base Models

AI summary

Overview

Research area: Unsupervised ("label-free") reinforcement learning for improving reasoning in large language models, evaluated on mathematical and science reasoning benchmarks.

Technical level: Intermediate. Comfort with reinforcement learning basics (policy optimization, reward signals, group-relative advantages) helps, but the paper's core argument is intuitive and the training modifications are simple.

Scope: A systematic empirical study of how label-free RL methods (TTRL and Intuitor) behave across Qwen2.5 models from 0.5B to 7B parameters, plus a proposed curriculum-based fix called CuMa.

What This Paper Is About

Label-free RL methods promise to improve a model's reasoning without any human labels or ground-truth answers, typically by using majority voting over the model's own sampled answers as a reward signal. The problem is that these methods have mostly been tested on large models that are already strong reasoners. This paper asks what happens when you apply them to smaller, weaker base models, finds that they frequently fail or even collapse, and proposes a training recipe that makes them work across model sizes.

Key Contributions

1. A systematic failure analysis of label-free RL across model scale. The authors compare TTRL and Intuitor against a verifier-based GRPO baseline and a base model on Math 500, GPQA, AIME24, GSM8K, and LiveCodeBench (LCB) for Qwen2.5 models at 0.5B, 1.5B, 3B, and 7B parameters, all reproduced in an identical training setup.

2. Identification of why weak models fail. The paper traces collapse to the absence of correct rollouts early in training (which makes majority voting produce wrong pseudo-labels), to the failure of weak models to produce longer chains of thought, and to a sensitivity to training-data difficulty.

3. CuMa (Curriculum-guided Masked Majority Voting Reinforcement Learning). A method that partitions unlabeled data into K = 5 difficulty bins and trains sequentially from easiest to hardest, masks the learning signal on prompts where no majority consensus exists among the N sampled candidates, and augments training with synthetic problems of predefined difficulty.

4. A data curation pipeline. An LLM-based generator that produces unlabeled reasoning problems at labeled difficulty levels 1 through 5, using dynamically refreshed in-prompt examples to avoid overfitting to a fixed pattern and generating batches of 25 samples per iteration.

Main Findings

• Label-free RL depends heavily on the base model's existing reasoning ability. The Qwen2.5-7B base model reaches 58.2 on Math 500, and the paper reports this rising to 74.6 with TTRL and 72.2 with Intuitor, comparable to the 73.8 achieved by verifier-based GRPO (Table 1 lists 73.6 for TTRL on Math 500). The Qwen2.5-0.5B base model reaches 23.4 on Math 500, declines under Intuitor, and completely collapses to 0.0 under TTRL. Similar degradation appears on the other benchmarks.

• Collapse is caused by incorrect pseudo-labels. The authors plot the number of correct rollouts in early training for Qwen2.5-0.5B and find that rollouts often contain no correct output at all. Majority voting then produces an incorrect pseudo-label, and training on it destroys the model.

• Weaker models do not develop long chains of thought. The reasoning CoT length for Qwen2.5-7B increases as training progresses, which prior work associates with self-reflection (the "Aha moment"). The weaker model does not exhibit such long chains. The paper also notes that length alone is not a reliable indicator: Qwen 0.5B and 1.5B show similar reasoning length even though the 1.5B variant performs much better.

• Training-data difficulty is decisive. Shifting training data from easier subsets (Level 1–2) to harder or less aligned subsets (Level 1–5) degrades performance on Math 500, GPQA, and GSM8K. On Math 500 the drop is sharp, with the model nearly collapsing by Level 1–4. Weak base models cannot learn from problems where they rarely, if ever, produce the correct solution.

• CuMa improves results across all model sizes and avoids collapse. On 0.5B it reaches 32.8 on Math 500 and 32.9 on GSM8K; on 1.5B, 54.2 and 59.96; on 3B, 64.4 and 72.85; on 7B, 74.0 and 84.49. The paper reports no model collapse on any dataset or scale, with gains most pronounced for smaller models.

• Every component matters. An ablation on Math 500 with Qwen2.5-0.5B gives 32.8 for the full method, 30.7 without reward masking, 24.5 without curated data, and 20.1 without curriculum learning.

Methodology in Plain English

The authors take a base language model and give it math or reasoning problems with no answer key. For each problem the model generates several candidate solutions, and the answer that appears most often is treated as the "correct" one, following the TTRL setup. A policy optimization algorithm (GRPO) then rewards the model for producing answers that match this majority vote. A second method, Intuitor, is also tested; instead of majority voting it uses the model's own confidence (self-certainty) as the reward.

Against this background, CuMa makes three changes. First, curriculum learning: the unlabeled problems are sorted into K = 5 difficulty bins, and training proceeds from bin 1 (easiest) to bin 5 (hardest), so the model builds basic reasoning before meeting harder problems. Second, reward masking: when the sampled answers for a prompt have no majority consensus, the training signal for that prompt is masked out entirely instead of being assigned an arbitrary reward, which prevents learning from noisy feedback. Third, data curation: an LLM generates additional unlabeled problems at a requested difficulty level, with the illustrative examples in the generator prompt refreshed each iteration, producing batches of 25 samples per iteration.

Implementation details reported in the appendix: GRPO is applied independently on each benchmark, with a cosine learning rate schedule peaking at 3×10⁻⁶ and the AdamW optimizer; 8 candidate responses per prompt at temperature 0.6; maximum generation length 3,072 tokens for all models; 1 training episode; runs on 4 NVIDIA H100 80GB GPUs.

Why This Matters

The paper pushes back on an implicit assumption in the label-free RL literature: that these methods are general-purpose. The evidence here suggests they are closer to a way of eliciting reasoning a model already has than a way of creating it, which reframes what "unsupervised self-improvement" can realistically claim.

Real-world applications:

• Deploying small models on-device or at the edge, where 0.5B–3B models are the practical option and labeled training data for a target domain may not exist.
• Mathematical and quantitative tutoring tools that need step-by-step reasoning from a compact, low-cost model rather than a frontier API.
• Code generation and program synthesis assistants, since LiveCodeBench is one of the evaluated benchmarks and the same reward-free setting applies.
• Domain adaptation in data-scarce settings such as scientific or enterprise reasoning tasks, where answer keys are expensive but unlabeled problem statements can be generated at scale.

Industry relevance: the practical takeaway is a set of cheap training modifications (difficulty ordering and reward masking) that prevent small models from degrading during unsupervised RL, plus a synthetic data pipeline that removes reliance on curated labeled corpora. The code is released at https://github.com/BorealisAI/CuMa.

Future Directions

• Quantifying the reasoning threshold. The paper shows 0.5B models fail while 7B models succeed, but does not characterize where in between the boundary lies or what property of a base model predicts success.
• Richer signals than majority voting. Reward masking handles prompts with no consensus by discarding them; an alternative would be to extract a usable learning signal from these inconclusive rollouts rather than ignoring them.
• Difficulty estimation without labels. CuMa relies on a data curation pipeline that assigns difficulty levels at generation time; automatically binning existing unlabeled corpora by difficulty remains open.
• Extension beyond math and code. The evaluated benchmarks all have verifiable or near-verifiable answers, so whether the curriculum and masking benefits transfer to open-ended reasoning is untested.

Target Audience

Researchers and practitioners working on reinforcement learning for language models, especially those training small or mid-size models (0.5B–7B) with limited labeled data. It is also useful for engineers deciding whether unsupervised RL is worth applying to a given base model, and for anyone studying the relationship between model scale, chain-of-thought behavior, and trainability. Readers without RL background can still follow the empirical findings and the curriculum/masking recipe.

Authors’ abstract

Recent advances in large language models have demonstrated the promise of unsupervised reinforcement learning (RL) methods for enhancing reasoning capabilities without external supervision. However, the generalizability of these label-free RL approaches to smaller base models with limited reasoning capabilities remains unexplored. In this work, we systematically investigate the performance of label-free RL methods across different model sizes and reasoning strengths, from 0.5B to 7B parameters. Our empirical analysis reveals critical limitations: label-free RL is highly dependent on the base model's pre-existing reasoning capability, with performance often degrading below baseline levels for weaker models. We find that smaller models fail to generate sufficiently long or diverse chain-of-thought reasoning to enable effective self-reflection, and that training data difficulty plays a crucial role in determining success. To address these challenges, we propose a simple yet effective method for label-free RL that utilizes curriculum learning to progressively introduce harder problems during training and mask no-majority rollouts during training. Additionally, we introduce a data curation pipeline to generate samples with predefined difficulty. Our approach demonstrates consistent improvements across all model sizes and reasoning capabilities, providing a path toward more robust unsupervised RL that can bootstrap reasoning abilities in resource-constrained models. We make our code available at https://github.com/BorealisAI/CuMa