CPMobius: Iterative Coach-Player Reasoning for Data-Free Reinforcement Learning

AI summary

Overview

Research area: Natural Language Processing, specifically reinforcement learning for large language model reasoning (mathematical problem solving) without human-curated training data.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning, policy optimization (GRPO, REINFORCE), and LLM post-training pipelines.

Scope: The paper proposes CPMöbius, a cooperative Coach–Player training loop in which two language models co-evolve to improve mathematical reasoning without any external training data during co-evolution, and evaluates it on six mathematical reasoning benchmarks across four base models.

What This Paper Is About

Improving LLM reasoning normally requires large amounts of human-written problems and labels, either through supervised fine-tuning or reinforcement learning with verifiable rewards. That dependence on human supervision is expensive and is showing signs of diminishing scalability. CPMöbius removes that dependence by having one model (the Coach) invent math tasks for another model (the Player), where the Coach is rewarded specifically for making the Player improve rather than for stumping it.

Key Contributions

1. A cooperative Coach–Player paradigm for data-free RL. Unlike adversarial self-play frameworks where one model tries to defeat another, CPMöbius treats the Coach and Player as independent but cooperative roles, inspired by multi-agent collaboration.

2. A progress-based reward for the Coach. The Coach receives the instruction reward R_i^Coach = R_i^Player · Δ_t, where R_i^Player is the Player's average training reward on instruction x_i and Δ_t is the change in the Player's accuracy on a fixed validation set after the update. Tasks are reinforced only when the Player both performs well on them and shows measurable overall learning progress.

3. A difficulty-filtered batching mechanism. Candidate tasks are kept only if the Player's rollout-dependent accuracy falls inside 0.2 ≤ acc_i ≤ 0.8; problems outside this range are discarded and resampled on the fly.

4. Empirical results across four base models and six benchmarks, with a codebase released at https://github.com/thunlp/CPMobius.

Main Findings

• CPMöbius beats other unsupervised RL methods. On Qwen2.5-Math-7B-Instruct the method improves overall average accuracy by +4.9 and out-of-distribution average by +5.4, exceeding RENT by +1.5 on overall accuracy and R-Zero by +4.2 on OOD accuracy. It achieves the highest overall average and OOD average scores across all four base models tested.

• Gains even on already-strong models. OpenMath-Nemotron-1.5B improves from 59.5 to 62.1 (the paper describes this as a 2.6-point overall improvement), even though it started from extensive SFT on 5.5 million instances. R-Zero failed on this model because it could not be trained as a Challenger as that method requires.

• Strong out-of-distribution generalization. On MATH-500, CPMöbius outperformed other methods with improvements ranging from 1.8 to 6.9 points over base models. The largest Minerva gains were from 16.3 to 28.0 (71.8%) on Qwen2.5-Math-1.5B and from 34.6 to 44.9 (29.8%) on Qwen2.5-Math-7B-Instruct.

• Improvement patterns vary by training stage. Qwen2.5-Math-1.5B (a math pre-training model) gained an overall 5.5 points (23.6% relative gain); Qwen2.5-Math-7B-Instruct gained 4.9 points; OctoThinker-3B-Hybrid-Zero gained a modest 2.3 points.

• Removing any core component hurts. Ablations on Qwen2.5-Math-1.5B: no Coach update drops average accuracy from 28.8% to 25.3% and OOD from 26.8% to 23.1%; no Coach warm-up drops to 23.7% average (OOD 21.2%); no instruction filter drops to 24.9% average (OOD 22.5%).

• A self-adjusting curriculum emerges naturally. Training dynamics show the Player's answer consistency on Coach-proposed tasks trending downward for the two models without prior RL training (Qwen2.5-Math-1.5B and OpenMath-Nemotron-1.5B), indicating progressively harder tasks. The Coach's proposed problems grow longer over training while the Player's responses grow shorter.

• Results are not specific to one validation environment. Replacing AMC with Minerva or OlympiadBench as the validation environment still produced consistent improvements on Qwen2.5-Math-1.5B and Qwen2.5-Math-7B-Instruct.

Methodology in Plain English

Two language models are set up as Coach and Player. The Coach generates a batch of m math task instructions aimed at the Player's current ability. The Player attempts each task n times; majority voting over those n attempts produces a pseudo-label, and each attempt is scored correct or incorrect against that pseudo-label. The Player is then updated with GRPO using these pseudo-label rewards.

During generation, each candidate task is screened: the Coach rolls out n Player answers, computes a majority-voted pseudo-label, and calculates the fraction of answers matching it. Only tasks where that accuracy falls between 0.2 and 0.8 make it into the batch, so tasks are hard enough to teach something but not so hard they are unsolvable.

After the Player updates, it is evaluated on a fixed held-out validation set (AMC in the main experiments), producing Δ_t, the accuracy change. The Coach's reward for each instruction is the Player's average reward on that instruction multiplied by Δ_t. The Coach is updated with REINFORCE over the batch of instruction-reward pairs, so it learns to generate tasks that cause real progress rather than tasks that merely look good. The Player uses GRPO with a KL trust region; the Coach never sees ground-truth solutions, only the scalar Δ_t signal.

Implementation details: the Coach is fixed to Qwen2.5-Math-7B-Instruct, warmed up with 4K PRIME Eurus-2-RL-Data. The authors are explicit that "data-free" refers only to the co-evolution stage after Coach–Player collaboration begins, not to any one-time model initialization beforehand. Training used verl on 4 to 8 NVIDIA A800-80GB GPUs per setting, with batch size 16 and 16 rollout samples per prompt. Evaluation used mean@32 for AIME, mean@10 for AMC, mean@6 for Minerva, mean@5 for MATH-500, and mean@3 for Olympiad-Bench.

Why This Matters

Impact on research: The paper argues that supervision-heavy training paradigms are becoming unsustainable, and demonstrates that a collaborative, progress-rewarded curriculum generator can substitute for human-curated data. It positions cooperation, rather than adversarial competition, as the more stable route to data-free self-improvement, and shows gains even on models already trained with SFT or RL.

Real-world applications:

• Training mathematical reasoning models in domains where labeled problems are scarce or expensive to produce.
• Continual improvement of deployed reasoning models without collecting new human annotations.
• Building domain-specific tutors or curriculum generators that adapt automatically to a learner's current level.
• Automatically generating practice problems calibrated to a solver's ability frontier.

Industry relevance: The approach reuses standard, widely deployed components (GRPO from DeepSeek, REINFORCE, the verl training framework) and requires no new human-labeled data pipeline, which lowers the cost of iterative model improvement. It also offers an alternative to entropy-minimization methods like RENT and adversarial self-play like R-Zero, both of which serve as baselines here.

Future Directions

• Applying the collaborative co-evolving paradigm to domains beyond mathematics, as the authors suggest for future work.
• Studying the emergent behaviors and long-term stability of interactions between co-evolving models, which the paper identifies as a promising direction.
• Investigating the Coach–Player loop when the Coach is not warmed up or is selected differently, given that removing warm-up caused the largest ablation degradation (average accuracy 23.7% versus 28.8%).
• Understanding why R-Zero failed to train on OpenMath-Nemotron-1.5B while CPMöbius succeeded, which may reveal broader conditions for when adversarial versus cooperative co-evolution is viable.

Target Audience

Researchers and engineers working on LLM reasoning, reinforcement learning for language models, and unsupervised or data-free training methods. It is most useful to readers already comfortable with policy-gradient methods, verifiable-reward RL, and LLM post-training, and to practitioners looking for a training recipe that does not depend on human-curated problem sets.

Authors’ abstract

Large Language Models (LLMs) have demonstrated strong potential in complex reasoning, yet their progress remains fundamentally constrained by reliance on massive high-quality human-curated tasks and labels, either through supervised fine-tuning (SFT) or reinforcement learning (RL) on reasoning-specific data. This dependence renders supervision-heavy training paradigms increasingly unsustainable, with signs of diminishing scalability already evident in practice. To overcome this limitation, we introduce CPMöbius (CPMobius), a collaborative Coach-Player paradigm for data-free reinforcement learning of reasoning models. Unlike traditional adversarial self-play, CPMöbius, inspired by real world human sports collaboration and multi-agent collaboration, treats the Coach and Player as independent but cooperative roles. The Coach proposes instructions targeted at the Player's capability and receives rewards based on changes in the Player's performance, while the Player is rewarded for solving the increasingly instructive tasks generated by the Coach. This cooperative optimization loop is designed to directly enhance the Player's mathematical reasoning ability. Remarkably, CPMöbius achieves substantial improvement without relying on any external training data, outperforming existing unsupervised approaches. For example, on Qwen2.5-Math-7B-Instruct, our method improves accuracy by an overall average of +4.9 and an out-of-distribution average of +5.4, exceeding RENT by +1.5 on overall accuracy and R-zero by +4.2 on OOD accuracy. Our codebase has been released at https://github.com/thunlp/CPMobius.