Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability

AI summary

Authors: Shobhita Sundaram (MIT; work done during an internship at Meta), John Quan (Meta FAIR), Ariel Kwiatkowski (Meta FAIR), Kartik Ahuja (Meta FAIR), Yann Ollivier (Meta FAIR), Julia Kempe (Meta FAIR and New York University) Venue: arXiv:2601.18778v3 [cs.LG], published 2026-01-26 License: CC BY-NC-SA 4.0

Overview

• Research area: Reinforcement learning for large language model reasoning, specifically reinforcement learning with verifiable rewards (RLVR), automated curriculum learning, and asymmetric self-play / meta-RL.
• Technical level: Intermediate. The core idea is intuitive, but the method involves bilevel optimization, RLOO, and meta-RL terminology.
• One-sentence scope: The paper asks whether a pretrained LLM can generate its own training curriculum of easier questions to escape a reasoning plateau on problems it cannot solve at all.

What This Paper Is About

RLVR methods stall when a dataset is so hard that the model almost never earns a reward, leaving essentially no training signal. The authors ask whether a pretrained LLM already holds latent knowledge that lets it generate useful "stepping stone" problems for a difficulty level it cannot currently solve, without any human-curated intermediate data. They propose SOAR (Self-Optimization via Asymmetric RL), a self-play framework in which a teacher model writes synthetic question-answer pairs and is rewarded only when those pairs measurably improve a student model's performance on the real hard problems.

Key Contributions

1. Decoupled teaching and solving. The paper shows that a model's ability to generate effective stepping stones for hard problems is distinct from its ability to solve those problems. The base model has the latent capacity to propose useful questions, and meta-RL sharpens that noisy distribution into a reliable learning signal.
2. A proof-of-concept grounded self-generated curriculum (SOAR). SOAR is an asymmetric self-play framework that rewards the teacher for measured student progress on real hard problems. On hard subsets of MATH and HARP, self-generated problems improve performance — reported as 4× pass@1 and 2× pass@32 on MATH, and 2× pass@1 and 1.5× pass@32 on HARP — and the questions also transfer to datasets they were not optimized for.
3. Grounded rewards beat intrinsic rewards. Grounding teacher rewards in student progress on real problems outperforms the intrinsic learnability-style rewards used in prior LLM self-play work, which the authors find are prone to instability and collapse of question diversity.
4. Question structure matters more than solution correctness. Problem structure and difficulty calibration are more critical for escaping plateaus than answer correctness; generated questions provide useful gradient signal even when most answers are wrong.

Main Findings

• Hard problems give no signal. The study targets the hardest subsets of math benchmarks where the model has a 0/128 success rate, called fail@128 datasets, and each is randomly split 50-50 into train and test sets.
• SOAR beats direct training. Inference with the promoted student (PS) achieves +8.5% pass@32 on fail@128-MATH and +3.6% pass@32 on fail@128-HARP over the Hard-Only baseline. Training a fresh student on the Promotion Questions (PQ) achieves +9.3% pass@32 on MATH and +4.2% on HARP — higher mean performance than PS, indicating the synthetic questions themselves, not a lucky student trajectory, drive the gains.
• Baselines lag. Both the Intrinsic Teacher (Intrinsic-T) baseline and SeRL, a contemporary self-play method trained with the MATH and HARP fail@128 train sets as seed sets, perform worse across datasets, supporting the claim that grounded rewards are needed to discover the right questions.
• Extra compute on hard data does not substitute. Hard-Only trained with a group size of 128 (4× extra compute) achieves only +2.8% pass@32, and extending Hard-Only training from 1500 to 6500 steps does not improve performance. Reference Hard-Only MATH pass@k for k ∈ {1,4,8,16,32} is {0.5, 1.7, 3.2, 5.7, 9.6}.
• Transfer to out-of-distribution data. Synthetic questions from PQ-MATH, PQ-HARP, and Intrinsic-T all transfer to OlympiadBench fail@128, a held-out dataset on which no out-of-distribution optimization was performed.
• A model can teach what it cannot solve. Direct inference on fail@128 test problems with the final trained teacher policy does not improve over base model performance, indicating generator and solver abilities are largely independent.
• Synthetic data recovers much of a curated-data upper bound. Compared against training on fail@128 plus the full official MATH training set (6750 problems) as an "oracle" of abundant easier questions, PQ-MATH questions recover 75% of the performance gains from full-MATH training and PQ-HARP recovers 50%. HARP-PQ (128/192 questions) outperforms 128 real HARP questions and matches 128 real MATH questions.
• Grounded teachers are more stable across seeds. Students trained with questions from different Grounded-T seeds follow highly similar trajectories. Intrinsic-T teachers produce worse and more volatile outcomes, and one out of three teacher seeds exhibits collapse across all datasets.
• Diversity is preserved by grounding. Measured with the Vendi Score using Qwen3-8B embeddings (standardized to 128 questions via bootstrap subsampling with k = 100 iterations), base model diversity is 34.91 (σ = 1.74), Grounded-T (HARP) 34.66 (σ = 1.74), Grounded-T (MATH) 31.99 (σ = 1.54), and PQ 28.33 (σ = 1.55), while Intrinsic-T collapses to 10.82 (σ = 1.01). The paper text also cites a Vendi Score of 31.75 for PQ in one passage while Table 1 reports 28.33.
• Question coherence beats answer correctness. Annotating synthetic questions with Claude-4.5-Sonnet as an oracle judge, only 32.8% of PQ problems contain a fully correct solution while 63% are considered mathematically well-posed. Intrinsic-T questions have higher correctness (55%) but perform worse, attributed to lack of diversity. Meta-RL also decreases question ambiguity errors relative to the base teacher.
• Well-posed but wrong answers still help. On HARP-PQ, training performance improves when well-posed questions with incorrect answers are added to the subset with correct answers.
• The base model already has the signal. Successful runs from the base teacher (Base-T) show that useful stepping-stone questions are latent in the model; meta-RL sharpens the teacher so it produces them more reliably, tracking the upper envelope of Base-T performance for MATH/HARP with lower variance on MATH. On OlympiadBench, Grounded-T and Base-T learning curves overlap more, although Grounded-T-HARP achieves the best performance.

Methodology in Plain English

• Setup. Two copies of the same model are initialized from the base weights: a teacher and a student. The teacher writes question-answer pairs; the student trains on them with standard RLVR; the teacher's reward is how much the student's accuracy on real hard problems improved after training.
• No peeking at the hard problems. The teacher is never shown the hard questions directly. It only receives the student improvement score on the hard dataset, which acts as a black-box grounding signal that implicitly penalizes degenerate or hackable questions.
• Nested RL loops. The outer loop trains the teacher with RLOO to generate question-answer pairs, including both the question and the answer, since the paper does not assume automatic verification of well-posedness or answer correctness. The inner loop trains the student with RLVR (also RLOO) on teacher-generated problems.
• Reward definition. Each generated dataset is scored as the student's accuracy on subsampled reward questions from the hard train set after inner-loop training, minus the initial student's accuracy on the same questions. Rewards are averaged over parallel student trainings to reduce variance.
• Promotion mechanism. A rolling moving average of teacher rewards is tracked; when it exceeds a fixed threshold, the student baseline is updated to the student trained on the best generated dataset, so later rewards measure improvement relative to a stronger starting point. The accumulated datasets that triggered promotions become the Promotion Questions.
• Experimental scale. The main experiments use Llama-3.2-3B-Instruct, with ablations extending to Llama-3.1-8B-Instruct. Training runs for 200 outer-loop steps, sampling 64 generated problems and 64 reward questions per iteration, with an inner loop of 10 student steps at batch size 8 and a promotion threshold of 0.01. Evaluation uses pass@k for k ∈ {1,4,8,16,32} with 32 samples per problem, averaged over 6-12 seeds, and the overall study comprises over 600 runs.
• Evaluation protocol. The teacher is evaluated where training rewards stabilize (step 200 for MATH, step 170 for HARP). The promoted student is tested directly, and separately a fresh base student is trained on Promotion Questions mixed with the fail@128 train set to isolate the value of the synthetic questions.
• Baselines. Hard-Only (direct training on the real fail@128 train set, group size 32, with a group-size-128 variant), an Intrinsic Teacher using a learnability objective that rewards moderately difficult questions, SeRL, and a curated-data upper bound. The paper reports an observed maximum of four promotions, so the promoted student has trained on one of {128, 192, 256} synthetic questions.

Why This Matters

• Impact on research. The paper argues that meta-RL can expand the envelope of learnability beyond what direct RLVR achieves, and provides the first instance the authors are aware of a "double meta-RL loop" in LLM self-play. It reframes the debate over whether RL fine-tuning expands a model's frontier or merely sharpens latent abilities, showing that a grounded self-generated curriculum can make previously unlearnable problems learnable without human-curated intermediate data.
• It offers a concrete failure-regime grounding. Rather than optimizing intrinsic proxies such as learnability, majority vote, or reward-model preferences, the method tethers question generation to measurable progress on a fixed verifiable target set. The paper reports that this avoids the reward hacking, diversity collapse, and seed instability that intrinsic self-rewards show.
• Potential real-world applications (framed as plausible uses of the approach, not as results reported in the paper):
  • Post-training reasoning models on genuinely hard domains — advanced mathematics, formal proofs, scientific reasoning — where curated intermediate problem sets do not exist.
  • Bootstrapping training in low-resource or emerging domains where experts can supply hard target problems but not the ladder of easier ones.
  • Cost reduction in data pipelines, since the method aims to replace some human-curated stepping-stone data with model-generated questions.
  • Curriculum generation for agentic or tool-use settings with sparse binary success signals, if the grounding signal can be defined.
• Industry relevance. The work is a Meta FAIR collaboration with MIT and NYU, supported by the NSF GRFP, the Simons Foundation, and the Swiss AI Initiative with compute from the Swiss National Supercomputing Centre. The practical implication for labs is a route past the RLVR data wall: when a dataset is too hard to train on, generating a curriculum may be more effective than collecting more curated problems. The paper's own caveat is cost — the bilevel loop requires parallel student trainings for stability.

Future Directions

• Reduce the cost of bilevel RL loops. The authors name computational cost as their primary limitation and call for more efficient reward proxies, noting the inner loop involves 10-20 steps and parallel students for stability.
• Scale beyond small models. The study covers 3-8B parameter models; the authors describe scaling beyond 3-8B as a rich avenue for further work.
• Improve the grounding signal. Since the teacher only sees a black-box improvement score, questions remain about how much better the curriculum could become with a more informative or cheaper progress signal that still avoids reward hacking.
• Extend to harder domains. The paper's "North Star" thought experiment concerns a future model trained on the entire mathematical literature, where a proof of a Millennium Problem such as the Riemann Hypothesis might already be latent but require recovering the right sequence of intermediate lemmas and theorems to become learnable. Testing whether grounded meta-RL can elicit such capabilities is left open.
• Understand why intrinsic rewards fail. The paper documents that intrinsic teachers collapse into a narrow conceptual space and that one of three seeds collapses entirely, but the exact mechanism connecting intrinsic objectives to diversity collapse is not fully resolved.

Target Audience

This paper is most useful to machine learning researchers working on LLM post-training, RLVR, and reasoning models; to practitioners building self-play or synthetic-data pipelines who need to know when intrinsic rewards are unsafe; and to readers interested in curriculum learning and bilevel optimization. It also suits graduate students and technically fluent engineers who want a clear demonstration that a model can generate training data for problems it cannot yet solve. Readers without a background in reinforcement learning will find the results accessible, but the method section assumes familiarity with RLVR-style training and meta-learning concepts.

Authors’ abstract

RL methods for scaling large reasoning models stall on datasets with low initial success rates, and thus little training signal. We investigate a fundamental question: Can a pretrained LLM leverage latent knowledge to generate an automated curriculum for problems it cannot solve? We explore this with SOAR: An asymmetric self-play framework that uses meta-RL to surface these pedagogical signals. A teacher model proposes synthetic problems for a student model, and is rewarded with its improvement on a subset of hard problems, thus grounding the curriculum in real student progress rather than intrinsic proxy rewards. Our study on the hardest subsets of math benchmarks (0/128 success) reveals three core findings. First, it is possible to realize bilevel meta-RL that unlocks learning under sparse, binary rewards by sharpening a latent capacity of pretrained models to generate useful problems. Second, grounded rewards outperform intrinsic learnability rewards used in prior LLM self-play, reliably avoiding typical instability and diversity collapse modes. Third, the structure and well-posedness of questions are more critical for learning progress than solution correctness. Our results suggest that the ability to generate useful stepping stones does not require the preexisting ability to solve the hard problems, paving a principled path to escape reasoning plateaus without additional curated data