Questioning the Questions: Sustaining Self-Evolution in Reasoning Models

AI summary

Overview

Research area: Self-evolving (self-improving) reasoning models, specifically the questioner–solver reinforcement learning loop in which a model generates its own training questions and answers them, using methods such as Group Relative Policy Optimization (GRPO).

Technical level: Advanced. The paper assumes familiarity with reinforcement learning for language models, reward shaping, GRPO, and self-play/self-training pipelines.

Scope: The paper diagnoses why questioner–solver self-evolution degrades over successive rounds and proposes R-Quest, a method that adds question-validity and task-novelty feedback to sustain gains across mathematics, code generation, and general-domain reasoning benchmarks.

What This Paper Is About

Self-evolving reasoning models reduce dependence on human-curated data by generating their own questions and learning to solve them, but repeated rounds of this process can reverse early gains and lead to collapse. The authors analyze questions generated by the R-Zero framework and identify two recurring failure modes: invalid questions and repeated variants of the same underlying mathematical task. The goal is to keep self-evolution producing useful training signal over many rounds rather than degrading after a few.

Key Contributions

1. A diagnostic analysis of self-evolution failure. The paper documents that the valid-question rate in R-Zero's question pool falls from 76.5% in round one to 39.5% in round five, that answer-consistency filtering lowers the valid rate among retained questions further to 27.64% in round five, and that lexical (BLEU-based) diversity controls fragment repeated mathematical task types across clusters — splitting the largest question type into 22 clusters in round five.

2. R-Quest, a self-evolution method with two feedback signals. Validity feedback comes from a solver initialized to recognize and reject invalid questions; novelty feedback comes from a frozen base model performing pairwise judgments of whether two questions share the same mathematical task type. Both signals shape the questioner's reward during alternating questioner–solver updates.

3. A sampled pairwise novelty gate that avoids quadratic cost. Instead of evaluating all pairs in a batch (O(B²) judgments), the method samples K other questions per candidate without replacement (O(BK)), with a pass probability of approximately (1−p)^K.

4. Sustained improvements across domains and model families. R-Quest achieves the highest domain-average scores among evaluated methods on 12 benchmarks across mathematics, code generation, and general reasoning on Qwen3-4B-Base and OctoThinker-3B-Hybrid-Base, and maintains gains over ten rounds of self-evolution.

Main Findings

• Invalid questions accumulate and filtering makes it worse. Sampling 200 questions from each of five R-Zero rounds and judging them with GPT-5.6-Sol, the valid-question rate drops from 76.5% (round one) to 39.5% (round five). Among questions retained by answer-consistency filtering (those with majority-vote agreement between 0.3 and 0.8), the valid rate falls to 27.64% in round five.

• Repairing invalid questions improves solver performance. With Qwen3-4B-Base, GRPO training on a repaired dataset (1,273 questions, including the same 766 valid questions, with 507 invalid questions replaced by GPT-5.6-Sol repairs) reaches a final average of 49.91 on seven mathematical benchmarks versus 46.54 on the original dataset.

• Repetition is severe and lexical metrics miss it. In 200 questions sampled from R-Zero's filtered fifth-round pool, the three largest question types account for 59.5% of the sample, and the largest type is split into 22 BLEU clusters, weakening the cluster-size-based repetition penalty.

• R-Quest leads on mathematics benchmarks. Averaged over seven mathematical reasoning benchmarks, R-Quest scores 51.64 on Qwen3-4B-Base (base model 46.17; R-Zero 49.10; OCNR 49.57; R-Diverse 49.82) and 30.69 on OctoThinker-3B (base model 24.90; R-Zero 28.25; OCNR 28.12; R-Diverse 28.61). This is an improvement of 5.47 points over Qwen3-4B-Base and 5.79 points over OctoThinker-3B, with additional gains of 2.54 and 2.44 points relative to R-Zero.

• Gains extend to code and general reasoning. Average code generation (HumanEval+ and MBPP+, pass@1) rises by 3.90 points over the base model on Qwen3-4B-Base (62.69 vs. 58.79) and by 5.23 points on OctoThinker-3B (18.90 vs. 13.67). General-domain reasoning averages (SuperGPQA, MMLU-Pro, BBEH) are also highest for R-Quest: 32.75 on Qwen3-4B-Base and 15.50 on OctoThinker-3B.

• Self-evolution is sustained over ten rounds. On Qwen3-4B-Base, R-Zero peaks at 49.10 in round three and declines to 34.60 by round ten, below the base model's 46.17. R-Quest exceeds R-Zero's best score at every checkpoint from rounds one through ten, reaches its highest score of 51.92 in round ten, and beats R-Zero by 17.32 points at the same iteration. After ten rounds, R-Quest is 5.75 points above the base model while R-Zero is 11.57 points below it.

• Both feedback signals matter, but validity is more critical. Ablations on Qwen3-4B-Base show R-Quest at 51.64 (math), 62.69 (code), 32.75 (general); removing novelty gives 51.02 / 61.98 / 31.80; removing validity gives 50.27 / 61.28 / 31.53; removing both (equivalent to R-Zero) gives 49.10 / 59.27 / 31.21. A frozen validity judge instead of continuous validity training scores 49.93 / 61.85 / 32.39. Removing validity feedback leads to late-stage collapse, while removing novelty feedback produces a milder decline that stays above the base model.

• Validity assessment improves during self-evolution. On 297 held-out questions, R-Quest's balanced accuracy — the average of valid-question recall and invalid-question recall — rises from 64.80% after initialization to 72.93% in round ten. Invalid recall reaches 86.36% in round ten, while R-Zero's stays between 10.23% and 26.14% across all ten rounds. A solver that accepts every question would score 50% balanced accuracy.

• Question quality and answer supervision stay healthier. Sampling 200 unfiltered questions per method and round, R-Quest's question validity rate rises from 88.0% in round one to 94.5–97.5% across rounds four to ten, while R-Zero's falls from 76.5% to 49.0%. Majority-answer accuracy on valid questions, checked against independently derived GPT-5.6-Sol references, is 54.69% for R-Quest by round ten versus 2.06% for R-Zero.

• Task concentration is controlled, but over-control hurts. Over the first five rounds, the top-five share of question types rises from 17.0% to 69.0% for R-Zero, while R-Quest with K=8 reaches 24.0% in round five. Increasing K to 16 lowers the top-five share further — notably from 30.0% to 16.8% in round four — but performance stays below the K=8 setting throughout.

Methodology in Plain English

The authors start by running the existing R-Zero questioner–solver loop and inspecting what it produces. Using GPT-5.6-Sol as an external judge, they check how many generated questions are actually well posed and how many distinct mathematical task types appear, then test whether replacing invalid questions with repaired versions changes solver outcomes under matched training conditions.

From these observations they build R-Quest in three parts. First, they create a validity-aware solver: they deduplicate questions from five R-Zero iterations, sample an equal number per iteration, annotate each with GPT-5.6-Sol for validity and reference answers, review the answers, and train the base model with GRPO to either give a mathematical answer or output INVALID. The reward gives credit for correct answers and applies a penalty (controlled by λ) when the model wrongly rejects a valid question. The final initialization dataset contains 2,240 questions (1,943 training, 297 validation), of which 1,317 are invalid; the most common error types are contradiction (380, 28.9%), no solution (362, 27.5%), ambiguity (315, 23.9%), missing information (181, 13.7%), poorly formed questions (72, 5.5%), and other (7, 0.5%).

Second, they add novelty checking. A frozen copy of the base model judges whether two questions share essentially the same mathematical setup and request the same kind of task; changing numbers or wording does not make a new type. Rather than comparing every pair in a batch, each candidate is compared against K randomly sampled others, and it is rejected if any comparison indicates the same type. With K=8, a same-type fraction of 10% yields a pass probability of roughly 43%, compared with about 6% at a fraction of 30%.

Third, they fold both signals into the questioner's reward. The validity-aware solver produces multiple responses per generated question; the fraction returning INVALID defines u(q) and the gate g_valid(q) = 1/2 − u(q), with questions classified invalid when more than half the responses say INVALID. Questions that pass are re-solved under a normal problem-solving instruction, producing a consistency statistic s(q) and an uncertainty reward of min{s(q), 1−s(q)}. The questioner receives the validity gate when the question is judged invalid, and otherwise receives the uncertainty reward multiplied by a binary novelty gate. The solver is then updated with GRPO on answer-consistency-filtered data with majority-vote pseudo-labels, mixed with a small fixed proportion of replay data from the initialization set so it retains its rejection ability. Training uses a per-round budget of 10,000 candidate questions, a validity gate based on majority voting over nine responses, K=8 novelty comparisons, and 10% replay each round.

Why This Matters

Impact on research. The paper reframes self-evolution collapse as a question-quality problem rather than only a reward-design or optimization problem, and shows that validity and task-level novelty can be judged by the model itself rather than by a stronger external model. It argues for self-contained feedback, contrasting with approaches that call external verifiers or auxiliary code generators and encoders (for instance, VHG's use of GPT-5.4, or R-Diverse's reliance on Qwen2.5-Coder-7B and Jina-Code-Embeddings-1.5B).

Potential real-world applications:

• Generating large volumes of self-checked mathematics training material without proportional human curation effort.
• Improving reasoning-heavy assistant behavior in domains where unstated assumptions or ambiguous problem statements lead to wrong answers.
• Training code-generation models, where the paper observes a 3.90-point gain on Qwen3-4B-Base and a 5.23-point gain on OctoThinker-3B and links this to reasoning that respects stated constraints.
• Reducing label-acquisition cost by keeping model-generated pseudo-labels reliable over long training runs (54.69% versus 2.06% majority-answer accuracy at round ten).

Industry relevance. Any pipeline that trains reasoning models on self-generated synthetic data faces the same failure mode: early gains followed by drift. The results indicate that a modest amount of data annotation plus two feedback signals can keep a training loop productive for ten rounds instead of three, which affects compute budgeting and the viability of long-running self-improvement cycles.

Future Directions

• Choosing the novelty budget K. The paper shows K=16 suppresses repeated task types more than K=8 (top-five share dropping from 30.0% to 16.8% in round four) but performs worse overall, leaving open how to balance exploration of new exercise patterns against progress on harder variants of existing ones.
• Extending validity-aware initialization to other domains. The initialization dataset is built entirely from mathematical questions; whether the same rejection ability transfers to code or general reasoning generation is not established.
• Scaling to larger models and more rounds. Results are reported for two base models (Qwen3-4B-Base and OctoThinker-3B-Hybrid-Base) over ten rounds; behavior beyond ten rounds and at larger parameter counts is not reported.
• Reducing dependence on strong external judges. GPT-5.6-Sol is used for annotation, repair, and every analysis measurement, though not during the self-evolution loop itself; understanding how much of the analysis is reproducible with weaker or no external judges remains open.

Target Audience

Researchers and engineers working on self-improving or self-play language models, reinforcement learning post-training, and synthetic data generation for reasoning tasks. It is also relevant to practitioners building long-running training loops who need to know why model-generated supervision degrades and what feedback signals keep it usable. Readers without a background in reinforcement learning for language models will find the method sections demanding, though the diagnostic findings are accessible on their own.

Authors’ abstract

Self-evolving reasoning models learn from their own generated questions, yet repeated self-training can lead to performance collapse. In this paper, we investigate why performance deteriorates over successive rounds and how to sustain self-evolution. Our analysis identifies two recurring quality problems in self-generated questions: invalid questions and repeated variants of the same mathematical questions. First, invalid questions become more prevalent across rounds, and answer-consistency filtering further increases their proportion in training data. Second, existing question diversity controls based on lexical similarity can miss mathematically equivalent questions expressed in different ways, which leads to question diversity collapse in later training rounds. Building on these findings, we introduce R-Quest, which uses question validity and novelty feedback to guide self-evolution. We first train the solver to recognize and reject invalid questions, then use its judgments to guide questioner rewards and filter solver training data. To avoid question repetition, we use a frozen base model to compare sampled question pairs and provide novelty feedback. Empirically, our method consistently achieves the highest average performance on 12 benchmarks in mathematical reasoning, general-domain reasoning, and code generation across two model families. Additionally, R-Quest maintains stable performance gains over ten rounds of self-evolution, peaking in the final round and outperforming R-Zero by 17.32 points.