DiagEvo: Diagnosis-Guided Self-Evolution via Hierarchical Error Memory

AI summary

Overview

• Research area: Artificial intelligence, specifically self-play and self-evolution for large language models (reinforcement learning with verifiable rewards, curriculum generation, and agent memory).
• Technical level: Advanced. The paper assumes familiarity with self-play loops, GRPO-style policy optimization, pseudo-labeling by majority vote, and curriculum learning.
• Scope (one sentence): The paper proposes a self-play framework in which a lightweight diagnostician turns a solver's own failed reasoning trajectories into a hierarchical memory of recurring error causes that steers question generation in later rounds, without any external task resources.

What This Paper Is About

In self-play, one model writes questions and another model learns by answering them, but without guidance the questions drift away from the solver's useful learning zone and performance can plateau or decline. Existing fixes either use indirect signals such as difficulty, learnability, or diversity, or they import direction from outside the loop (human examples, document corpora, difficulty labels). DiagEvo instead mines the direction from something the loop already produces: the solver's own failure history. The goal is to keep generated questions near the solver's evolving competence boundary using only information generated during self-play.

Key Contributions

1. A curriculum signal derived from recurring error causes. The authors extract a dynamic curriculum signal from the solver's failed trajectories, tracking unresolved weaknesses without relying on external task resources such as human examples, document corpora, or difficulty labels.
2. DiagEvo, a diagnosis-guided self-play framework. Error causes are organized in a hierarchical memory (causes grouped under skill nodes) with per-cause Active and Mastered states. States and recurrence frequencies coordinate cause-targeted generation, free exploration, and cross-state stitching, while double-confidence filtering excludes high-conflict question–pseudo-label pairs from solver training.
3. Best overall performance among compared methods without external task resources. Experiments on Qwen3-4B, Qwen3-8B, and OctoThinker-8B show the default 4B diagnostician beats every baseline in mean accuracy across all nine benchmarks for each solver. On Qwen3-8B it reaches 72.3% on mathematical reasoning, 4.5 percentage points above R-Zero, and 57.4% overall, 1.1 percentage points above DARC, which does use external resources.
4. Ablations isolating each component. Mixed generation, memory-state updates with cross-state stitching, and double-confidence filtering each contribute to the reported gains.

Main Findings

• Default diagnostician wins across the board. With the default 4B diagnostician, DiagEvo exceeds every baseline in overall score on all three solvers: 53.5% on Qwen3-4B, 57.4% on Qwen3-8B, and 41.9% on OctoThinker-8B. On Qwen3-4B and OctoThinker-8B both overall scores exceed DARC by 1.3 points.
• Qwen3-8B headline numbers. DiagEvo reaches 72.3% on the five-benchmark mathematical average and 38.8% on general reasoning, exceeding R-Zero by 4.5 and 2.6 points respectively, and DARC by 1.2 and 1.0 points respectively, while using no external task resources.
• Bigger diagnosticians help only modestly. Going from 4B to 235B-A22B raises the mathematical average by 1.0 point on Qwen3-4B, 1.0 point on Qwen3-8B, and 1.2 points on OctoThinker-8B. The largest gains appear on harder math: OlympiadBench improves by 2.4, 2.4, and 2.8 points. General reasoning changes by only 0.2 to 0.5 points, and the default 4B configuration already exceeds every baseline by at least 1.1 points in overall score.
• Math training transfers to general reasoning. General averages rise from 33.2% to 38.8% on Qwen3-8B, from 27.8% to 35.0% on Qwen3-4B, and from 10.7% to 28.4% on OctoThinker-8B, despite training only on mathematical questions.
• Both generation modes matter. On Qwen3-8B-Base, freezing the challenger drops the math average by 3.8 points, the largest drop in that ablation group. The full mixed policy (72.3%) beats pure free exploration (69.5%) and pure cause-targeted generation (70.1%).
• Stitching must stay within a skill node. Cross-state stitching inside the same skill node exceeds random-pair stitching by 1.0 point in mathematical average.
• Relative confidence adds real value. Absolute confidence filtering alone gives 70.9% math average versus 69.4% with no filtering; adding the relative confidence constraint raises it a further 1.4 points.
• Improvement lasts five rounds. DiagEvo improves in each of the first five rounds, reaching 72.3%, then falls to 72.0% at round 6 and 71.4% at round 7 (general averages 38.4% and 37.9%). The round-5 checkpoint is used in all experiments. Pure free exploration peaks after round 3 with a peak of 69.5% and ends at 69.0%.
• Memory lifecycle. Error causes grow from 151 to 244 across rounds, skill nodes stabilize near 36 after round 3, Active causes peak at 188 in round 2 and fall to 117 by round 5, and Mastered causes reach 127, more than half the memory.
• Pseudo-label quality. With pure free exploration and only the absolute confidence constraint, oracle agreement falls from 83% to 65%. Adding the memory under the same constraint raises the round-5 value to 72%, and adding the relative confidence constraint raises it further to 75% — gains of 7 and 3 points at round 5.

Methodology in Plain English

DiagEvo runs a loop with two models that update in turn; only one is trained at a time, and only the challenger sees the memory. Each round works in four stages:

1. Mixed question generation. The challenger draws questions either by free exploration or by targeting a specific Active error cause from the memory. The balance is frequency-driven: the more failures currently attributed to active causes, the more the challenger targets causes rather than exploring. When targeting, the challenger may also "stitch" in a Mastered sibling cause from the same skill node to keep old weaknesses in play.
2. Challenger update. Following R-Zero, a frozen solver answers a batch of challenger questions, and GRPO updates the challenger. The reward combines an uncertainty term (maximized when the solver's majority-vote share is near 50%) with a repetition penalty based on clustering of similar questions, with λ = 1.
3. Double-confidence filtering for solver training. For each candidate question, the solver samples N responses. The majority answer becomes the pseudo-label. A question is kept only if its majority vote share falls in an intermediate range (absolute confidence constraint) and the top answer's share is at least τ times the runner-up's share (relative confidence constraint, τ = 1.6 in the full configuration). Separate "optimization responses" then supply the GRPO learning signal.
4. Memory maintenance. For each question, the diagnostician compares a failed response with one that agreed with the pseudo-label and locates their earliest reasoning difference. That difference becomes a candidate error cause, stripped of question-specific values. The system then deduplicates against existing causes, assigns new causes to skill nodes, and consolidates redundant nodes. Each cause carries a state (Active or Mastered) and an active-episode frequency. A cause is promoted to Mastered when the mean self-consistency on questions targeting it reaches a threshold, resetting its frequency; later failures can reactivate it.

Diagnosticians tested: Qwen3-4B-Instruct-2507 (default), Qwen3-30B-A3B-Instruct-2507, and Qwen3-235B-A22B-Instruct-2507. Evaluation uses five math benchmarks (MATH-500, GSM8K, OlympiadBench, Minerva Math, AMC, with 40 AMC questions and mean@32) and four general benchmarks (MMLU-Pro, SuperGPQA, GPQA-Diamond, BBEH), following the simple-evals protocol with GPT-4o as judge, matching DARC. DiagEvo results are reported as mean and standard deviation over three independent runs.

Why This Matters

DiagEvo shows that a self-improvement loop can generate its own curriculum from failure data it already produces, which reduces dependence on curated human examples, document corpora, or difficulty labels — resources that are expensive and that cap how far a loop can continue on its own. It also connects memory research on LLM agents with curriculum learning inside self-play, treating past failures as a schedulable teaching signal rather than as a retrieval aid for a single task-solving agent.

Real-world applications:

• Math and science tutoring systems that adapt practice problems to a specific learner's recurring misconceptions instead of climbing a fixed difficulty ladder.
• Domain-specific model training in settings where no labeled corpus exists (internal codebases, proprietary manuals, regulated workflows), so the curriculum has to come from the model's own errors.
• Continual learning pipelines that need to keep improving a deployed model over many update rounds while controlling question quality and pseudo-label noise.
• Agent training for long-horizon tasks where recurring failure modes across episodes are the natural training signal, as the authors gesture toward with multi-turn tool use.

Industry relevance: the framework targets the practical middle of the market — Qwen3-4B, Qwen3-8B, and OctoThinker-8B solvers with an inexpensive 4B diagnostician — and the results show that a large diagnostician is not the main source of gains. That makes the approach attractive for teams that want self-improving post-training without paying for external annotation or very large inference-time models.

Future Directions

• Better verification than vote agreement. Double-confidence filtering measures agreement among solver responses, not ground-truth correctness, so a shared error can receive strong agreement and pass the filter. The authors suggest studying other self-play verification signals to catch these cases.
• Adaptive stopping for the training horizon. The number of rounds is currently fixed in advance, while peak performance was observed at round 5 of 7. The authors propose using changes in the set of Active causes and solver self-consistency to decide whether another round is worth running.
• Broader domains. The curriculum is built from mathematical questions only; the general-reasoning gains demonstrate transfer but do not test direct curriculum construction elsewhere. Extending diagnosis and memory to open-ended tasks with long interaction sequences, including multi-turn tool use, is left open.
• Improving the memory lifecycle itself. With error causes growing from 151 to 244 and Mastered causes reaching 127, questions remain about how the hierarchy should grow, consolidate, or prune over longer runs.

Target Audience

Researchers and engineers working on self-play, reinforcement learning from verifiable rewards, and post-training for language models, along with practitioners building self-improving or curriculum-driven training pipelines under tight annotation budgets. Readers interested in agent memory systems and curriculum learning will also find the memory-state design relevant, since the paper's distinctive move is to make the memory schedule the curriculum rather than merely store experience.

Authors’ abstract

Self-play is an effective paradigm for language-model self-evolution, but without guidance, solver performance can plateau or decline across rounds. Unguided methods steer question generation with signals such as difficulty, learnability, or diversity. These signals keep questions challenging and varied but do not specify which unresolved reasoning weaknesses later rounds should target. Guided methods obtain direction from external task resources, including human examples, document corpora, or specified difficulty targets, and therefore rely on task information supplied outside the self-play loop. We show that the needed direction can instead be derived from the solver's own failure history. We introduce DiagEvo, whose diagnostician extracts recurring error causes from this history and stores them in a hierarchical error-cause memory. The memory groups related causes under skill nodes and tracks each as Active or Mastered according to self-consistency on targeted questions. The challenger uses these states and recurrence counts to balance cause-targeted generation with free exploration. Double-confidence filtering retains intermediate-difficulty questions only when the most common solver answer has a clear vote lead. DiagEvo derives its curriculum from information produced during self-play, without external task resources. With the default 4B diagnostician, DiagEvo outperforms every baseline in mean accuracy across all nine benchmarks for each of the three solvers: Qwen3-4B, Qwen3-8B, and OctoThinker-8B. On Qwen3-8B, it reaches 72.3% mean accuracy across five mathematical reasoning benchmarks, 4.5 percentage points above R-Zero. Its mean accuracy across all nine benchmarks is 57.4%, 1.1 percentage points above DARC. Ablations show that the hierarchical error-cause memory and double-confidence filtering both contribute to these gains.