Selection Is Not Learning: Choosing Model Futures

AI summary

Overview

Research area: Machine learning — specifically self-evolving / self-training large language models, and the problem of deciding which model updates to keep across training rounds.

Technical level: Advanced. The paper combines a formal regret analysis (sub-Gaussian concentration bounds, union bounds) with a full empirical training pipeline; the core intuition is accessible, but the guarantees and training details assume familiarity with reinforcement learning from self-generated data and LoRA-based distillation.

Scope (one sentence): The paper shows that screening which tasks a self-evolving model should learn from does not predict whether the resulting update helps, and proposes two mechanisms — HeirCommit (accept an update only if it wins on fresh questions) and EvoHeir (separate which parameters are inherited from which experience is retained) — that reduce harmful commits by 68–83% and lead on math and general-reasoning averages across four backbones.

What This Paper Is About

Self-evolving language models write their own practice problems, train on them, and then pass the updated model to the next round. Existing loops such as R-Zero, Agent0, and Skill Self-Play make that inheritance decision on the task side: they admit problems that look learnable and then keep every update those problems produce. The authors show this leaves a "selection-to-update gap" — a problem that looks learnable often yields an update that makes correct answers less likely, yet the update is inherited anyway. Their goal is to move the decision from the task to the model future: judge the actual updated model on fresh questions, and keep useful experience even when the model that produced it is discarded.

Key Contributions

1. Identification and formalization of the selection-to-update gap. The authors prove (Proposition 1) that any rule using only pre-update behavioral observations — pass rate, disagreement, frontier proximity — can face two learning worlds with identical observations, equally valid targets, and the same native step size, but values of +Δ and −Δ, giving worst-case expected regret of at least Δ/2 against the better of that update and the current model.

2. HeirCommit, a source-independent commit rule. The updated model is frozen and compared with the current model (the "root") on a fresh probe pool; the successor is inherited only if its paired root-relative gain is strictly positive. Applied to the native updates of R-Zero, Agent0, and Skill Self-Play, it turns negative pool gain positive on two of the three hosts and cuts the harmful share of accepted updates by 68–83% relative to random inheritance at the same acceptance rate.

3. EvoHeir, which separates parameter inheritance from experience inheritance. Each round explores native branches, screens for a parameter recipient, repairs that recipient's failures using current and archived reference solutions while preserving its successes, then confirms the completed post-absorption state with HeirCommit. Over ten rounds on four backbones it raises the base model's math average by 7.9–30.1% and has the highest math and general-reasoning averages of any compared method.

4. Evidence that experience outlives the model that produced it. Donors weaker than the current model (mean donor gain −1.09 pp) still yield positive absorption gain (mean +0.71 pp), and admitting history from rejected rounds adds 0.66 pp beyond committed-round history; a matched-record swap shows the gain comes from the sources, not from more references.

5. A separation theorem for search versus inheritance (Theorem 1): reusing screening evidence to accept a branch pays a union bound over the K branches, while a fresh confirmation pool does not, even when absorption follows recipient selection.

Main Findings

• Most verified updates do not help. In an audit of real Skill Self-Play updates on exact-valid tasks, 60 of 96 updates (62.5%) do not reduce correct-target negative log-likelihood.
• Admitted tasks frequently leave the frontier. Under fresh responses, 52.2% of Agent0's and 42.0% of R-Zero's admitted tasks leave the native frontier (reported as 42–52%).
• Selection inflates a target's apparent quality. Selected-target optimism — the drop from a candidate's score on its selection responses to its score on fresh responses — is 8.60 pp for Agent0 and 4.10 pp for R-Zero, while fixed-target controls sit at 0.20 and −0.10 pp. Proposition 2 states this winner's-curse inflation holds even when each candidate's score is unbiased.
• HeirCommit reduces harmful commits on every host. Over 64 native updates per host, harmful share falls from 77% to 13% (R-Zero), 58% to 16% (Skill Self-Play), and 47% to 15% (Agent0), while committing 23–52% of updates and retaining 78–87% of the useful ones.
• EvoHeir leads on all four backbones. Math averages at round 10: Qwen3-4B-Base 25.81 → 31.82 (Agent0 30.01); Qwen3-8B-Base 28.06 → 36.51 (Agent0 34.11); OLMo-3-7B 19.92 → 24.94 (Agent0 23.35); Gemma-4-E4B-it 43.77 → 47.21 (Agent0 46.66). The abstract reports a 5.3% math-average gain over Agent0; Table 1 reports +6.0%, +7.0%, +6.8%, and +1.2% per backbone.
• Wins are broad but not universal. EvoHeir beats Agent0 on 45 of 48 benchmark–backbone pairs across the twelve benchmarks; the three exceptions are 30-problem AIME sets.
• It keeps climbing where baselines stall. Over ten rounds EvoHeir reaches +5.7 math points, versus +4.1 for Agent0 and +2.7 for R-Zero, which peaks at round 8 and then slips (means over four backbones).
• Weak donors still teach. Eight of ten donors weaker than the current model improve the recipient, with mean donor gain −1.09 pp but mean absorption gain +0.71 pp.
• Recovery beats direct fine-tuning. From the same 64 references, Cache SFT gains 2.08 pp of new successes but loses 2.61 pp (net −0.54 pp), while Recovery gains 2.54 pp and loses 1.15 pp (net +1.39 pp) — a 1.93 pp average advantage across four checkpoints, though Recovery's own net gain is negative at round 7 (−0.20 pp).
• Including rejected-round history matters. At a fixed 64-reference budget, current-only material gives +0.39 pp, adding committed-round history gives +0.73 pp, and admitting all qualified history including rejected rounds gives +1.39 pp. A matched swap of 32 records shows non-inherited records add 0.34–0.73 pp at ages one to four with 95% bootstrap intervals above zero.
• Both mechanisms are load-bearing. Ablations on Qwen3-4B-Base drop math/general averages from 31.82/30.52 (Full) to 30.77/29.99 (Same-Lineage), 30.54/29.87 (Random Experience), 30.85/30.04 (Current-Round Only), and 29.80/29.33 (NoConfirm); removing confirmation causes the largest drop, 2.02 and 1.19 pp.
• The advantage depends on budget. Under a normalized accounting budget where a single-stream native baseline costs 100.0 units and Full's default costs 123.7 units (a 23.7% overhead), Full has the lowest mean at 0.8 of its default budget but the highest from 1.0 onward.
• A concrete failure-and-repair example. On Qwen3-4B-Base at round 5, the recipient drops the cross term in √(x+9) = √x + 1 and answers x = 0; an archived round-4 reference from a proposal that was not inherited recovers x = 16, and Recovery reproduces it without the reference. Cache SFT under the same 64 references changes the correct remainder of 2^100 mod 7 from 2 to 4, while Recovery keeps 2. The fresh-probe gain is −0.31 pp, so HeirCommit retains the root.

Methodology in Plain English

The setup is a round-based loop. A Curriculum Scout proposes problems; an Evolution Controller admits those sitting at the learner's frontier, using pre-update signals such as pass rate over ten learner responses; native training (RL or SFT) then produces updated branches. Value is defined per state as expected gain over the round's starting model, which for binary correctness splits into successes gained minus successes lost.

The audit asks whether the admission signal predicts the update's realized value. The authors compare a target's score on the responses that selected it with its score on fresh responses, measure how often admitted tasks leave the frontier, and measure the correct-target NLL gain of real updates.

HeirCommit changes the decision point. Instead of trusting the screening evidence, it freezes the candidate model and scores it against the current model on a held-out probe pool, accepting only if the paired gain is positive. The theory explains why the pools must be disjoint: reusing the screening set to accept a branch pays a union bound over all branches, whereas a fresh pool pays only a single-tail bound.

EvoHeir adds structure before that decision. Two native branches are trained from the same root, then screened on 16 questions with ties going to the root — this picks only the starting parameters for the next step. Experience Recovery then takes that recipient and, on reference-free attempts, splits its failures and successes into disjoint sets: failures paired with qualified references become repair material, successes become preservation material. A frozen copy of the recipient serves as teacher, conditioned on a donor's reference for repair and on the problem alone for preservation; a LoRA student initialized from the recipient's own weights matches these teachers via forward KL, with λ_keep = 1, rank 8, and at most 16 updates. Crucially, the donor need not be the inherited model — it only supplies a reference. The completed proposal is then confirmed by HeirCommit against the root on 32 fresh questions, and qualified experience enters a bounded archive regardless of whether the round commits.

Evaluation covers four backbones (Qwen3-4B-Base, Qwen3-8B-Base, OLMo-3-7B, Gemma-4-E4B-it), ten rounds, three training seeds, seven mathematics and five general-reasoning benchmarks, with disjoint question pools for training, Recovery, screening, confirmation, and final evaluation, and frozen consensus labels requiring at least 0.7 support.

Why This Matters

Impact on research. The paper reframes a design choice that self-evolution pipelines have largely taken for granted — that inheriting every update from a well-chosen task is safe. Its non-identifiability result says no amount of clever task-side scoring can substitute for evaluating the realized model, and its separation theorem gives a concrete reason (a union bound over branches) why confirmation must use fresh data. The distinction between parameter inheritance and experience inheritance opens a design axis that whole-state acceptance collapses.

Real-world applications:

• Continual fine-tuning pipelines that retrain on synthetic or self-generated data can gate each update on held-out performance rather than on data-quality heuristics, avoiding silent regressions that accumulate across rounds.
• Agent systems that revise their own scaffolds or adapters can apply the same accept-or-retain rule before deploying a changed harness to users.
• Cost-constrained training fleets can reuse reference solutions from discarded checkpoints, recovering value from compute that would otherwise be written off.
• Safety and reliability monitoring benefits from a rule that defaults to retaining the known-good model when fresh evidence is not clearly positive.

Industry relevance. The overhead is quantified: 23.7% over a single-stream native baseline in normalized accounting units, and a Full round takes 8–9 wall-clock hours on two 96-GB GPUs. The finding that the method wins only from roughly 1.0 budget onward is directly actionable — teams must reserve enough compute for native training after paying for screening and confirmation, or the gate becomes a net cost.

Future Directions

• Replacing the zero-threshold gate with a positive confidence margin to obtain the certified per-round and adaptive-round guarantees that the reported runs trade away for acceptance sensitivity.
• Reducing the 23.7% construction overhead, since the equal-budget study shows the method loses at 0.8 of its default budget; cheaper screening, shorter recipient attempts, or smaller confirmation pools are natural targets (four candidates and 32–64 confirmation questions were already close to the defaults).
• Extending the donor/experience decoupling beyond the tested setting — the archive currently admits qualified responses under completeness, answer-validity, and de-duplication checks with a predeclared retrieval budget.
• Testing the framework on hosts outside the three studied loops and on instruction-tuned starts with more headroom, since Gemma-4-E4B-it showed the smallest gain (+0.55 pp over Agent0, +1.2% relative) and the only negative benchmark cells were 30-problem AIME sets.

Target Audience

Researchers and engineers building self-improving or continually trained language models — particularly those working on self-play curricula, synthetic-data self-training, and agent self-evolution. It is also relevant to practitioners who need a principled accept/reject rule for model updates in production, and to theoretically inclined readers interested in why selection and evaluation must use disjoint evidence. Readers without background in self-training loops will find the empirical sections readable but the regret analysis demanding.

Authors’ abstract

Self-evolving language models train on tasks they generate, so every round must decide which updated model the next round inherits. Current loops such as R-Zero, Agent0, and Skill Self-Play make this decision through the task, admitting tasks that look learnable and inheriting every resulting update. We identify a selection-to-update gap in these loops, where a task that looks learnable need not produce a useful update. Most updates on verified tasks fail to make correct answers more likely, yet these loops inherit them all. HeirCommit freezes the updated model and uses fresh evaluation questions to decide whether to inherit it or retain the current model. EvoHeir separates parameter inheritance from experience inheritance, using current and archived experience to repair failures and preserve successes before final confirmation. Our analysis characterizes selection-induced optimism and decomposes regret into selection, absorption, and confirmation errors. HeirCommit cuts the harmful share of accepted updates by 68-83% relative to random inheritance at the same acceptance rate. Across four backbones, EvoHeir achieves the best mathematics and general-reasoning averages, raising the math average by 5.3% relative to the strongest baseline, Agent0. The framework lets useful experience persist even when the model that produced it is not inherited.