Self-Improving Large Language Models via Progressive Experience Evolution

AI summary

Overview

Research area: Natural language processing — specifically self-improving large language models, post-training, reinforcement learning for reasoning, and knowledge distillation.

Technical level: Advanced. The paper assumes familiarity with on-policy distillation, KL-divergence objectives, group-relative policy optimization, mutual-information framing of representation learning, and LLM post-training pipelines.

Scope: The paper proposes SPEE, a two-stage post-training framework that extracts transferable "experience" from interaction trajectories, internalizes it into model parameters via on-policy self-distillation, and then applies reward-driven reinforcement learning — evaluated on five mathematical reasoning benchmarks across three Qwen3 model scales.

What This Paper Is About

Existing self-improvement methods for LLMs are split into two camps: test-time methods can pull explicit experience out of interactions but cannot write it into the model's weights, while training-time reinforcement learning can update weights but only absorbs experience indirectly through sparse scalar rewards. SPEE targets the missing middle step — "experience distillation" — by first building and refining a pool of transferable experience from both successful and failed trajectories, then distilling that experience into the policy before running reward-driven optimization. The goal is a self-reinforcing loop in which better experience produces a better policy, which in turn produces richer experience.

Key Contributions

1. A framing of self-evolution as experience distillation. The authors formalize the problem with mutual information: the extractor should minimize I(T;E) − λI(E;C), meaning experience should discard trajectory-specific detail while preserving information predictive of future competence, and they identify this intermediate stage as underexplored.
2. SPEE, a unified two-stage post-training framework. Stage I performs explicit experience evolution plus privilege-guided On-Policy Self-Distillation (OPSD); Stage II performs implicit policy optimization with Group Relative Policy Optimization (GRPO) initialized from the distilled policy.
3. A continuously evolving global experience pool. The pool consolidates knowledge from successful and failed trajectories, merges overlapping items, resolves redundancy and conflicts, and filters low-utility experience using marginal utility measured on a held-out probe set, mitigating post-hoc rationalization caused by instance-specific details.
4. An empirical demonstration across three model scales and five benchmarks, with the source code released at https://github.com/rrrsj/SPEE.

Main Findings

• SPEE achieves the highest average accuracy at all three scales. On qwen3-1.7b/4b/8b-base, SPEE improves over the base checkpoints by +4.87, +6.96, and +6.53 percentage points respectively, reaching average scores of 30.56, 36.67, and 38.71.

• SPEE beats GRPO at every scale, with an average improvement of 1.16%. For example, on qwen3-4b SPEE scores 36.67 average versus 34.75 for GRPO, and on qwen3-8b it scores 38.71 versus 37.92.

• SPEE consistently outperforms SDPO, the self-distillation baseline. SDPO actually falls below the base model on qwen3-1.7b (24.51 average, Δ −1.18).

• Domain Prompt (test-time baseline) gains are small and uneven. It improves qwen3-4b by +3.11 and qwen3-8b by +2.09, but only +0.46 on qwen3-1.7b, and it drops the AIME24 score for the 8B model from 7.50 to 4.17 — supporting the paper's claim that context-level experience is not preserved as intrinsic capability.

• Both stages are necessary (ablation, Table 2). Removing Stage II drops qwen3-4b average from 36.67 to 35.16 and qwen3-8b from 38.71 to 36.14. Removing the shared experience pool drops them further to 34.13 and 35.48 respectively, indicating the pool plays a particularly important role.

• Stage I increases response diversity rather than collapsing it. Using 8 rollouts per problem, Stage I substantially reduces the proportion of problems where all sampled responses are incorrect, while the post-Stage-I policy retains an entropy level comparable to the base model.

• SPEE is more data-efficient than GRPO. At the same performance level, SPEE requires approximately 28% less training data.

• More evolution iterations steadily raise sampling accuracy. Defined as the proportion of sampled responses that are correct and receive positive reward, it rises from 18.83% at iteration 0 to 20.55%, 21.33%, 21.80%, and 22.66% after one through four iterations — an increase of 3.83 percentage points, or roughly a 20.34% relative improvement. The main experiments, however, use only a single evolution round for computational efficiency.

• Qualitative case study. On a problem requiring the last two digits to satisfy both a digit-sum constraint and divisibility by 4, the baseline produced an incorrect answer from an incomplete candidate set, while the experience-guided model enumerated all ordered digit pairs and verified divisibility, arriving at 7 × 6 = 42. The injected experience contained neither the answer 42 nor problem-specific candidates.

Methodology in Plain English

SPEE runs in two sequential stages.

Stage I — Explicit experience evolution. For each problem, the current policy samples G candidate responses, which get binary rewards and are split into successful and failed sets. To save compute, only a random subset of S trajectories is kept, preserving the original proportion of successes and failures. The same model, acting as an experience extractor under a special prompt, converts each selected trajectory into an experience item: positive items capture effective reasoning patterns, negative items capture invalid patterns and recurring failure modes. These items are treated as provisional hypotheses rather than final knowledge. An evolution operator — again the same model — merges the new items into a global pool by consolidating overlaps and abstracting trajectory-specific observations into more transferable strategies. Each experience's marginal utility is then measured on a held-out probe set by checking how much it improves expected reward; a two-stage filter screens candidates on a small subset first and only evaluates items with positive utility on the full probe set, accepting those above a threshold ε. The pool therefore follows a path from E(0) to E(M).

Stage I — Privileged experience distillation (OPSD). The refined pool is then used as privileged information available only at training time. Two branches share the same parameters: a teacher branch that sees the problem plus the global experience, and a student branch that sees only the problem. The student generates on-policy trajectories; the teacher rescored them at the token level with the experience in context; the student is trained to match the teacher distribution by minimizing reverse KL divergence. The teacher branch is detached from gradient computation.

Stage II — Implicit policy optimization. The distilled policy is then optimized with GRPO. For each problem a group of G responses is sampled, group-relative advantages are computed from the mean and standard deviation of rewards within the group, and a clipped importance-sampling surrogate objective is maximized. The argument is that because Stage I moves the policy toward high-reward regions, groups are more likely to contain a mix of correct and incorrect responses, producing non-zero advantages and therefore informative gradient signals — unlike the directly-applied GRPO baseline, where all-same-reward groups yield zero advantage and an uninformative update step. The authors note that other GRPO variants could be substituted in Stage II but that they use standard GRPO for simplicity.

Experimental setup. Three scales of the Qwen3 family (qwen3-1.7b/4b/8b-base) are trained on DAPO-math-17k, with training steps and total sampled rollouts controlled across methods. Five benchmarks are used: AIME24, AIME25, GSM8K, MATH500, and Minerva Math. GSM8K, MATH500, and Minerva Math are reported as pass@1; AIME 2024 and AIME 2025 are reported as pass@16 average. Baselines are Base, Domain Prompt, GRPO, and SDPO. Specific values for G, S, ε, and the learning rate are not given in the paper content provided.

Why This Matters

Impact on research. The paper makes a structural argument rather than only a performance one: it locates a missing stage between test-time experience use and reward-driven parameter updates, and gives it a formal objective. If that framing holds, it reframes self-improvement as a problem of representation — what to keep from a trajectory and how to compress it — rather than purely a problem of optimization algorithm choice. The mutual-information objective also gives a principled reason why distilling full reference trajectories can fail (instance-specific detail leaks in and invites post-hoc rationalization), which the SDPO results support.

Real-world applications.

• Mathematical and quantitative reasoning assistants that must improve from their own practice attempts rather than from new human-labeled data.
• Enterprise or domain assistants where feedback is verifiable (rules, compilers, unit tests) and trajectories accumulate over long deployment periods.
• Agentic workflows that need long-horizon memory to persist as capability rather than as retrieved context that disappears when the context window is cleared.
• Cost-constrained training pipelines, given the roughly 28% reduction in training data needed to reach a given performance level.

Industry relevance. The framework is a post-training recipe rather than a new architecture, so it is directly applicable to existing model checkpoints. The modest absolute scores on the hardest benchmarks (AIME-scale results remain in the single digits to low double digits) suggest the technique is a training-efficiency and reliability improvement rather than a step change, which matters for organizations that repeatedly retrain or fine-tune models and want more capability per rollout.

Future Directions

• Extending experience evolution beyond mathematical reasoning to task domains with sparser, less reliable, or more complex environmental feedback, as the authors themselves propose.
• Exploring multiple rounds in the main pipeline. The controlled experiment shows sampling accuracy rising monotonically from iteration 0 through 4, but the main results use only one evolution round; the authors state that multiple rounds may further strengthen the self-improvement loop.
• Substituting other GRPO variants in Stage II. The authors explicitly leave this open, noting it may yield better performance than the standard GRPO they used.
• Understanding when experience distillation fails. SDPO drops below the base model on qwen3-1.7b, which raises open questions about how small or weak a starting policy can be before the privileged-teacher signal becomes harmful, and how the experience pool should scale with policy capability.

Target Audience

Researchers and engineers working on LLM post-training, reinforcement learning for reasoning, and self-improving or self-evolving model systems. It is most useful to readers already comfortable with policy-gradient methods and distillation, since the mutual-information framing and the OPSD/GRPO formulations are presented at a technical level. Practitioners who repeatedly fine-tune models on verifiable-reward tasks and want to reduce rollout cost will also find the data-efficiency results directly relevant, while those new to RL-based post-training would benefit from the related-work sections before the method.

Authors’ abstract

Large language models (LLMs) capable of self-improvement require not only effective policy optimization, but also a principled mechanism for transforming transient interaction experience into persistent model capabilities. Existing self-improvement paradigms remain fragmented: test-time methods can explicitly extract experience but cannot internalize it into model parameters, whereas training-time optimization methods can update model parameters but lack an explicit mechanism for accumulating transferable experience. Bridging these two paradigms requires a critical intermediate stage that remains underexplored, namely \emph{experience distillation}. To address this gap, we propose \textbf{SPEE} (\textbf{S}elf-\textbf{P}rogressive \textbf{E}xperience \textbf{E}volution), a unified post-training framework that sequentially performs explicit experience evolution followed by implicit policy optimization. During explicit experience evolution, SPEE reflects on trajectories collected from multiple interactions to extract, verify, and progressively evolve transferable experience, which is subsequently internalized into the policy through privilege-guided On-Policy Self-Distillation (OPSD). During implicit policy optimization, reward-driven reinforcement learning leverages these internalized priors to explore novel solution strategies. In the experience evolution stage, a continuously evolving global experience pool consolidates knowledge from both successful and failed trajectories, filters out low-utility experience, and mitigates post-hoc rationalization induced by individual trajectories. Experiments on five mathematical reasoning benchmarks demonstrate that SPEE consistently outperforms both test-time and training-time self-evolution baselines across three model scales. The source code is available at https://github.com/rrrsj/SPEE.