EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecycle

AI summary

Overview

Research area: Large language model (LLM) agents, experience-driven continual learning, tool-augmented reasoning, and reinforcement learning for multi-turn question answering.

Technical level: Intermediate. The paper assumes familiarity with LLM agents, retrieval-augmented generation, and policy-gradient reinforcement learning (specifically GRPO), but its central ideas are conceptual and readable without deep RL expertise.

Scope: EvolveR proposes a closed-loop lifecycle in which an LLM agent collects its own interaction trajectories, distills them into reusable natural-language "principles," retrieves those principles to guide future reasoning, and updates its policy via reinforcement learning — outperforming strong search-augmented baselines on multi-hop QA.

What This Paper Is About

Most LLM agents treat every task as an isolated episode: they solve a problem, discard everything they learned, and start fresh on the next one. Existing remedies — storing raw past trajectories or having a stronger external model write reflections — either fail to generalize or produce insights misaligned with the agent's own reasoning style. EvolveR's goal is to give an agent a complete, self-contained learning cycle so it can convert its own successes and failures into abstract strategic principles, apply them to new problems, and reinforce the behaviors that worked.

Key Contributions

1. An experience-driven self-evolution paradigm. A closed-loop lifecycle alternating between an online phase (interaction, tool use, RL policy updates) and an offline phase (frozen-parameter self-distillation and experience-base maintenance), enabling the agent to continually transform raw trajectories into curated strategic knowledge.

2. A dynamic experience curation system. A full pipeline for building and maintaining the experience base: self-distillation of principles (each a natural-language statement plus structured knowledge triples), semantic deduplication, merging of new evidence into existing principles, and quality control via a usage/success metric score with periodic pruning.

3. Reinforcement learning that learns to use experience, not just solve tasks. Because online trajectories are generated while conditioned on retrieved principles, GRPO updates explicitly teach the policy how to exploit its own distilled wisdom, coupling retrieval quality directly to reward.

4. Extensive empirical validation across model scales and ablations. Seven QA benchmarks, four model sizes (0.5B–7B), and component-level ablations that isolate the roles of distillation, retrieval, and RL, including a direct comparison of self-distillation against distillation by a stronger external teacher (GPT-4o-mini).

Main Findings

• EvolveR outperforms strong agentic baselines. On Qwen2.5-3B it reaches an average Exact Match of 0.382 and on 7B 0.417, beating all baselines including Search-R1-instruct (0.325 and 0.385) and RAG, SFT, and rejection-sampling approaches. Gains appear across in-domain (NQ, HotpotQA) and out-of-domain (TriviaQA, PopQA, 2WikiMultiHopQA, Musique, Bamboogle) benchmarks.

• Self-distillation overtakes teacher-distillation as models scale. At 0.5B, using GPT-4o-mini as the distiller clearly wins (0.220 vs. 0.150 average). By 3B the relationship reverses: self-distilled principles win (0.382 vs. 0.370), which the authors attribute to better "cognitive alignment" between the principles and the student model's own failure modes.

• Retrieval at inference time is indispensable. Removing access to the experience base while keeping identical RL training drops 3B performance from 0.382 to 0.340, and 0.5B from 0.150 to 0.078 — a steeper collapse at smaller scales.

• Distillation, retrieval, and RL are complementary. Component ablation on 3B: no components 0.134; RL alone 0.325; distillation + retrieval without RL 0.357; distillation + RL without retrieval 0.340; full framework 0.382. Retrieval supplies the guidance; RL optimizes how to use it.

• Dynamic scoring effectively filters bad principles. Manual annotation of 100 principles shows the low-score tier contains 26% ideal, 50% vague, and 24% incorrect/misleading principles, while the high-score tier contains 82% ideal, 10% vague, and 8% misleading — confirming that the metric score and pruning mechanism separate useful from harmful advice. The dominant failure mode is vagueness rather than outright misinformation.

• The experience base scales gracefully. Growing the base to roughly 50,000 principles raised top-3 retrieval latency only to 0.20 seconds and did not degrade performance, which peaked at the 45k scale (0.410) before a mild dip at ~50k.

• Performance scales monotonically with base model size. Average scores rise from 0.150 (0.5B) to 0.270 (1.5B) to 0.382 (3B), indicating EvolveR acts as a synergistic layer rather than a size-specific trick.

Methodology in Plain English

EvolveR runs two alternating phases. In the online phase, the agent solves multi-hop questions in a think-act-observe loop with three possible actions: search its internal experience base for principles, search an external knowledge base for facts, or emit a final answer. Its policy is updated with GRPO using a composite reward: a sparse binary outcome reward for answer correctness (exact match) plus a dense format reward that encourages a balanced number of reasoning steps, use of both search types, and structurally complete trajectories (at least one think, one search, and one answer).

In the offline phase, parameters are frozen and the agent reviews its own trajectories, adopting an expert persona via prompting to extract either a guiding principle from a success or a cautionary principle from a failure. Each principle is a natural-language rule plus structured knowledge triples. New principles go through two levels of deduplication — pairwise equivalence checks among principles from the same problem, then embedding-similarity retrieval followed by an LLM equivalence judgment against the existing base (similarity threshold 0.85). Novel principles are added; near-duplicates are merged, attaching the new trajectory as supporting evidence.

Each principle tracks how often it is used and how often it leads to success, yielding a Beta-style score of (successes + 1) / (uses + 2). Principles whose scores fall below 0.3 are periodically pruned. Training begins with a cold-start SFT stage on roughly 700 curated CoT trajectories from NQ and HotpotQA to teach the interaction format, then proceeds with LoRA fine-tuning and GRPO (G = 8 sampled trajectories per prompt, 128 prompts per batch) on 8 A100 GPUs using the Verl framework.

Why This Matters

Impact on research. EvolveR reframes agent improvement as building and refining internal expertise rather than acquiring more external knowledge. It provides a concrete alternative to raw-trajectory replay and externally supervised reflection, and it contributes an unusually detailed empirical result: self-distilled experience eventually beats distillation by a stronger teacher, which challenges the common assumption that better distillation always requires a bigger model.

Real-world applications:

• Multi-hop research and enterprise search assistants that improve their query-decomposition strategies over repeated use rather than starting each session cold.

• Domain-specific copilots in medicine, law, or finance, where an agent can build a private library of validated reasoning heuristics tailored to its own model and the recurring failure modes of its domain.

• Customer support and IT triage agents that accumulate cautionary principles from past misdiagnoses and avoid repeating them.

• Autonomous tool-using pipelines (data analysis, code agents, robotics planners) where procedural correctness matters as much as the final output and the format reward provides a way to shape structured behavior.

Industry relevance. The framework is model-agnostic, works at small parameter counts (down to 0.5B), and its retrieval cost stays negligible even at 50,000 stored principles. That combination matters for teams that need continuously improving agents without retraining from scratch or paying for a frontier model on every reflection step.

Future Directions

• Breaking the ceiling imposed by base-model capability. The authors note explicitly that principle quality is bounded by the model's own reasoning ability; investigating how to bootstrap or augment weaker models remains open.

• Extending beyond question answering. The lifecycle is demonstrated only on multi-hop QA. Whether the same distillation-and-RL loop transfers to code generation, embodied control, or long-horizon planning is untested.

• Richer experience representations. Principles are currently one natural-language statement plus knowledge triples. Hierarchical, compositional, or conditional principles (e.g., strategy A works when X holds) could capture more structure than flat entries.

• Better handling of vague principles. Vagueness, not incorrectness, is the dominant quality failure at 50% of the low-score tier, suggesting room for refinement prompts or automatic specificity checks before a principle enters the base.

Target Audience

Researchers and engineers working on LLM agents, agentic RL, and continual learning who want to move beyond stateless prompting and retrieval. It is most useful to practitioners building self-improving systems on open-weight models (the paper's small-model scaling results are directly actionable for resource-constrained deployments), and to RL researchers interested in how experience retrieval conditionally shapes policy optimization. Readers seeking purely prompt-engineering solutions will find the RL and infrastructure components heavier than they need, while those without any RL background will need to read the GRPO objective as a black box.

Authors’ abstract

Current Large Language Model (LLM) agents show strong performance in tool use, but lack the crucial capability to systematically learn from their own experiences. While existing frameworks mainly focus on mitigating external knowledge gaps, they fail to address a more fundamental limitation: the inability to iteratively refine problem-solving strategies. In this work, we introduce EvolveR, a framework designed to enable agent to self-improve through a complete, closed-loop experience lifecycle. This lifecycle comprises two key stages: (1) Offline Self-Distillation, where the agent's interaction trajectories are synthesized into a structured repository of abstract, reusable strategic principles; (2) Online Interaction, where the agent interacts with tasks and actively retrieves distilled principles to guide its decision-making, accumulating a diverse set of behavioral trajectories. This loop employs a policy reinforcement mechanism to iteratively update the agent based on its performance. We demonstrate the effectiveness of EvolveR on complex multi-hop question-answering benchmarks, where it achieves superior performance over strong agentic baselines. Our work presents a comprehensive blueprint for agents that learn not only from external data but also from the consequences of their own actions, paving the way for more autonomous and continuously improving systems. Code is available at https://github.com/Edaizi/EvolveR.