Dr. Zero: Self-Evolving Search Agents without Training Data

AI summary

Overview

• Research area: Large language model agents, reinforcement learning for LLMs, agentic search / retrieval-augmented reasoning, and data-free self-evolution.
• Technical level: Intermediate. The paper assumes familiarity with policy-gradient methods such as GRPO and PPO, but the proposer-solver design and the reward scheme are explained in largely plain terms.
• Scope: The paper introduces Dr. Zero, a proposer-solver self-evolution framework that trains multi-turn search agents on Qwen2.5 3B/7B Instruct without any curated QA training data, using only an external search engine as the knowledge environment.

What This Paper Is About

Self-evolving LLMs can generate their own training problems, but existing data-free methods were built for narrow domains like math and coding, where questions are structurally simple. When applied to open-domain search agents, these methods produce mostly one-hop questions and require expensive nested sampling (multiple questions per seed prompt, each evaluated with multiple solver predictions). Dr. Zero's goal is to let a proposer and a solver co-evolve into competent multi-turn search agents with no human-written questions or answer annotations, while keeping the training compute practical.

Key Contributions

1. Dr. Zero framework. A unified self-evolution framework in which a proposer and a solver both act as search agents over an external search engine, combining a refined multi-turn tool-use rollout pipeline with a difficulty-guided reward to produce complex, multi-hop questions.
2. Hop-grouped relative policy optimization (HRPO). A new optimization method that clusters structurally similar questions (grouped by hop count) to build a robust group-level baseline for advantage estimation, removing the need for expensive nested sampling in self-evolution.
3. Empirical validation. Extensive experiments show that without curated QA training data, the search agents match the aggregate performance of a fully supervised search agent at the 3B scale and outperform supervised baselines on several individual benchmarks.
4. Application to open-domain QA specifically. Unlike prior data-free work targeting math/coding, Dr. Zero targets open-domain question answering, where it uses no demonstrations, human-written questions, or annotated answers, relying on external search as the knowledge environment.

Main Findings

• Aggregate parity with supervised agents at 3B. On Qwen2.5-3B-Instruct, Dr. Zero reaches an average of 0.326 across the seven benchmarks versus 0.327 for the supervised Search-R1 baseline.
• Single-hop gains for the 3B model. Dr. Zero outperforms supervised Search-R1 on the one-hop tasks by 22.9% on NQ (0.397 vs 0.323), 6.5% on TriviaQA (0.572 vs 0.537), and 18.4% on PopQA (0.431 vs 0.364).
• 7B multi-hop behavior. The 7B variant achieves roughly 90% of Search-R1's performance on complex multi-hop scenarios and outperforms it on 2WikiMQA (0.347 vs 0.326); its overall average is 0.372 versus 0.384 for Search-R1.
• Large margin over few-shot methods. At 3B on NQ, Dr. Zero achieves 0.397 EM, compared with 0.106 for standard prompting, 0.111 for IRCoT, and 0.238 for Search-o1.
• Beat existing data-free baselines. Against data-free methods augmented with multi-turn reasoning and search, Dr. Zero averages 0.326 versus 0.256 for R-Zero* and 0.233 for SQLM*, exceeding them by an average of 39.9% and 27.3% respectively, and yielding an average relative gain of 83.3% over R-Zero* across the four multi-hop benchmarks.
• Iterative self-evolution does help, then plateaus. The 3B model improves from 0.304 (iteration 1) to 0.319 (iteration 2) to 0.326 (iteration 3); the 7B model goes 0.366, 0.372, then drops to 0.360 in iteration 3. Second-iteration average gains were 4.93% and 1.64%; the third iteration gave only a modest 2.2% increase for 3B. Both solvers peaked within approximately 50 steps.
• HRPO is far cheaper than standard GRPO. Per proposer-training iteration, HRPO uses 6 rollouts (1 question plus 5 predictions), 4.16 hours wall-clock, 33.28 GPU-hours, and 21.63% GPU utilization, versus 20 rollouts (4 questions plus 16 predictions), 10.58 hours, 84.64 GPU-hours, and 17.84% utilization for GRPO, measured on a single 8xH100 node for the 3B model.
• HRPO is also slightly better on average. HRPO averages 0.326 versus 0.320 for GRPO, though GRPO is higher on HotpotQA (0.303 vs 0.298), MuSiQue (0.100 vs 0.091), and Bamboogle (0.272 vs 0.200).
• Mixed hop ratios matter more than maximizing multi-hop share. The default 4:3:2:1 ratio (1-, 2-, 3-, 4-hop) gives the 3B model its best multi-hop average of 0.220 EM; for the 7B model it gives the highest overall average (0.372), while other compositions do slightly better on some complex benchmarks.
• Ablations. Removing the format reward or the initial document hurts performance (e.g., without the initial document NQ falls to 0.273), and training for 100 steps instead of 50 does not consistently help.
• Training instability at larger scale. The most common failure mode stems from inconsistent token IDs across multi-turn search and reasoning steps, which the 7B model exhibits more frequently than the 3B variant.

Methodology in Plain English

Two copies of the same base model are created: a proposer, which writes questions using a search tool, and a solver, which answers them using the same search tool. The solver's success rate on each generated question becomes the proposer's reward: questions that every solver attempt gets right, or that no attempt gets right, earn no difficulty reward, while questions solved by exactly one of five attempts earn the most. A small format reward (maximum 0.5 total) encourages well-formed output with proper <think>, tool call, <question>, and <answer> structure, and the proposer prompt requires each hop and final answer to be supported by the seed passage or retrieved passages.

The key efficiency trick is HRPO. Standard GRPO would sample several candidate questions per seed prompt and then several solver answers per question, giving m(n+1) rollouts; in the paper's implementation that means 4 proposer rollouts and 4x4 solver rollouts, 20 total. HRPO instead generates one question per seed prompt and computes advantages by grouping questions with the same hop count, standardizing rewards within each hop group. That cuts the count to 6 rollouts. HRPO is strictly on-policy with ratio clipping omitted.

Training alternates in a loop: the proposer is trained for 50 steps with HRPO, then the solver is trained for 50 steps with GRPO on the generated data. Three iterations (150 steps per model) are used in total, which the authors note is significantly fewer than baselines like R1 and Search-R1. Evaluation uses exact match across NQ, TriviaQA, PopQA, HotpotQA, 2WikiMQA, MuSiQue, and Bamboogle, with an E5 base search engine over an English Wikipedia dump and top-3 documents retrieved per query.

Why This Matters

The work suggests that strong agentic search and evidence-grounded reasoning can emerge without any human-annotated training questions, which matters as high-quality training data becomes harder to obtain. It also shows that the compute bottleneck of self-evolution for multi-turn tool-use agents can be substantially reduced.

Real-world applications:

• Question answering systems that must keep improving in domains where labeled QA data is scarce or expensive to produce.
• Enterprise and research search assistants that operate over private corpora with a search engine as the only external knowledge source.
• Fact-checking or evidence-grounded answering pipelines where answers must be backed by retrieved passages.
• Cost-constrained agent deployment where reducing rollouts and GPU-hours per training iteration directly lowers the barrier to iterating on models.

Industry relevance: The 6-rollout versus 20-rollout comparison (4.16h vs 10.58h, 33.28 vs 84.64 GPU-hours) is a concrete efficiency argument for organizations that cannot afford nested sampling at multi-turn latency. The released code at https://github.com/facebookresearch/drzero and the use of widely available Qwen2.5 3B/7B Instruct backbones make the approach relatively reproducible.

Future Directions

• Extending self-evolution stability to overcome the performance plateaus the paper observes after roughly two to three iterations, and to prevent entropy collapse in larger models.
• Safeguarding the feedback loop against reward hacking and bias amplification, so that the self-evolution process stays reliable without human supervision.
• Handling less verifiable settings. The paper notes that grounding requirements and the absence of a difficulty reward for zero-pass cases reduce but do not eliminate incorrect or ambiguous synthetic answers, and suggests incorporating richer signals such as retrieval uncertainty or answer ambiguity.
• Isolating the sources of gain. The authors state that their evaluation does not separately isolate gains in agentic search from gains in reasoning over retrieved passages, leaving that decomposition open.

Target Audience

Researchers and engineers working on reinforcement learning for LLMs, agentic retrieval, and search-augmented question answering will benefit most. It is also relevant to practitioners building data-efficient training pipelines for tool-using agents, and to readers already familiar with GRPO-style group-based optimization who want to see it adapted to structurally heterogeneous, multi-hop queries.

Authors’ abstract

As high-quality data becomes increasingly difficult to obtain, self-evolution without curated training data has emerged as a promising paradigm. This approach allows large language models (LLMs) to autonomously generate and solve complex problems, thereby improving their reasoning capabilities. However, multi-turn search agents struggle in this setting due to limited question diversity and the substantial compute required for multi-step reasoning and tool use. In this work, we introduce Dr. Zero, a framework that enables search agents to effectively self-evolve without human-annotated training data, relying solely on an external search engine as their knowledge environment. In particular, we design a self-evolution feedback loop where a proposer generates structurally diverse questions to train a solver initialized from the same base model. As the solver evolves, it incentivizes the proposer to produce increasingly difficult yet solvable tasks, thus establishing an automated curriculum to refine both agents. To enhance training efficiency, we also introduce hop-grouped relative policy optimization (HRPO). This method clusters structurally similar questions to construct group-level baselines, effectively minimizing the sampling overhead in evaluating each query's individual difficulty and solvability. Consequently, HRPO significantly reduces the compute requirements for proposer training and reward estimation without compromising performance or stability. Extensive experimental results demonstrate that Dr. Zero matches or surpasses fully supervised search agents on several question answering benchmarks, showing that strong agentic search and evidence-grounded reasoning can emerge solely through self-evolution.