IterSynth: Rethinking Deep Search Agents via Role-Decoupled Iterative Synthesis

AI summary

Overview

Research area: Natural language processing, specifically deep-search / deep-research LLM agents trained with reinforcement learning.

Technical level: Advanced (the paper formalizes the agent as a Markov Decision Process and introduces a policy-optimization algorithm, though the workflow idea itself is explainable without that background).

Scope: The paper proposes IterSynth, a dual-role (Planner–Synthesizer) iterative search workflow built on a single shared 8B policy, plus a role-decoupled reinforcement learning method (RDPO), and evaluates it on five long-horizon deep-search benchmarks.

What This Paper Is About

Deep-search agents — systems that take a hard question, issue web searches, read the results, and produce a grounded answer — are usually built as a single ReAct-style policy that reasons, searches, and writes in one ever-growing context. This works poorly as the search horizon gets longer, because one policy must juggle very different skills and because accumulated retrieved text crowds out the useful evidence.

IterSynth's goal is to split that single tangled loop into two alternating roles played by the same model: a Planner that decides what to search for next, and a Synthesizer that folds retrieved evidence into a compact, evolving summary. The paper then asks how to train this workflow so that both roles actually become competent.

Key Contributions

1. IterSynth, a role-decoupled, summary-based deep-search paradigm. Instead of a single monolithic context, the agent alternates between a Planner (chooses the next query or emits the final answer) and a Synthesizer (filters retrieved evidence, resolves inconsistencies, and writes an updated global summary). Both roles are executed by one policy with shared parameters, distinguished only by role-specific prompts and action constraints, so no extra models or parameter count are added.

2. Role-Decoupled Policy Optimization (RDPO). An RL method that combines a terminal outcome reward (final-answer correctness) with turn-level rubric scores from an LLM judge across five predefined dimensions, then normalizes advantages separately within each role's reward pool rather than across mixed-role samples.

3. A two-stage training recipe. Cold-start supervised fine-tuning on roughly 10K filtered Planner–Synthesizer trajectories synthesized with Qwen3.5-397B-A17B, followed by RDPO. Each trajectory averages 4.37 iterations, and per-turn unrolling yields about 87.4K role-conditioned supervised samples.

4. Comprehensive evaluation on five long-horizon benchmarks (BrowseComp, BrowseComp-ZH, GAIA text-only, Xbench-DeepSearch 2505 and 2510), showing IterSynth works both as a trained agent and as a training-free prompting workflow for frontier proprietary models.

Main Findings

• IterSynth-8B reaches an average of 50.7% across the five benchmarks, the best average among trained agents at or below 8B parameters, improving on the strongest prior small agent (MiroThinker-v1.0-8B, 46.5%) by +4.2%.

• Gains are concentrated on exploration-intensive tasks. On BrowseComp-ZH, IterSynth-8B scores 55.4%, surpassing the strongest small-agent baseline by +15.2%; on Xbench-DS-2510 it gains +5.4% (46.0%). On BrowseComp (30.9%) and GAIA text-only (55.3%) it scores below MiroThinker-v1.0-8B (31.1% and 66.4% respectively), which the paper describes as remaining competitive rather than leading.

• The 8B agent is competitive with several 30B-scale agents. It exceeds ReSum-30B (33.3%), AgentFold-30B-A3B (50.2%), and OpenSeeker-30B-SFT (50.6%) in average score, and approaches IterResearch-30B-A3B (56.6%) and WebSailor-V2-30B (56.8%) at less than one third of the parameter budget.

• Role-decoupled advantages, not dense rewards alone, drive the improvement. SFT alone scores 44.1% average; outcome-only GRPO raises this to 48.9%; RDPO reaches 50.7%. A variant using the same composite reward but normalizing advantages across mixed roles ("w/o RD") falls to 47.2%, below even the outcome-only GRPO baseline.

• Both roles are genuinely trained, and the Planner is the harder role. Replacing the trained Synthesizer with the untrained Qwen3-8B drops the three-benchmark average by 17.4% (58.9 → 41.5), while replacing the trained Planner drops it by 41.1% (58.9 → 17.8).

• IterSynth also works as a prompting-only workflow for frontier models. With no parameter updates, it beats ReAct by +5.5% average on Claude-4.5-Opus (66.1 vs 60.6) and +4.5% on DeepSeek-V3.1 (47.9 vs 43.4), and outperforms IterResearch on both backbones overall. The largest single gap is on BrowseComp-ZH with Claude-4.5-Opus, up to +10.0% over ReAct.

• Context exhaustion in ReAct is empirically severe. The paper reports in its appendix that even with a 64K context, ReAct trajectories on BrowseComp fail to terminate before running out of context in over 59% of cases.

Methodology in Plain English

The agent's memory is not a transcript. It is a single global summary, M_t, built from the original question plus everything the Synthesizer has decided is worth keeping so far.

Each round works like this. The Planner sees only the question and the current summary — not the full history of past queries and retrieved pages — and decides either to issue one search query or to stop and give the final answer. If it searches, the environment returns results. The Synthesizer then reads those results, throws away irrelevant material, extracts useful findings, reconciles contradictions, and rewrites the summary. The workspace is rebuilt from the question and the new summary at the start of the next round, so context never grows without bound. Critically, both roles are the same model with the same weights; only the prompt and the permitted actions differ.

Training happens in two stages. First, cold-start SFT: a frontier model (Qwen3.5-397B-A17B) rolls out the dual-role loop in a live search environment, and the resulting trajectories are cleaned — invalid tool calls repaired, hallucinated steps removed, only trajectories ending in verifiably correct answers kept — leaving roughly 10K trajectories from the Qwen3-8B backbone. This teaches the format but not when to search or what to keep.

Second, RDPO. For each query the policy runs G independent rollouts, each unfolding over several iterations, so one query yields many interaction rounds rather than a handful of trajectory-level samples. Each turn gets a composite reward: a sparse 0/1 correctness signal from the final answer, broadcast to every turn of that rollout, plus a dense rubric score from an LLM judge. The key trick is that advantages are normalized separately for Planner turns and Synthesizer turns, since the two roles receive rewards of very different density — mixing them into one baseline, the ablation shows, corrupts credit assignment. The resulting advantages plug into a standard group-relative policy optimization update with no further modification.

For the environment, the agent gets a SerpAPI Google Search tool returning titles, URLs, and snippets of the top-10 results, plus a browser that summarizes a given URL conditioned on a sub-query, using Jina Reader for HTML and PDF parsing. During RL rollouts all tool calls are served from an offline cache to keep cost predictable; at evaluation the same interface connects to the live web. SFT used 32 H20 GPUs (96 GB each) for about 24 hours with a learning rate of 1e-5, batch size 512, 4 epochs, and a 32,768-token max context; RL used the verl framework.

Evaluation covers BrowseComp (1,266 questions in the public release), BrowseComp-ZH (289 questions), GAIA text-only validation (103 questions), and Xbench-DeepSearch 2505 and 2510 (100 questions each), with no benchmark data used in training.

Why This Matters

The paper argues that training algorithms alone cannot fix deep-search agents, because the trained policy is still bound to whatever workflow it is deployed in. IterSynth's claim is that workflow design and optimization method must be co-designed: decoupling the roles makes the context bounded and the credit assignment tractable, and role-aware RL makes the decoupling pay off. If the pattern generalizes, small models can take on long-horizon research tasks that currently require much larger ones — the paper shows an 8B agent matching or exceeding several 30B agents at less than one third of the parameter budget.

Real-world applications:

• Automated research assistance, where a system decomposes a broad question, searches the web across many rounds, and returns a cited synthesis rather than a list of links.
• Multi-hop fact verification and due diligence, where evidence must be gathered across many documents and contradictions reconciled before an answer is issued.
• Cross-lingual information retrieval, given the especially large gain on the Chinese-language BrowseComp-ZH split.
• Deployment-constrained assistants, since a trained 8B agent is far cheaper to serve than a 30B or proprietary frontier model while the same workflow can also be used as prompting for existing frontier APIs.

Industry relevance: the paper comes from Zhejiang University and Tencent, and it targets the deep-research product category currently occupied by OpenAI Deep Research, Gemini Deep Research, Grok DeepSearch, Perplexity, and Claude. Its practical pitch to industry is a structural change to the agent loop that costs no additional parameters, plus a training recipe that a mid-sized model can absorb.

Future Directions

• Why the Planner matters so much more than the Synthesizer. The Planner-swap ablation (-41.1%) versus the Synthesizer-swap ablation (-17.4%) suggests planning is the bottleneck; whether better planning-specific training signals, or a different division of labor, could close the remaining gap to 30B agents is open.

• Closing the per-benchmark gaps. IterSynth-8B trails MiroThinker-v1.0-8B on GAIA text-only and BrowseComp while leading sharply on BrowseComp-ZH; understanding whether this reflects training data composition or something structural about the workflow is unresolved.

• Scaling the recipe. The paper trains only on Qwen3-8B and evaluates 30B-scale agents trained under other recipes. Whether RDPO's role-decoupled advantages deliver the same benefit at 30B or beyond is not tested.

• Training and inference cost trade-offs. The paper notes in an appendix that a lightweight "not-attempted resampling" strategy converts extra inference compute into consistent accuracy gains, and that RL rollouts use cached retrieval while evaluation uses live tools — leaving open how the method behaves on the live distribution during training and how far compute scaling can substitute for parameters.

Target Audience

Researchers and engineers working on LLM agents, tool-use training, and reinforcement learning for multi-turn decision making — particularly those building or serving deep-research systems who need long-horizon behavior from models small enough to deploy cheaply. The paper assumes familiarity with ReAct-style agents, group-relative policy optimization, and MDP formalisms, so readers without an RL background will get the most from the workflow design and empirical comparisons rather than the optimization section.

Authors’ abstract

Deep search requires LLM agents to decompose complex queries, search for evidence, and synthesize grounded answers, yet existing ReAct-style agents suffer from two limitations: role coupling, where one policy must handle planning, evidence use, and synthesis; and context accumulation, where growing search histories introduce noise and obscure useful information. To address these issues, we propose IterSynth, a role-decoupled and summary-based paradigm that alternates between a Planner for identifying information needs and a Synthesizer for integrating evidence into an evolving summary state. This design separates planning from synthesis while using the summary as the persistent state of search, reducing both capability coupling and context noise. To train IterSynth effectively, we further introduce Role-Decoupled Policy Optimization (RDPO) for reinforcement learning, which combines terminal outcome rewards with turn-level rubric evaluations and computes role-specific advantages for more precise credit assignment. Experiments on five long-horizon deep-search benchmarks such as BrowseComp and Xbench-DS show that IterSynth-8B achieves an average score of 50.7, surpassing the strongest prior $\leq$8B agent by +4.2\%. Moreover, IterSynth serves as a model-agnostic prompting paradigm, delivering substantial zero-shot gains over ReAct and similar prompting paradigms on frontier proprietary models.