IterResearch: Rethinking Long-Horizon Agents with Interaction Scaling

AI summary

Overview

Research area: AI agents and long-horizon reasoning — specifically deep-research agents that autonomously browse the web and synthesize information.

Technical level: Intermediate. The paper is accessible to readers with basic familiarity with LLM agents and reinforcement learning, though the MDP formalism and policy optimization details are more advanced.

Scope: This paper proposes IterResearch, an iterative agent architecture that replaces the standard "accumulate everything in one context window" approach with periodic synthesis and workspace reconstruction, and demonstrates its benefits in both trained and prompting settings.

What This Paper Is About

Deep-research agents — systems that autonomously search the web and reason over many steps — currently keep every search result and reasoning step in a single, ever-growing context window. This causes two failures: the window fills up ("context suffocation"), and irrelevant or erroneous early information permanently poisons later reasoning ("noise contamination"). IterResearch replaces this linear accumulation with an iterative loop that keeps only a constant-sized workspace: the question, an evolving report that acts as compressed memory, and the most recent interaction. The goal is to let agents keep reasoning at full quality no matter how deep the exploration goes.

Key Contributions

1. An MDP-inspired iterative deep-research paradigm. The agent operates in a workspace reconstructed at every round containing only the question, an evolving report (memory), and the immediate prior action and its result. Because the report is generated naturally by the LLM as part of its structured decision output, no separate memory module is needed, and the workspace size stays roughly O(1) instead of growing as O(t).

2. Efficiency-Aware Policy Optimization (EAPO). A training strategy combining geometric reward discounting (γ^(T−t)·R_T) — which rewards shorter successful trajectories more than long, meandering ones — with adaptive downsampling to handle the variable number of per-round samples in distributed training while losing under 1% of data. Built on top of GSPO.

3. Empirical validation across six benchmarks. IterResearch-30B-A3B (Qwen3-30B-A3B backbone, RFT then RL training) beats all open-source agents by an average of 14.5 percentage points and rivals or surpasses some proprietary systems.

4. Three broader findings: extreme interaction scaling (up to 2048 interactions), cross-paradigm knowledge transfer (iterative trajectories improve mono-contextual agents), and usefulness as a training-free prompting strategy for frontier models.

Main Findings

• Large open-source gains: IterResearch-30B-A3B averages 49.1% across HLE, BrowseComp, BrowseComp-zh, GAIA, Xbench-DS, and SEAL-0 — a +14.5pp average margin over the best open-source agents. It surpasses OpenAI DeepResearch on HLE (28.8 vs 26.6) and BrowseComp-zh (45.2 vs 42.9), and is competitive on BrowseComp and GAIA.

• Interaction scaling to 2048 turns: Performance rises monotonically with the interaction budget on BrowseComp, with the steepest gains between 2^4 and 2^7 turns. The paper reports improvement from 3.5% to 42.5% (abstract) and 5.5% to 50.1% on a BrowseComp-200 subset (Section 4.4) as budget increases from 2 to 2048 turns — all while staying within a constant 40K-token workspace. Mono-contextual agents cannot reach this horizon at all.

• Intelligent budget use: Given 2048 turns, the agent uses only ~80.1 turns on average, terminating once enough information is gathered. Average turn count grows sublinearly with the budget, indicating the agent learns more efficient search strategies rather than simply searching longer.

• EAPO improves efficiency without hurting accuracy: Compared to standard GSPO, EAPO reduces average interactions by 5.7% (18.04 vs 19.13 turns) while maintaining or slightly improving accuracy (49.1 vs 48.3 average). SFT alone lags at 45.5 average.

• The paradigm beats a larger-context mono-contextual agent: Under identical training data, the iterative paradigm outperforms a mono-contextual baseline by 12.6pp on average — even though the baseline was deliberately given a larger context window (64K vs 40K tokens). The paper argues this shows workspace suffocation is not fixable by simply enlarging the context.

• Cross-paradigm knowledge transfer: Adding IterResearch trajectories to a mono-contextual agent's training data yields a +5.4pp average improvement, showing the iterative paradigm produces higher-quality training signals that transfer across architectures.

• Works as a prompting strategy: Without any training, the iterative structure applied to o3 and DeepSeek-V3.1 beats ReAct on every benchmark, with the largest gains on the most exploration-heavy task, BrowseComp (+12.7pp for o3, +19.2pp for DeepSeek). The authors highlight this as model-agnostic.

Methodology in Plain English

The core loop. Each round, the agent sees a small workspace: the fixed question, a running report that summarizes what has been found so far, and the previous action plus its result. From that workspace, the model produces three outputs: a private reasoning step (Think), an updated report (Report), and an action taken in the environment — either a search or the final answer (Action). The environment returns a result (like a web page). The next workspace is then rebuilt from only the question, the new report, and this latest action-result pair. Everything older is deliberately discarded.

Why this works. The report acts as compressed, filtered memory. Because the model itself writes the report, it performs the information-selection work naturally. Noise and early errors must survive the report-writing step to affect later rounds, so transient mistakes get filtered out instead of accumulating. This also keeps the workspace size constant, so reasoning quality does not decay as exploration deepens.

Training. Two stages: rejection-sampling fine-tuning (RFT) first, to teach the model the iterative format, then reinforcement learning. The RL reward is binary at termination (correct answer or not), but is geometrically discounted by how many steps remained after each action — so actions on shorter successful trajectories get higher credit. Every round of every trajectory becomes an independent training sample, producing many more samples per question than mono-contextual training. Adaptive downsampling trims the sample set to the largest multiple of the data-parallel size, keeping distributed training stable.

Evaluation. Six benchmarks spanning web navigation, multi-hop information seeking, complex reasoning, and cross-lingual tasks, compared against direct-inference frontier models, proprietary deep-research systems, and open-source agents.

Why This Matters

Impact on research. The paper reframes long-horizon agent design: instead of asking "how big should the context window be?" it asks "what should be kept in the workspace at all?" The reported interaction scaling suggests the perceived difficulty of long-horizon tasks may partly reflect insufficient exploration capacity rather than task-inherent hardness. The parallel the authors draw to AlphaEvolve — which independently converged on a similar "evolving report plus immediate context" structure for code evolution — hints this is a general design pattern for long-horizon AI, not a web-search-specific trick.

Real-world applications:

• Automated literature and market research that must comb hundreds of sources without losing coherence.
• Investigative and due-diligence workflows where evidence must be progressively distilled rather than dumped.
• Customer-support and operations agents running long multi-step troubleshooting at bounded token cost.
• Scientific and engineering assistants performing extended hypothesis search with periodic synthesis.

Industry relevance. Every additional interaction in a deployed agent costs API calls and latency, so EAPO's efficiency pressure has direct operational value. The prompting result is arguably the most immediately deployable finding: teams can apply the iterative structure to existing frontier models today, with no training pipeline, and get double-digit gains on long-horizon tasks. The fact that a 30B-parameter open model approaches proprietary deep-research systems also matters competitively.

Future Directions

• Why does interaction scaling plateau? Accuracy flattens after roughly 2^7 turns even with 2048 available. Understanding whether this reflects task saturation, report bottlenecking, or policy limitations could unlock further gains.
• Report quality under pressure. The whole paradigm hinges on the model's ability to compress findings without losing decision-relevant information. How the report degrades on very noisy or adversarial sources, and whether explicit report-verification mechanisms would help, is unresolved.
• Deeper cross-paradigm transfer analysis. The +5.4pp transfer result is presented without a mechanistic explanation. Which exploratory behaviors actually transfer, and whether they can be distilled directly into a mono-contextual model, remains open.
• Extending beyond web research. The paper itself notes the structural similarity to AlphaEvolve in code evolution. Testing the same iterative-synthesis architecture on other long-horizon domains — robotics planning, multi-session coding, scientific discovery — would test the generality claim.

Target Audience

This paper is most useful to researchers and engineers building autonomous LLM agents, particularly those working on tool use, web navigation, or retrieval-augmented reasoning. It is also valuable to reinforcement-learning practitioners interested in reward shaping for long-horizon, sparse-reward tasks, and to applied teams who want a drop-in prompting pattern for improving existing frontier models without training. Readers wanting to understand the MDP formulation and the EAPO objective in detail will need a working knowledge of policy-gradient methods.

Authors’ abstract

Recent advances in deep-research agents have shown promise for autonomous knowledge construction through dynamic reasoning over external sources. However, existing approaches rely on a mono-contextual paradigm that accumulates all information in a single, expanding context window, leading to context suffocation and noise contamination that limit their effectiveness on long-horizon tasks. We introduce \textbf{IterResearch}, a novel iterative deep-research paradigm that revisits long-horizon research through the lens of Interaction Scaling. Instead of relying on linear context accumulation, we adopt an MDP-inspired architecture with strategic workspace reconstruction. By maintaining an evolving report as memory and periodically synthesizing insights, our approach preserves consistent reasoning capacity across arbitrary exploration depths. To effectively train this paradigm, we employ Efficiency-Aware Policy Optimization (EAPO), a training strategy that adapts geometric reward discounting to incentivize efficient exploration and utilizes adaptive downsampling for stable distributed training. Extensive experiments demonstrate that IterResearch achieves substantial improvements over existing open-source agents with average +14.5pp across six benchmarks and narrows the gap with frontier proprietary systems. Remarkably, our paradigm exhibits unprecedented interaction scaling, extending to 2048 interactions with dramatic performance gains (from 3.5\% to 42.5\%), and serves as an effective prompting strategy, improving frontier models by up to 19.2pp over ReAct on long-horizon tasks. These findings position IterResearch as a versatile solution for long-horizon reasoning, effective both as a trained agent and as a prompting paradigm for frontier models.