Incremental Open-Ended Deep Research with Structured Harness

AI summary

Overview

Research area: Natural Language Processing, specifically LLM-based agents for Open-Ended Deep Research (OEDR) and the automated maintenance of long-form research reports.

Technical level: Intermediate. Readers need some familiarity with LLM agents, retrieval workflows, and report-level evaluation metrics such as RACE, ROUGE-L, BLEU-4, and Exact Match.

Scope: The paper proposes a new research setting, Incremental Open-Ended Deep Research (Incremental-OEDR), a supporting agent framework called Structured Harness, and a ten-year temporal evaluation protocol comparing incremental report updating against regenerating reports from scratch.

What This Paper Is About

Existing deep research systems treat every report as a one-off artifact: given a query, the system searches from the original query and writes a brand-new report, even if a nearly identical report already exists from an earlier point in time. The authors argue this is wasteful and inconsistent for practical scenarios where reports must be revisited periodically as new information appears, such as industry analyses or economic updates.

The paper's goal is to make report maintenance an incremental process, where a previous report is treated as part of the research state, and each update preserves knowledge that is still valid, revises what is outdated or incomplete, and adds newly discovered information. To do this, the authors build a structured interface that lets an agent retrieve, refresh, and edit existing report components instead of regenerating everything.

Key Contributions

1. A new research setting, Incremental-OEDR. Instead of mapping a query directly to a new report, the system produces the next report version from the query, the previous report, and currently available information, forming an update chain R(t-1) → R(t) → R(t+1) → ...

2. Structured Harness. A framework that represents a report as a structured collection of outlines, sections, and evidence, and adds three capabilities: Structured Retrieval (selectively access prior outline, sections, and citations), a Structured Evidence Pool (persistently store and refresh prior evidence), and Structured Generation (edit specific sections while preserving unaffected content).

3. A temporal evaluation framework spanning ten years. The authors define a Single-Step Task (SST) for one incremental update and a Long-Chain Task (LCT) for sequences of consecutive updates, with metrics grouped into Report Quality, Report Continuity, and Research Cost.

4. Extensive experiments on two benchmarks under two model configurations. Results on DeepResearch Bench and DeepConsult under an Open-source Configuration (OC) and a Proprietary Configuration (PC) show competitive report quality alongside large continuity gains and lower cost.

Main Findings

• Report quality is preserved or slightly improved. On DeepResearch Bench, IOEDR(OC) averages a RACE Overall score of 49.11 in the Single-Step Task versus 48.75 for the MS-Agent(OC) baseline, and 49.20 in the Long-Chain Task, which the authors describe as a 0.45-point improvement over MS-Agent(OC). Under the proprietary configuration, IOEDR reaches 51.07 (SST) and 51.25 (LCT) versus 51.05 for independently generated reports.

• The same quality trend holds on DeepConsult. IOEDR(OC) scores 6.77 (SST) and 6.80 (LCT) against 6.73 for MS-Agent(OC); IOEDR(PC) scores 6.98 and 7.06 against 6.97 for MS-Agent(PC).

• Report continuity improves sharply. On DeepResearch Bench, IOEDR(OC) achieves content-level ROUGE-L F1 of 0.57 (SST) and 0.68 (LCT), while MS-Agent(OC) achieves 0.17 and 0.13. At the outline level, IOEDR(OC) reaches 0.60 and 0.67 EM F1 versus 0.04 for MS-Agent(OC). On DeepConsult, content-level ROUGE-L F1 rises from 0.14 for MS-Agent(OC) to 0.59 (SST) and 0.67 (LCT). The abstract summarizes these as up to 0.51 higher content-level ROUGE-L F1 and 0.63 higher outline-level EM F1.

• Research cost drops substantially. On DeepResearch Bench, IOEDR(OC) reduces total token consumption from 3.78M to 2.96M in SST and 2.52M in LCT (described as approximately 22% and 33% reductions), and search calls from 45.19 to 18.62 and 17.85. On DeepConsult, total tokens go from 4.47M to 2.79M (SST) and 2.81M (LCT), and search calls from 49.77 to 33.54 and 25.22. The abstract reports up to 33% lower token consumption and 61% fewer search calls than OEDR on DeepResearch Bench.

• Long-chain updates cost less per report than single-step updates. The authors attribute this to the persistent Structured Evidence Pool: in LCT each update starts from the previous Incremental-OEDR report and its evidence pool, so prior evidence can be reused and only changed sources refreshed. On DeepResearch Bench, total tokens fall from 2.96M (SST) to 2.52M (LCT), a 14.9% reduction, with search calls falling from 18.62 to 17.85 (4.1%). On DeepConsult, the reported token totals are 2.79M (SST) and 2.81M (LCT) — the text describes this as a reduction even though the two figures are nearly equal — and search calls fall from 33.54 to 25.22 (24.8%).

• OEDR performance varies by timestamp. On DeepResearch Bench, MS-Agent(OC) ranges from 47.57 in 2016 to 49.28 in 2025; on DeepConsult it rises from 6.31 to 7.27 over the same period. The authors use this to argue that single-timestamp evaluation does not fully characterize system performance.

• More frequent updates show diminishing returns. In the update-interval study under SST(OC) on DeepResearch Bench, normalized quality is 0.984 (10 years), 1.002 (5 years), 1.007 (1 year), and 1.004 (1 month); normalized continuity is 1.951, 2.839, 3.353, and 3.360; normalized cost is 1.446, 0.972, 0.783, and 0.694. The 10-year interval costs 1.446 times the OEDR baseline while yielding lower quality (0.984), so the paper uses a 1-year update interval throughout.

Methodology in Plain English

The authors build on ms-agent, an open-source OEDR agent, and organize work across three collaborating agents: a Researcher that coordinates the plan and delegates, a Searcher that retrieves external information, and a Reporter that constructs and updates the report.

Rather than letting the agent write a report from scratch, they wrap the research environment in a "harness" that makes the report machine-editable. Each report is stored as a structure of outline, sections, and evidence, so the agent can list chapters, read a specific chapter, or read the whole prior report instead of re-reading everything. A persistent evidence pool stores each source's URL, title, raw content, and update timestamp. During an update, a batch refresh tool refetches prior sources and computes character-level similarity against the stored version; if similarity falls below a threshold τ of 0.9, an LLM performs claim-level diff analysis to identify what changed. Newly retrieved sources are added to the pool for later reuse. Generation then operates as structured editing: a section is preserved if its knowledge is still valid, revised if outdated or incomplete, or extended if new information is relevant but missing.

To test this, the authors simulate ten timestamps from 2016 to 2025, each fixed to December 31, and use Tavily's end_date parameter to restrict search results to information published no later than the target timestamp. The Single-Step Task compares one incremental update against a report independently generated for the same timestamp; the Long-Chain Task follows a sequence of consecutive updates. Model configurations are Qwen3.5-397B-A17B as Researcher with Qwen3.5-122B-A10B for other roles (OC), and GPT-5 as Researcher with Qwen3.5-397B-A17B for other roles (PC). Crawl4AI handles URL fetching and webpage parsing. DeepResearch Bench's RACE Overall score is computed with GPT-5.5 as judge following the benchmark's updated protocol after Gemini-2.5-Pro was deprecated on June 17, 2026; DeepConsult's average quality score uses GPT-4.1.

Why This Matters

Impact on research. The paper reframes research report generation as maintenance of an evolving state rather than repeated one-off generation, and it contributes a reusable temporal evaluation protocol. Its timestamp-controlled results show that OEDR scores shift meaningfully depending on what information is available when the research is run, which is a methodological point for how future OEDR systems are compared.

Real-world applications (as identified by the authors):

• Technology and industry intelligence, where reports must absorb newly released models and emerging techniques.
• Financial and economic research, where updates follow new statistics and indicators.
• Scientific literature monitoring, where accumulated findings must be preserved while new work is incorporated.
• Business and consulting research, the domain covered by the DeepConsult benchmark.

Industry relevance. The cost results are the main practical draw: on DeepResearch Bench, search calls fall from 45.19 to 18.62 (SST) and 17.85 (LCT) under OC, and token consumption falls by roughly a third in LCT. For organizations that maintain standing reports on a schedule, reusing a persistent evidence pool and editing only changed sections is cheaper than rerunning full research, and continuity metrics such as 0.68 content-level ROUGE-L F1 suggest the output stays recognizable to readers rather than being silently rewritten.

Future Directions

• Simulating change in existing webpages. The authors note that Tavily's temporal search restriction controls which pages are accessible at each timestamp but cannot simulate how the content of an existing webpage evolves over time.

• Building dedicated Incremental-OEDR benchmarks. Current evaluation uses existing OEDR benchmarks that were not designed for incremental report maintenance; the authors call for benchmarks with temporally evolving research scenarios and explicitly updated sources.

• Understanding the update-interval trade-off. Quality and continuity largely plateau around a 1-year interval while cost keeps decreasing with diminishing returns, leaving open how interval should be chosen for different report types.

• Extending long-chain maintenance. LCT was more cost-efficient than SST in the reported token and search figures, suggesting that the value of a persistent evidence pool grows with update frequency and horizon, which the paper does not yet characterize beyond the reported transitions.

Target Audience

Researchers and engineers working on LLM agents, deep research systems, and retrieval-augmented report generation who care about long-horizon maintenance rather than single-shot generation. It is also relevant to evaluation researchers interested in temporally controlled benchmarking, and to product teams building standing research or monitoring pipelines where reports must be refreshed on a schedule while preserving prior structure and content.

Authors’ abstract

Existing Open-Ended Deep Research (OEDR) systems primarily generate reports from scratch, making them inefficient for scenarios where research reports need to be continuously maintained as new information emerges. We introduce \textbf{Incremental Open-Ended Deep Research (Incremental-OEDR)}, a research setting that treats a report as an evolving research state and incrementally updates it by preserving valid knowledge, revising outdated or incomplete content, and incorporating newly available information. To support this setting, we propose \textbf{Structured Harness}, which represents reports as structured collections of outlines, sections, and supporting evidence, and provides structured retrieval, a persistent structured evidence pool, and structured generation for selective report updating and evidence reuse. We further establish a temporal evaluation framework spanning ten years, with \emph{Single-Step Task} and \emph{Long-Chain Task} to evaluate incremental updates over both individual transitions and long-term update chains. Extensive Experiments on DeepResearch Bench and DeepConsult under both the Open-source Configuration (OC) and Proprietary Configuration (PC) show that Incremental-OEDR maintains competitive report quality while substantially improving report continuity and reducing research costs. As shown in Figure~\ref{fig:profile}, it achieves up to 0.51 higher content-level ROUGE-L F1, 0.63 higher outline-level EM F1, 33\% lower token consumption, and 61\% fewer search calls than OEDR on DeepResearch Bench. For more details, please refer to our project page: https://ioedr-project.github.io/.