A Benchmark for Deep Information Synthesis

AI summary

Overview

Research area: AI agent evaluation, LLM benchmarking, and information-seeking/synthesis capabilities of tool-using language models.

Technical level: Intermediate — readers should be familiar with LLM agents, tool use, and standard NLP metrics (F1, exact match, LLM-as-judge), but the paper's core argument is accessible.

One-sentence scope: The paper introduces DeepSynth, a 120-task benchmark spanning 67 countries that tests whether LLM-based agents can gather, combine, and reason over multi-source information to produce verifiable insights, and shows that current state-of-the-art models fail badly on it.

What This Paper Is About

Existing benchmarks for LLM agents mostly test shallow fact retrieval, single-source lookups (often Wikipedia), or artificial multi-hop questions — not the slow, messy, multi-source synthesis that real analysts do. DeepSynth is built to close that gap: tasks that require browsing multiple websites, reading both narrative text and tables, performing arithmetic and comparison, and producing a structured answer (a JSON object with multiple key-value pairs). The paper both releases the benchmark and demonstrates empirically that today's best LLMs and "deep research" agents score near zero on it.

Key Contributions

1. Release of the DeepSynth benchmark: 120 expert-designed, real-world, time-consuming information synthesis tasks, covering 7 domains and 67 countries, with gold-standard answers, intermediate reasoning chains, and supporting evidence URLs.

2. A four-stage, reverse-engineered annotation pipeline: Rather than starting with a fact and hiding it, annotators first identified official data sources, generated hypotheses, validated them through manual analysis, and only then wrote the tasks — ensuring answers are non-retrievable by verbatim lookup and robust against memorisation.

3. A comprehensive empirical evaluation: 11 models and agent frameworks (standalone LLMs plus o3-deep-research, smolagents, and OWL) are benchmarked, with the best F1 score being only 8.97 and the best LLM-judge score 17.5, with almost no task solved perfectly under exact match.

4. Diagnostic analysis of failure modes: The paper quantifies navigation vs. synthesis errors, error propagation through reasoning chains, performance decay as intermediate steps increase, and strong geographic bias (all models score 0.0 on Africa-related tasks).

Main Findings

• Ceiling performance is extremely low: The best standalone model, Gemini-2.5-Pro, achieves an F1 of 6.25; the best agent, o3-deep-research, reaches F1 8.97 and an LLM-judge score of 17.5. Only three of 120 tasks are fully solved by any specialised deep research agent.

• Reasoning ability is not the bottleneck — retrieval is: The gap between dedicated reasoning models (Gemini-2.5-Pro, GPT-5.1, DeepSeek-R1) and general-purpose models (GPT-4.1) on F1 is small, suggesting the limiting factor is access to the right information, not inference quality.

• Exact match is near-zero across the board: Almost all models score 0.0 on the strict exact-match metric; GPT-5.2-Pro reaches only 6.25. This supports the claim that tasks are robust against memorisation from pre-training data.

• Tool augmentation helps, but modestly: o3-deep-research improves over base o3 by 5.68 F1 points; smolagents and OWL improve GPT-4.1 by 0.29 and 1.95 F1 points respectively. Removing the search toolkit from OWL causes the largest single ablation drop (1.81 F1), confirming search is critical.

• Planning is a major weakness: When given the gold intermediate steps (without the answers), GPT-4.1 jumps from 3.46 to 9.36 F1, and smolagents from 3.75 to 10.50 F1, with exact match rising to 5–10%. Models appear to lack the ability to plan their own decomposition.

• Performance collapses as task complexity grows: All models degrade sharply as the number of intermediate steps increases; agentic frameworks hold up slightly better in the 11–15 step range but not on simpler tasks.

• Direct errors are split between navigation and synthesis: In a 32-task manual audit of OWL, 15 failures came from navigation errors (wrong page, document, or section) and 16 from synthesis errors (correct information, wrong conclusion). Four were "no answer" and four were technical/tool failures.

• Errors cascade irrecoverably: In a per-step evaluation on 40 tasks, retrieval steps achieve only 2–12% F1, computation steps collapse near zero, and when a step fails, the next step fails 91–100% of the time, with recovery below 10%.

• High output variance prevents reliable self-consistency: Under Best@5, smolagents (GPT-4.1) reaches 25% LLM-judge accuracy, but majority-voting self-consistency at N=5 yields only 5% — correct answers rarely form the majority prediction.

• Strong geographic bias: All models score 0.0 F1 on Africa-related tasks. Europe (38.3% of tasks) and Asia (29.2%) see the strongest performance; o3-deep-research is the most consistent cross-regionally.

• Operation-specific difficulty: o3 reaches the highest F1 on anomaly detection (26.51%), and Gemini-2.5-Pro and o3 perform relatively well on trend detection and ranking, while no model shows measurable performance on filtering tasks.

• Compute cost varies enormously: o3-deep-research costs roughly $184.61 per full benchmark run and averages 645 seconds per instance; OWL averages 1025.5 seconds, versus 5–83 seconds for standalone LLMs.

Methodology in Plain English

The researchers deliberately inverted the usual benchmark-construction method. Instead of finding a fact and then writing a question that hides it, they had 16 domain experts first propose 223 trustworthy, official data sources across seven domains (socio-economic, finance, environment, science, education, transportation, political/socio-political). From those, experts wrote hypotheses — realistic claims that could be checked against the data, such as whether air quality correlates with pneumonia deaths across UK regions.

Each hypothesis was then manually investigated. Unverifiable hypotheses were refined or discarded, leaving 155 candidate sources, then 130 after validation. Only at that point were actual task questions, intermediate reasoning steps, supporting evidence, and answers written — and each question then had to be independently answered by a second annotator, with only fully matching answers retained. This cross-checking reduced the set to 120 tasks.

Tasks were designed to meet five criteria: they must require synthesising information across multiple sources; they must be inspired by real-world scenarios relevant to analysts, policymakers, or travel agents; they must have closed-form, automatically verifiable answers stable over time; they must be diverse across geographies, time periods, and reasoning modes; and they must resist memorisation.

Evaluation used two metric families: strict metrics (exact match, plus precision/recall/F1 over key-value pairs) and a softer LLM-as-judge score that tolerates small numeric differences (1–5.5%) and semantic equivalences. All models were given identical prompts. Three agent frameworks were tested with their real tool stacks: o3-deep-research (web search plus code interpreter), smolagents (a minimal ReAct-style loop that expresses actions as executable code), and OWL (a planner-executor role-playing architecture). Finally, the authors ran targeted ablations — removing individual toolkits, injecting gold intermediate steps, and scoring each reasoning step separately against gold answers — to isolate where failures originate.

Why This Matters

Impact on research: DeepSynth shifts the agentic benchmarking conversation from "can a model find a fact?" to "can a model produce a defensible analytical insight?" The paper's diagnostic approach — step-level error propagation, regional breakdowns, and tool ablations — provides a template for future benchmark design and a concrete diagnostic target for agent developers. It also provides evidence that reasoning-model improvements alone will not close the gap without better browsing and extraction tooling.

Real-world applications:

• Policy and economic analysis: Tasks that require compiling statistics across national statistical offices to compare trends, rankings, or anomalies between countries.

• Business and market intelligence: Analysts who need to aggregate reports, tables, and press releases into structured comparative summaries for decision-making.

• Journalism and fact-checking: Structured claims with citations across multiple official sources, where hallucination is the central failure mode.

• Scientific and medical evidence synthesis: Aggregating findings across studies or health datasets (e.g., correlating environmental variables with health outcomes) into comparable, verifiable outputs.

• Travel, logistics, and regional planning: The paper's motivating example is a travel agency query about post-COVID visitor recovery — a multi-source, multi-step question typical of real operational research.

Industry relevance: DeepSynth is directly relevant to teams building "deep research" products, retrieval-augmented agents, and enterprise search tools. The finding that cost per run can reach ~$185 and still produce an F1 below 10 is a strong economic signal that current agent architectures are far from production-reliable for synthesis-heavy work. The two most common failure modes — navigation and synthesis — are both actionable engineering targets.

Future Directions

1. Improving web navigation and extraction: Since navigation errors caused nearly half of observed failures and GPT-5.2's intermediate steps collapsed when faced with JavaScript-rendered tables, binary spreadsheets, and interactive bulk downloads, better browsers, document parsers, and API-aware retrieval are the highest-leverage next step.

2. Training models to plan decompositions: The large gains from supplying gold intermediate steps show that explicit planning is a major missing capability; distilling or training planning behaviour for multi-source tasks is a clear open problem.

3. Correcting geographic and data-source bias: All models scoring 0.0 on Africa-related tasks points to unequal training-data coverage. Future work should investigate whether targeted retrieval, multilingual sources, or region-aware training can equalise performance across the 67 countries.

4. Reducing output variance and hallucination: The mismatch between Best@5 (25%) and self-consistency (5%) accuracy shows agents succeed occasionally but not reliably. Research into answer verification, uncertainty estimation, and consistency mechanisms is needed before these systems can be trusted.

5. Extending to open-form answers: The paper notes that its tasks are artificially constrained to closed-form answers for verifiability, while real synthesis often produces argued, cited prose. A complementary benchmark for grounded open-form synthesis is a natural follow-up.

Target Audience

This paper is most valuable for: AI agent researchers and benchmark designers; engineers building retrieval-augmented or deep-research agent systems; evaluation and trust-and-safety teams in industry who need to know where current tool-using LLMs break down; and NLP researchers studying planning, error propagation, and geographic bias in language models. Domain analysts (policy, economics, journalism) will also find the task framing useful as a description of what realistic synthesis work looks like, and graduate students entering the agent-evaluation subfield will benefit from the benchmark's criteria and diagnostic methodology.

Authors’ abstract

Large language model (LLM)-based agents are increasingly used to solve complex tasks involving tool use, such as web browsing, code execution, and data analysis. However, current evaluation benchmarks do not adequately assess their ability to solve real-world tasks that require synthesizing information from multiple sources and inferring insights beyond simple fact retrieval. To address this, we introduce DEEPSYNTH, a novel benchmark designed to evaluate agents on realistic, time-consuming problems that combine information gathering, synthesis, and structured reasoning to produce insights. DEEPSYNTH contains 120 tasks collected across 7 domains and data sources covering 67 countries. DEEPSYNTH is constructed using a multi-stage data collection pipeline that requires annotators to collect official data sources, create hypotheses, perform manual analysis, and design tasks with verifiable answers. When evaluated on DEEPSYNTH, 11 state-of-the-art LLMs and deep research agents achieve a maximum F1 score of 8.97 and 17.5 on the LLM-judge metric, underscoring the difficulty of the benchmark. Our analysis reveals that current agents struggle with hallucinations and reasoning over large information spaces, highlighting DEEPSYNTH as a crucial benchmark for guiding future research.