LiveResearchBench: A Live Benchmark for User-Centric Deep Research in the Wild

AI summary

Overview

• Research area: AI agent evaluation — specifically benchmarks and judge-based evaluation suites for "deep research" systems that search the live web and write long, citation-grounded reports.
• Technical level: Intermediate. The paper is readable without deep technical background, but benefits from familiarity with LLM agents, tool use, and LLM-as-a-judge evaluation.
• One-sentence scope: The paper introduces a 100-task, expert-curated live benchmark (LiveResearchBench) and a six-dimension evaluation suite (DeepEval) to measure how well single-agent and multi-agent deep research systems search, synthesize, cite, and analyze.

What This Paper Is About

Deep research agents are expected to search hundreds of live web pages and produce comprehensive, well-cited reports, but existing benchmarks are static, narrow in domain, or pose ambiguous questions that make fair comparison impossible. The authors diagnose these gaps through a user survey and distill four design principles — user-centric, dynamic, unambiguous, and multi-faceted/search-intensive — then build a benchmark and evaluation suite that satisfy them. The goal is to produce a rigorous, contamination-resistant way to measure whether systems genuinely research or merely assemble plausible-sounding text.

Key Contributions

1. Four task design principles for deep research benchmarks — user-centric, dynamic (time-varying), unambiguous, and multi-faceted/search-intensive — derived from a user survey covering enterprise professionals, academics, and everyday users.
2. LiveResearchBench — 100 expert-curated tasks with detailed per-query checklists, built with over 1,500 hours of human labor, spanning seven domains (Science & Technology, Economy & Business, Health & Wellbeing, Law & Governance, Society & Culture, Education & Knowledge, Media & Entertainment) and ten task categories.
3. DeepEval — an evaluation suite covering six complementary dimensions: Presentation & Organization, Factual & Logical Consistency, Coverage & Comprehensiveness, Analysis Depth, Citation Association, and Citation Accuracy, each paired with a protocol chosen empirically for human alignment.
4. A comprehensive evaluation of 17 frontier systems across single-agent web search, single-agent deep research, and multi-agent architectures, revealing systematic failure modes and structural bottlenecks.

Main Findings

• Multi-agent families lead on average, but not uniformly. Averaged by family, multi-agent deep research systems score highest overall (69.5), ahead of single-agent web search (62.8). Open Deep Research achieves the best individual average (73.6), followed by GPT-5 (72.7).
• Single web agents are strongest at factual and logical consistency. They average 69.7 on consistency, benefiting from a single persistent memory stream that avoids logical drift across agent handoffs. Gemini 2.5 Pro scores highest of any system at 76.5.
• Multi-agent systems dominate citation association and presentation. They average 61.9 on citation association (Deerflow+ at 77.0, Open Deep Research at 76.9) and 77.7 on presentation. However, presentation quality correlates only weakly with citation association or consistency — polished reports can still contain contradictory claims.
• Single-agent deep research systems are the worst at citation association. OpenAI o3 Deep Research scores 25.6 and o4-mini Deep Research 27.2, largely because key statements go uncited, URLs are mismatched, or links are fictional.
• Coverage collapses as retrieval scales. Grok-4 Heavy Deep Research achieves the highest coverage score (89.3), while Deerflow+ (61.6) and Open Deep Research (65.3) lag despite strong citation handling. Multi-agent pipelines can push retrieval past 3,000 pages, exceeding context windows; without memory labeling, deduplication, and lossless compression, breadth is lost.
• Most systems are "deep searchers, not deep researchers." On analysis depth measured as win rate over Open Deep Research, only Deerflow+ and Gemini Deep Research exceed the baseline. Systems producing very long reports, such as o3 Deep Research, often fail to synthesize across sources.
• Longer reports are not consistently better. Report length varies far more by model than by task. Some length inflation comes from formatting artifacts — provider-specific redirect URLs or the same link cited both inline and in the bibliography — rather than substantive content.
• Citation errors persist even with web access. A rubric-tree analysis of top performers (GPT-5, Grok-4 Deep Research, Open Deep Research) on the two most search-intensive task categories found that most citation errors in wide information search come from unsupported claims — statements the cited link does not actually verify — rather than dead or irrelevant URLs.
• No system wins on all dimensions. Coherence, verifiability, breadth, and depth trade off against each other under context limits; scaling retrieval mostly increases report length rather than reliable insight.
• Holistic LLM judging is unreliable. Even with state-of-the-art judges (Gemini 2.5 Pro, GPT-5), single-score 0–10 ratings fell below 60% agreement with human judgments and showed swings exceeding 50 points on analysis depth, because judges commit to an overall impression instead of auditing sentence by sentence.

Methodology in Plain English

The authors first surveyed real users — enterprise professionals, researchers, students, and general users — asking what they would actually ask a deep research agent, then used the responses to set the task and domain distribution. Domain experts drafted research questions to fit that distribution. To remove ambiguity, two frontier deep research models were asked to generate possible clarification questions for each draft; experts reviewed both sets, merged them with domain knowledge, and finalized queries with explicit scope, audience, and output format. GPT-5 then decomposed each query into a checklist of unit-test-style questions that a report must satisfy. A five-stage human verification pipeline — independent annotation, two quality-control rounds (tolerance: no more than 5% deviation from true data quality), and a final conflict-resolution and cross-check pass — validated every query and checklist item.

For evaluation, the team ran human alignment studies to pick the best protocol per metric rather than defaulting to a single approach. Checklist-based scoring is used for presentation and coverage; pointwise (additive) error-counting for factual/logical consistency and citation association; pairwise comparison with position-swap averaging for analysis depth; and a rubric tree with a web-enabled agentic judge for citation accuracy. Every judge score is produced by an ensemble of Gemini 2.5 Pro and GPT-5, averaged, which improved agreement with human experts substantially.

Why This Matters

Deep research is one of the most commercially active frontiers for LLM agents, yet the field lacked a benchmark that resists data contamination, avoids ambiguous queries, and measures long-form report quality across multiple dimensions rather than a single holistic score. This paper supplies that infrastructure and, more importantly, quantifies where current systems actually break: not in fluency, but in citation discipline, memory management under scale, and analytical synthesis. The finding that multi-agent architectures win on average but not on every dimension gives builders concrete architectural guidance instead of a simplistic "more agents is better" message.

Real-world applications:

• Enterprise market and competitive analysis — generating sourced reports on market size, growth rates, and competitor positioning, where uncited claims are a compliance risk.
• Policy and regulatory evaluation — synthesizing legislation, rulings, and commentary with traceable citations for legal and governance teams.
• Healthcare and scientific literature review — producing evidence-grounded summaries where unsupported claims are materially dangerous.
• Consumer and educational research — answering everyday multi-faceted questions with correct audience calibration and formatting, such as undergraduate-level academic reports.

Industry relevance centers on procurement and model selection: the benchmark and DeepEval give teams a reproducible way to compare proprietary systems (OpenAI, Gemini, Perplexity, Grok, Manus) against open-source stacks (Open Deep Research, DeerFlow), and the results suggest that citation reliability, not raw report length or surface polish, is the current differentiator.

Future Directions

• Long-horizon memory architectures with update operations, so agents can track what has already been retrieved and processed across thousands of pages without losing evidence.
• Hierarchical compression that is both importance- and preference-aware — merging redundant content while preserving unique evidence when information overlaps, and assigning importance levels to fit critical details into the context window when all content is relevant.
• Explicit synthesis and argumentation modules, since retrieval scaling alone increases length rather than insight; the gap between coverage and depth needs a dedicated architectural answer.
• Stronger citation verification and generation pipelines, given that unsupported claims, not dead links, are the dominant citation error — and that hallucinated citations persist even with live web access.

An open question the work raises is whether any single architecture can simultaneously achieve coherence, verifiability, breadth, and depth, or whether these are inherently competing objectives under fixed context budgets.

Target Audience

AI agent researchers and benchmark designers will find the task-design principles and the protocol-selection methodology most valuable, particularly the evidence that holistic LLM judging is unreliable and that per-metric protocol choice plus judge ensembling is needed. Practitioners evaluating or building deep research products — enterprise AI teams, RAG and agent infrastructure engineers, and technical product managers — benefit from the comparative system rankings and the concrete failure-mode taxonomy. Evaluation researchers working on long-form or open-ended generation will find the six-dimension DeepEval design reusable beyond web research, and the citation-accuracy rubric tree directly applicable to any citation-grounding task.

Authors’ abstract

Deep research -- producing comprehensive, citation-grounded reports by searching and synthesizing information from hundreds of live web sources -- marks an important frontier for agentic systems. To rigorously evaluate this ability, four principles are essential: tasks should be (1) user-centric, reflecting realistic information needs, (2) dynamic, requiring up-to-date information beyond parametric knowledge, (3) unambiguous, ensuring consistent interpretation across users, and (4) multi-faceted and search-intensive, requiring search over numerous web sources and in-depth analysis. Existing benchmarks fall short of these principles, often focusing on narrow domains or posing ambiguous questions that hinder fair comparison. Guided by these principles, we introduce LiveResearchBench, a benchmark of 100 expert-curated tasks spanning daily life, enterprise, and academia, each requiring extensive, dynamic, real-time web search and synthesis. Built with over 1,500 hours of human labor, LiveResearchBench provides a rigorous basis for systematic evaluation. To evaluate citation-grounded long-form reports, we introduce DeepEval, a comprehensive suite covering both content- and report-level quality, including coverage, presentation, citation accuracy and association, consistency and depth of analysis. DeepEval integrates four complementary evaluation protocols, each designed to ensure stable assessment and high agreement with human judgments. Using LiveResearchBench and DeepEval, we conduct a comprehensive evaluation of 17 frontier deep research systems, including single-agent web search, single-agent deep research, and multi-agent systems. Our analysis reveals current strengths, recurring failure modes, and key system components needed to advance reliable, insightful deep research. Our code is available at: https://github.com/SalesforceAIResearch/LiveResearchBench.