Benchmarking and Error Analysis of Deep Search and Deep Research LLM Agents

AI summary

Overview

Research area: Evaluation and diagnostics of LLM-based autonomous agents — specifically Deep Search agents (iterative web exploration to find specific facts) and Deep Research agents (long-horizon synthesis of multi-source evidence into analytical reports).

Technical level: Intermediate to Advanced. The paper assumes familiarity with tool-using LLM agents, agent frameworks, and benchmark evaluation, but its core idea — diagnosing failures from execution traces rather than final answers — is stated plainly.

Scope: A process-level evaluation framework and error taxonomy for diagnosing where and how Deep Search and Deep Research agents fail across intermediate execution steps, validated with trace-level experiments on four Deep Search benchmarks and one Deep Research benchmark.

What This Paper Is About

Most benchmarks judge agentic systems only by their final answer or report, which hides why an agent succeeded or failed — a correct answer may come from a broken process, and a wrong answer reveals nothing about the cause. The authors argue that long-horizon agents are especially vulnerable to cascading errors during intermediate steps (tool calls, searching, fetching, reading, stopping decisions, synthesis), and that these are invisible to end-to-end metrics. Their goal is to build a reusable diagnostic framework that classifies failures into a two-level taxonomy (System-Level and Task-Level), then empirically measures how those error types appear and propagate.

Key Contributions

1. A process-oriented evaluation framework that captures full agent trajectories (every tool call, search, fetch, and reasoning step), enabling reproducible diagnosis beyond end-to-end accuracy.
2. A structured two-level error taxonomy that separates System-Level failures (shared by both paradigms: tool-call parsing failures, incorrect tool names, argument/schema errors, tool–function mismatch, irrelevant queries, domain tool misalignment, syntactical query failure, unsupported content modality, non-existent pages, anti-scraping blockers) from Task-Level failures (paradigm-specific: five categories for Deep Search, three for Deep Research).
3. A hybrid diagnostic pipeline combining rule-based regex/structured parsing (to isolate deterministic system faults) with a human-guided LLM-as-a-Judge protocol (to classify semantic, cognitive failures) — with the judge prompts documented for reuse.
4. A trace-based empirical study quantifying how error types distribute and compound across four Deep Search datasets and two Deep Research frameworks, culminating in actionable design guidance for training, toolset design, and agent architecture.

Main Findings

• System-level failures are benchmark-dependent, not just length-dependent. XBench-DS has the highest concentration of system-level failure steps (No Error Rate 27.90 across 43.76 average steps per task), while BrowseComp (75.10) and BrowseComp-zh (78.78) remain comparatively stable — despite BrowseComp-zh having the longest trajectories (61.7 average steps, 6170 total steps).

• Low-level syntax errors have become rare; semantic misalignment dominates. Tool Call Parsing Failure counts are small across all four datasets (BrowseComp 22, BrowseComp-zh 6, GAIA-val-text-only 3, XBench-DS 15), whereas Tool-Function Mismatch (434, 286, 1227, 650) and Domain Tool Misalignment (72, 426, 1163, 369) dominate the error composition. The authors conclude modern models handle tool-invocation mechanics reliably, but not which tool to use for which intent.

• Each benchmark has a distinct bottleneck profile. BrowseComp is constrained by tool–function alignment plus web accessibility (Anti-Scraping 392); BrowseComp-zh shows strong domain–tool alignment pressure (Domain Tool Misalignment 426) alongside accessibility failures (Anti-Scraping 437); XBench-DS is dominated by tool-orchestration failures, where functional mismatches (1227) outweigh parsing errors.

• System-level disruption does not prevent success. For MiroThinker-30B, GAIA-val-text-only has a No Error Rate of only 27.90 yet reaches 63.11 accuracy, showing agents compensate for imperfect execution through iterative adjustment and recovery rather than flawless tool use.

• Process reliability correlates with task accuracy. Across Deep Search agents, higher No Error Rate tracks higher final accuracy (e.g., Miro-30B on XBench-DS: 72.00 No Error Rate and 73.00 accuracy; GPT-5-mini on XBench-DS: 19.00 No Error Rate and 41.00 accuracy), which the authors read as validating the diagnostic taxonomy.

• Model families fail for opposite cognitive reasons. GPT-5-mini is dominated by Premature Stop (75 on XBench-DS, 41 on GAIA, 38 on BrowseComp-zh) and Surface-Level Skimming (20 on BrowseComp), indicating reluctance to sustain multi-hop exploration. MiroThinker-30B instead shows execution endurance but is bottlenecked by Ineffective Query (47 on BrowseComp, 41 for GPT-5-mini on BrowseComp too) and Observation Neglect (12 on GAIA, 18 on XBench-DS).

• Deep Research frameworks fail differently from each other. Open Deep Research reaches a No Error Rate of 44 and is chiefly damaged by Goal Drifting (51 occurrences), losing user constraints and formatting instructions as context grows. Step Deep Research has a No Error Rate of only 13 and is dominated by Synthesis Hallucination (80), with 60 cases of hallucination alone and 17 compound cases of Contradiction Neglect plus Synthesis Hallucination — evidence that aggressive synthesis triggers cascading cognitive failures when retrieval is already flawed. Report quality scores reported are RACE 0.536 and FACT 0.468 for Open-DR versus RACE 0.4802 and FACT 0.4143 for Step-DR (the paper as provided does not define these two metrics).

Methodology in Plain English

The authors deliberately split errors into two buckets so that a broken API call is never mistaken for a reasoning failure.

System-Level errors are caught by a two-stage filter. First, regular expressions and structured parsing pull each tool invocation and the reasoning attached to it out of the raw logs. Second, an LLM evaluates whether the agent's stated intent matches the tool it actually used — catching mismatches like passing a raw URL into a text search engine or querying a Wikipedia tool with an arXiv ID. Errors are then slotted into three pipeline stages: Agent Tool Call, Web Search, and Web Fetch.

Task-Level errors are harder to pin down with rules, so the authors first read sampled traces by hand to inductively build a behavioural taxonomy, then scaled that rubric using LLM-as-a-Judge with human guidance. The rubric is paradigm-specific because the cognitive objectives differ: Deep Search asks "did it find and use the right evidence and stop at the right time?", while Deep Research asks "did it reconcile conflicting sources, ground its claims, and stay on the original brief?"

Experimental setup. The design deliberately constrains the environment rather than chasing leaderboard scores. Deep Search uses the Miroflow agent framework restricted to just two tools (search and fetch) with GPT-5-mini and Mirothinker-30b as backbones, on tasks sampled from BrowseComp, BrowseComp-zh, GAIA-val-text-only, and XBench-DS. Deep Research uses open-deep-research (with GPT-5) and step-deep-research (with Step-3.5-Flash) on Deep Research Bench. GPT-5 serves as the judge for both error levels. The benchmarks are used only as sources of realistic long-horizon trajectories, not for conventional performance ranking.

Why This Matters

Impact on research. The paper reframes agent evaluation from "did it get the answer right?" to "where in the trajectory did it go wrong?", which changes what a benchmark score means. It also gives the field a shared vocabulary — specific, nameable failure modes — that makes results comparable across papers, models, and frameworks. The observation that heavy system-level error does not preclude task success challenges the assumption that end-to-end accuracy is a proxy for reliability.

Real-world applications:

• Enterprise research assistants producing due-diligence, competitive-intelligence, or market reports, where Goal Drifting and Synthesis Hallucination are exactly the failure modes that make output legally or commercially unsafe.
• Agent observability and monitoring tooling — the taxonomy functions as a runtime checklist for flagging a trajectory before it fails, not after.
• Browser-embedded search agents used by consumers, where Premature Stopping and Surface-Level Skimming produce confidently wrong answers from snippets.
• Tool and API platform design for agent developers: better schema constraints, semantic error messages, and built-in recovery affordances reduce the dominant mismatch errors.

Industry relevance. The findings point directly at where engineering effort should go. Purely syntactic reliability is largely solved; the remaining friction is semantic tool selection and adversarial web environments (anti-scraping blockers appear 392 times on BrowseComp and 437 on BrowseComp-zh). The recommendation to decouple tool-execution supervision from reasoning supervision maps naturally onto staged training and RLAIF reward design, and the argument for modular planning/verification/repair stages is a direct critique of monolithic agent loops.

Future Directions

• Extending beyond the current environments and tool ecosystems. The authors explicitly name broader environments and richer toolsets as next steps; the present study deliberately restricted Deep Search to two tools, so behaviour under realistic, crowded toolkits is untested here.
• Automated online error recovery. The paper's diagnostics are applied after the fact. Turning the taxonomy into runtime self-diagnosis that lets an agent detect and repair its own trajectory mid-execution is named as future work and remains open.
• Trace-aware training objectives. The authors propose rewarding intermediate reasoning quality, evidence coverage, and post-failure recovery, and using taxonomy-derived reward signals for iterative fine-tuning and RLAIF — but the paper does not report training experiments, so the payoff is untested.
• Adaptive stopping and coverage checks for Deep Search, staged synthesis pipelines for Deep Research. These architectural prescriptions follow from the error distributions but are proposed as design guidance rather than validated as interventions.

Target Audience

Agent researchers and engineers building or evaluating tool-using LLM systems will get the most from this paper — particularly those working on web navigation, retrieval-augmented agents, or long-form report generation. It is also valuable for practitioners who need to debug production agent pipelines and want a structured vocabulary for failure modes, and for benchmark designers interested in process-level rather than outcome-level metrics. Readers seeking a leaderboard comparison of models will not find one here; the paper is explicitly a diagnostic study, and some dataset configuration details referenced to Appendix B are not included in the provided content.

Authors’ abstract

As Large Language Models (LLMs) evolve into autonomous Deep Search and Deep Research Agents capable of executing long-horizon, multi-step tasks, evaluating their reliability becomes increasingly critical. However, existing evaluation paradigms predominantly focus on final task outcomes, leaving a significant gap in understanding the intermediate execution traces, where agents are highly susceptible to various errors. To address this critical gap, we introduce a comprehensive methodology and analytical framework designed specifically to diagnose errors in intermediate workflows. Building upon this framework, we establish a structured taxonomy that categorizes these vulnerabilities into two primary dimensions: System-Level Errors and Task-Level Errors. Through a systematic examination of step-by-step execution traces and agent-environment interactions, we reveal how these underlying errors affect overall performance. Finally, through empirical experiments and systematic analysis of the identified error types, we provide actionable design guidelines, aiming to enhance the robustness, reasoning capabilities, and reliability of Deep Search and Deep Research agents.