TRACE: Trajectory-Aware Comprehensive Evaluation for Deep Research Agents

AI summary

Overview

Research area: Evaluation methodology for LLM-powered Deep Research Agents, positioned at the intersection of natural language processing, agent benchmarking, and reinforcement-learning-inspired process analysis. Published at the ACM Web Conference 2026 (WWW '26), Dubai, April 13–17, 2026 (DOI: 10.1145/3774904.3792738).

Technical level: Advanced. The paper is written for readers comfortable with formal notation (utility functions, geometric means, policy entropy, constrained optimization) and with agent-training terminology such as SFT, RL post-training, and entropy collapse.

Scope in one sentence: The paper proposes a trajectory-level evaluation framework, TRACE, that scores not just whether a Deep Research Agent answered correctly but how efficiently, how soundly, and how robustly it got there, plus a purpose-built benchmark (DeepResearch-Bench) that makes those scores measurable.

What This Paper Is About

Current evaluations of Deep Research Agents lean on a single end-result number such as Pass@1, which rewards a correct final answer no matter how inefficient, circuitous, or hallucination-dependent the reasoning path was. The authors call this the "high-score illusion," and argue that static benchmarks also cannot measure an agent's robustness against misleading web content or its latent ability to succeed with minimal help. TRACE is a framework designed to score the whole problem-solving trajectory and to expose those hidden attributes.

Key Contributions

1. A Hierarchical Trajectory Utility Function. A single holistic score U(ℋ) that combines final-answer correctness with Process Efficiency (ℰ) and Cognitive Quality (𝒞) via a weighted geometric mean, so that a deficiency in any one dimension disproportionately penalizes the overall score.
2. Two new diagnostic protocols for latent attributes. A Scaffolded Capability Assessment that computes the Minimum Hint Rate (λ_min) — the minimum fraction of an oracle solution an agent needs before its expected success rate reaches a target threshold θ_succ (given as e.g. 0.9) — and policy diagnostics including Entropy Adaptability (ℰ_A) and the Trajectory Reproducibility Score (TRS).
3. DeepResearch-Bench, a benchmark with controllable complexity. 650 tasks in total, split into TRACE-Core (500 tasks, average complexity C(q) = 3.5, 20% traps), TRACE-Robustness (100 tasks, C(q) = 4.2, 100% traps), and TRACE-Scaffolding (50 tasks, C(q) = 5.8, 40% traps). It embeds "information traps" and provides an oracle trajectory for every task — features the paper states are absent from BrowseComp-en and GAIA.
4. An empirical re-ranking of state-of-the-art agents. A multi-dimensional evaluation showing that Pass@1 rankings diverge from utility rankings, plus a controlled study applying four training paradigms to a single Qwen-30B-Base model.

Main Findings

• Pass@1 and trajectory utility disagree. On TRACE-Core, DeepSeek-V3.1-671B records the highest Pass@1 among open-source SOTA agents at 65.8%, yet its utility U(ℋ) = 0.65 is the lowest among top-tier competitors, which the authors attribute to poor Process Efficiency (ℰ = 0.68). AgentFounder-30B, with a lower Pass@1 of 60.1%, earns the highest open-source utility at 0.81.
• Closed-source systems are not uniformly ranked by accuracy. OpenAI Deep Research scores higher Pass@1 than Gemini-2.5-pro-DR on TRACE-Core (78.2 vs. 75.4), but Gemini-2.5-pro-DR achieves the higher utility (0.88 vs. 0.85) and higher efficiency (0.90 vs. 0.82). The same inversion appears on GAIA (text-only), where Gemini-2.5-pro-DR reaches 72.8 Pass@1 and OpenAI Deep Research 70.5, with utilities of 0.89 and 0.82 respectively.
• Metric-level "fingerprinting" separates agents with similar accuracy. AgentFounder-30B leads on Efficiency (0.88), Cognitive Quality (0.91), and Evidence Grounding (𝒢_E = 0.95). WebSailor-V2-30B shows strong Reasoning Robustness (ℛ_R = 0.84) but lower grounding (0.92). ReSum-GRPO's efficiency (0.86) is credited to its context summarization architecture. The paper states these distinctions are invisible to Pass@1-only evaluation.
• Latent capability, not size, predicts how much help an agent needs. AgentFounder-30B requires a Minimum Hint Rate of 0.22, versus 0.35 for the much larger DeepSeek-V3.1-671B and 0.41 for GLM-4.5-355B. The vanilla ReAct (Qwen-30B) baseline needs the most guidance at 0.51, and WebSailor-72B (v1) at 0.45.
• Generalization benchmarks preserve the pattern. On BrowseComp-en, WebSailor-V2-30B scores 35.3 Pass@1 but utility 0.71, while AgentFounder-30B scores 31.5 Pass@1 with a higher utility of 0.73 and a TRS of 0.89 versus WebSailor-V2-30B's 0.75.
• Controlled training-paradigm comparison. Starting from a single Qwen-30B-Base model, the authors report that the "AgentFounder" method yields the highest overall Utility (0.80) and Pass@1 (58.2%) among the four paradigms tested. The remaining details of this study, and the full RQ2/RQ3 and ablation analyses, are cut off in the provided paper content and are not reported here.
• A design choice with a stated rationale. The geometric-mean formulation is deliberately used instead of a weighted sum, on the argument that a highly efficient trajectory producing an ungrounded answer is a critical failure. The evidence-grounding term is likewise a geometric mean over claims, so a single claim with near-zero entailment probability severely penalizes the score.

Methodology in Plain English

The researchers treat an agent's work on a question as a trajectory — the full sequence of actions and observations — rather than as a single answer.

To build the utility score, they first check whether the final answer is correct. If it is not, the utility is zero. If it is correct, they multiply together two scores:

• Process Efficiency, which compares a small bonus for tackling longer tasks against the accumulated cost of the agent's actions. Each action's cost is inflated by a Redundant Exploration Penalty if the agent takes repeated steps that gain no new relevant information. Whether a step is productive is judged by Marginal Information Gain: how much closer the new observation moves the agent toward the ground-truth answer than any previous observation did.
• Cognitive Quality, a blend of Evidence Grounding (each atomic claim in the final answer is checked against the evidence cited for it by an NLI model, then combined geometrically) and Reasoning Robustness (how quickly the agent recovers after falling into a deliberately embedded information trap, measured as an exponential decay over average recovery latency).

To measure latent ability, they feed the agent a controlled fraction of an oracle solution trajectory as a hint and search for the smallest fraction at which its expected success rate crosses a threshold. To characterize an agent's "personality," they correlate information gain with reductions in policy uncertainty (Entropy Adaptability) and measure consistency across repeated runs (TRS).

For data, they generated tasks from expert-verified academic seminars, extracting research concepts and having a "TaskWeaver" agent synthesize them into benchmark items with controllable complexity and embedded traps. Evaluation runs a dual-pathway assessment: one path checks claims against cited evidence for support, conflict, or omission, while separate Judge LLMs apply adaptive criteria to score overall quality. All agents were run in a unified framework with fixed interaction, inference, and metric-computation settings; hyperparameters are in the paper's Appendix A.3.

Why This Matters

Deep Research Agents are increasingly deployed to do real investigative work, and if the field optimizes against a metric that ignores how an answer was produced, it will keep shipping agents that look good on leaderboards while relying on unverified evidence. TRACE reframes evaluation from "did it get the right answer" to "was the process trustworthy, efficient, and recoverable from misinformation" — which is closer to what a human reviewer would actually care about.

Real-world applications that follow from this framing:

• Enterprise and market research automation, where an agent's cited sources must be verifiable before a report informs a decision, making Evidence Grounding the operative metric rather than answer accuracy alone.
• Fact-checking and misinformation triage, where the Reasoning Robustness metric — how fast an agent escapes a misleading but plausible source — maps directly onto the job.
• Long-horizon research assistants with bounded context, where the efficiency penalty for redundant exploration reflects a real operational cost in latency and token spend.
• Model selection and procurement, where a buyer choosing between a large accurate model and a smaller well-rounded one can now compare them on minimum-hint-rate and efficiency rather than a single score.

Industry relevance is direct: the framework gives agent developers a diagnostic vocabulary for failure modes (expensive trajectories, weak grounding, slow trap recovery) and gives evaluation teams a benchmark whose traps and oracle paths are controllable rather than incidental.

Future Directions

• Extending the controlled training-paradigm study. The paper compares four training paradigms on Qwen-30B-Base, but the provided content truncates before the full breakdown; a complete picture would clarify which methodology drives which TRACE dimension.
• Calibrating the geometric-mean weights. The utility function depends on hyperparameters ω_E, ω_C, β, γ, α, and λ. It is not reported in the provided content how sensitive the rankings are to these choices, or whether they generalize across task domains.
• Validating the Minimum Hint Rate against downstream outcomes. λ_min is proposed as a predictor of latent potential; open questions include whether a low λ_min in a benchmark setting translates into better performance on genuinely novel tasks.
• Broadening beyond question answering and text. The evaluation centers on complex, open-domain, knowledge-intensive QA with web tools. Whether TRACE's trap generation, evidence grounding, and efficiency metrics transfer to multimodal, code, or embodied research tasks is not addressed.
• Scaling the benchmark's trap design. Every task includes an oracle trajectory, which enables λ_min but also constrains how tasks can be synthesized; whether that requirement limits task diversity or introduces construction artifacts is an open question.

Target Audience

This paper is aimed at researchers and engineers who build or benchmark LLM-based research agents: agent-training teams deciding between architectures and RL post-training recipes, evaluation and benchmark designers looking for process-aware metrics, and practitioners who need to justify model selection beyond a single accuracy number. Readers in LLM evaluation and reinforcement learning will get the most out of the formal sections; readers primarily interested in deployment will find the multi-metric comparison tables the most actionable part. A passing familiarity with agent trajectories and metrics like Pass@1 is assumed throughout.

Authors’ abstract

The evaluation of Deep Research Agents is a critical challenge, as conventional outcome-based metrics fail to capture the nuances of their complex reasoning. Current evaluation faces two primary challenges: 1) a reliance on singular metrics like Pass@1, creating a "high-score illusion" that ignores the quality, efficiency, and soundness of the reasoning process; and 2) the failure of static benchmarks to quantify crucial attributes like robustness and latent capability. To address these gaps, we introduce TRACE (Trajectory-Aware Comprehensive Evaluation), a framework that holistically assesses the entire problem-solving trajectory. To counter the "high-score illusion", we propose a Hierarchical Trajectory Utility Function that quantifies process efficiency and cognitive quality, including evidence grounding, alongside accuracy. To measure deeper attributes, TRACE introduces a Scaffolded Capability Assessment protocol, quantifying an agent's latent ability by determining the minimum guidance needed for success. Our contributions include the TRACE framework, its novel metrics, and the accompanying DeepResearch-Bench with controllable complexity. Experiments show TRACE delivers a granular ranking that uncovers critical trade-offs between agent accuracy, efficiency, and robustness entirely missed by singular metrics.