LifeSearch: Benchmarking Deep Research Agents for Real-World Life and Productivity

AI summary

Overview

Research area: Evaluation of Deep Research (DR) agents — autonomous systems that plan multi-step web searches and synthesize evidence into reports — with a focus on real-world life and professional productivity tasks.

Technical level: Intermediate. Understanding the paper fully requires familiarity with LLM agent harnesses, LLM-as-a-Judge evaluation, and rubric-based scoring.

Scope: The paper introduces LifeSearch, a bilingual benchmark of 203 expert-annotated tasks and 4,014 rubrics across seven life and productivity domains, and reports a comparative evaluation of 12 frontier DR agents under a unified harness.

What This Paper Is About

Deep research agents solve open-ended information needs by repeatedly searching the web, reading pages, and writing up findings, but existing benchmarks mostly test academic research or synthetic tasks rather than everyday life and professional work. LifeSearch fills that gap with a benchmark built from real user query seeds, refined by experts, and graded against fine-grained rubrics that often pin down exact numbers, units, dates, and URLs. The goal is to measure how well current frontier DR agents actually perform on realistic life and productivity research requests.

Key Contributions

1. The first benchmark centered on real-world life and professional productivity DR evaluation. LifeSearch shifts the evaluation lens toward realistic life and work search, revealing a concrete capability bottleneck in current frontier models.
2. A large-scale, expert-annotated bilingual benchmark. 203 tasks (102 Chinese, 101 English) spanning 7 domains, with more than 4,000 expert-written fine-grained rubrics — 4,014 in total, averaging 19.8 per task — produced through over 4,500 hours of human annotation labor.
3. Judge model selection for the DR rubric setting. Four frontier LLMs across 11 configurations of reasoning effort and temperature were compared on 402 human-annotated rubric labels to choose the grading configuration.
4. A comprehensive empirical study. 12 current DR agents benchmarked under a unified harness, with agreement with human judgments examined at both the rubric level and the overall-ranking level.

Main Findings

• Substantial headroom remains. The best-performing frontier agent, GPT-5.6 Sol, scores only 63.44% (± 0.27 across three runs), followed by Claude Opus 5 at 56.10% (± 0.28). All other systems score below 50%, and no system exceeds 70% in any of the seven domains. GPT-5.6 Sol leaves 36.6% of the normalized requirement weight unsatisfied and falls below 50% on 41 of 203 tasks (20.2%).
• Quality transcends volume. GPT-5.6 Sol has both the highest score and the largest output volume (51.5k tokens per task). Claude Opus 5 and GLM-5.3 generate similar volumes (48.9k and 49.3k) yet differ by 6.9 percentage points. Hy4 Preview reaches 49.73% with only 33.6k tokens, comparable to GLM-5.3 at 49.19%.
• Evidence collection correlates with score. Mean interaction count correlates positively with overall score (Spearman's ρ = 0.76). GPT-5.6 Sol averages 86.2 iterations and scores 63.4%, whereas GLM-5.2 averages 11.0 iterations and scores 37.1%. The authors note that iterations include other tool actions, so this does not isolate a causal effect of extra searches.
• Final-report length correlates more weakly with score. Final-report length in characters correlates with score at Spearman's ρ = 0.60. Doubao-Seed-2.1-Pro and Gemini-3.6-Flash produce 12,089 and 10,256 characters per report while scoring 43.1% and 34.7%, whereas Hy4 Preview reaches 49.7% with 9,959 characters.
• Three recurring failure modes. Premature clarification (Doubao-Seed-2.1-Pro and GPT-5.6 Sol sometimes ask follow-up questions and, because the harness provides no replies, terminate before producing a report), temporal scope violations (agents default to current web data even when a query specifies an earlier cutoff, notably Hy4 Preview), and missing required details (fluent summaries omit requested entities, numerical values, dates, sources, or edge cases).
• Judge selection results. Gemini 3.1 Pro with low reasoning effort and zero temperature achieves the highest mean accuracy (84.25%) over three runs, with 98.8% self-consistency and 84.33% majority-voted accuracy. Inter-annotator agreement on the 402 rubrics was a mean pairwise Cohen's κ = 0.715, with all 65 initial disagreements resolved through discussion.
• Majority voting adds little. Majority voting improves over mean accuracy by at most 0.42 percentage points across all configurations, moving Gemini 3.1 Pro from 84.25% to 84.33%, so a single judge pass per report was adopted.
• Stability is not accuracy. Temperature zero improves self-consistency (Gemini 3.1 Pro from 94.5% to 98.0%; DeepSeek V4.1 Flash from 91.3% to 94.8% at high reasoning effort), yet Gemini 3.5 Flash without reasoning reaches 99.0% self-consistency at only 75.62% mean accuracy.
• Strong human agreement. A human ranking study on 30 tasks (15 per language) covering 210 reports yields Spearman's ρ = 0.89 and Kendall's τ = 0.81 between aggregate rubric scores and human rankings, with identical ordering of the top four systems. Hy3 moves down 2 ranks relative to human judgment; Claude Opus 4.8 and GLM-5.2 each move up 1.
• Domain variation across models. GPT-5.6 Sol and Claude Opus 5 rank first and second in all seven domains. GLM-5.3 outperforms Hy4 Preview on everyday facts (51.9% versus 47.1%), while Hy4 Preview leads on academic and professional literature (51.4% versus 49.0%); the authors describe these differences as descriptive, especially given the nine-task everyday-facts subset.

Methodology in Plain English

The benchmark was built in four stages. First, crowd workers wrote initial query seeds spanning workflows such as file operations, software engineering, and deep research. An LLM then filtered for search relevance, query quality (rated high, medium, or low), and dependence on an unspecified recent event, and classified the retained queries into seven productivity domains. A rewriting step stripped out operational actions such as generating local files or HTML visualizations while preserving research intent, source constraints, temporal expressions, and organization requirements. Finally, experienced AI researchers manually selected tasks that were complex, diverse, clear, and feasible to evaluate.

Experts then refined each task through three steps: decomposing it into typically two to four subtasks typed as Lookup, Compute, Compare, or Reason; enriching those subtasks around typically two capabilities chosen from Search Planning, Information Synthesis, Conflict Resolution, and Uncertainty Handling; and rewriting everything into a coherent natural-language request grounded in a concrete life or work scenario.

For each task, experts wrote rubrics — self-contained Pass/Fail acceptance statements. Each rubric records a criterion, a type (Objective or Subjective), a necessity (Explicit or Implicit), a priority (Must Have or Nice to Have), and a weight — a floating-point value in [0, 1] to one decimal place that reflects relative importance within the task. Roughly 40% of rubrics explicitly specify numerical values, specific events, or named documents. Quality control used an iterative annotation–inspection–revision cycle plus outlier screening, where low-scoring tasks were checked for flaws and overly easy tasks were tightened.

Evaluation uses a rubric-based LLM-as-a-Judge: the judge assigns a binary Pass/Fail to each rubric, and the weighted satisfaction score is normalized within each task, then averaged uniformly across tasks so that every task counts equally regardless of rubric count. All 12 agents run in a shared ReAct-style harness with web search, web fetch, and code execution, using Serper as the search backend and Jina Reader to retrieve pages. Each task is run three independent times, and results are reported as the mean and sample standard deviation.

Why This Matters

Impact on research. Existing DR benchmarks lean toward academic or synthetic tasks and their rubrics often lack the numerical grounding and independent verifiability needed to judge whether a report is usable as a real-world deliverable. LifeSearch supplies that missing evaluation surface, and its judge-selection study gives the community a validated grading configuration. The top score of 63.44% establishes a concrete, quantified capability gap rather than a qualitative claim.

Real-world applications. The benchmark's seven domains map directly onto everyday and professional use:

• Comparing companies and competitors for business decisions.
• Reviewing academic and professional literature.
• Analyzing financial markets.
• Looking up policy and government data.
• Aggregating industry news and trends.
• Evaluating technical tools and developer resources.
• Answering everyday factual questions.

Industry relevance. The paper comes from the LLM Department at Tencent and benchmarks 12 frontier systems including GPT-5.6 Sol, Claude Opus 5, Hy4 Preview, GLM-5.3, Qwen3.8-Max, DeepSeek-V4 Pro, Kimi-K3, Doubao-Seed-2.1-Pro, Claude Opus 4.8, Hy3, GLM-5.2, and Gemini-3.6-Flash. The finding that report volume does not reliably predict quality, and that autonomous runs can silently fail through premature clarification or temporal scope violations, is directly relevant to anyone deploying research agents in products where users expect a delivered report rather than a clarifying question.

Future Directions

• Closing the capability gap. With the strongest agent at 63.44% and all systems below 70% in every domain, it is an open question which improvements — better search planning, stronger evidence synthesis, or better requirement tracking — would most efficiently raise scores.
• Mitigating the identified failure modes. Premature clarification, temporal scope violations, and omitted required details are observed but no fixes are proposed or evaluated in this paper.
• Isolating causal effects of effort. The correlation between interaction count and score (ρ = 0.76) is confounded because iterations include non-search tool actions; controlled experiments would be needed to establish whether more searching causes better outcomes.
• Extending coverage. The authors note that the everyday-facts subset contains only nine tasks, and the human ranking study covers 30 of the 203 tasks; expanding both would test whether the reported patterns hold at larger scale.

Target Audience

Researchers and engineers working on deep research agents, agentic web search, and long-horizon LLM systems will benefit most, along with practitioners who need to select or deploy a research agent for real business or consumer workflows. Evaluation specialists interested in rubric design, LLM-as-a-Judge validation, and human-agreement studies will also find the judge-selection methodology and the 4,014-rubric construction process directly useful.

Authors’ abstract

Deep research (DR) agents address open-ended information needs through multi-step reasoning, long-horizon web search, and evidence synthesis. However, existing benchmarks primarily focus on academic research or synthetic tasks, leaving DR agents' capabilities in real-world life and professional productivity settings insufficiently evaluated. To bridge this gap, we introduce LifeSearch, a challenging benchmark comprising 203 Chinese and English tasks derived from real-user query seeds across 7 concrete life and productivity domains. With more than 4,500 hours of human annotation, we construct over 4,000 expert-written, fine-grained rubrics, covering both report-content criteria for coverage and accuracy and concrete criteria specifying values, units, dates, and URLs. We conduct extensive experiments on 12 advanced DR agents under a unified evaluation harness, systematically analyzing their performance patterns and failure modes. We find that even the best-performing frontier agent, GPT-5.6 Sol, achieves a score of only 63.4%, leaving substantial room for improvement on realistic life and professional DR tasks. We select the grading configuration by comparing candidate LLM judges against human rubric labels, and a human ranking study shows strong alignment between aggregate benchmark scores and human judgments (Spearman's $ρ= 0.89$, Kendall's $τ= 0.81$). LifeSearch provides a rigorous evaluation framework and foundation for advancing DR agents toward real-world life and productivity.