Beyond Watching and Searching: Benchmarking Agents for Video-centered Deep Research Reports

AI summary

Overview

Research area: Computer Vision / multimodal LLM agents — specifically benchmark design for long-horizon, video-grounded "deep research" agents.

Technical level: Advanced. The paper assumes familiarity with LLM agent harnesses, tool-use workflows, rubric-based evaluation, and multimodal model-as-judge protocols.

Scope (one sentence): The paper defines a new task — producing an interactive, cited research report organized around one source video — and releases a 50-task, 6,414-criterion benchmark (OmniDR-Bench) built with an audited rubric-construction pipeline, then measures 18 model–harness configurations on it.

What This Paper Is About

Existing "deep research" agents write long cited reports from web text, video deep-research benchmarks ask short-answer questions about clips, and text-image report systems synthesize web material without ever reading a source video. None of them evaluates a complete report that stays anchored to one specific video while pulling in external multimodal evidence. The authors define this missing setting, Video-Centered Deep Research Report Generation, and build the benchmark needed to measure it.

Key Contributions

1. A new task definition: Video-Centered Deep Research Report Generation — given a source video, a user request, and an open tool environment, an agent must identify research-worthy claims, procedures, parameters, entities, and open questions, retrieve external webpage/image/video/document evidence, and produce an interactive HTML report whose findings link back to specific video moments.

2. Omni Atomic Rubric Trees (Omni-ARTs): A three-level evaluation hierarchy — three dimensions (content, citation, presentation), 23 shared diagnostic directions, and task-specific atomic binary criteria — where each leaf is either a multimodal model judgment (J) or a deterministic program/browser check (M).

3. Audited Rubric Construction (ARC): A staged protocol (S0–S6) that builds an independent fact library before writing any criterion, using isolated agents to produce a Video Evidence Map and a Web Evidence Map, then linking them through support, contradiction, update, explanation, comparison, and application relations. Includes static checks, behavioral calibration across three report tiers, and up to three human-review rounds.

4. OmniDR-Bench and a reference workflow: 50 Chinese and English tasks across 19 real-world subcategories with 6,414 criteria, plus a seven-stage skill-based agent workflow packaged as a "video-research skill" within a suite of thirteen skills that invokes video-reader and doc-reader subagents.

Main Findings

• Best system reaches 63.3/100. Claude Opus 5 under Claude Code has the highest point estimate (63.3 overall, 66.0 on successful reports) but misses 4.0% of tasks. GLM-5.3 completes every task at 62.0; the 1.3-point gap has a task-bootstrap 95% confidence interval of [-4.0, 5.6], so their ordering is uncertain.

• Presentation vastly outpaces research substance. Under the shared interface template, presentation averages 81.8 points, versus 45.4 for content and 47.8 for citation — agents satisfy format requirements far more consistently than audited coverage, reasoning, and citation-support criteria.

• Direct observation works; synthesis collapses. The mean pass rate across configurations is 79.9% for direct video facts, but drops to 8.7% for external evidence, 8.8% for cross-source verification, and 5.0% for multi-fact composition. Fact coverage, causal analysis, and comparison average 7.6%, 9.5%, and 4.6% respectively. Video faithfulness averages 92.4%, while logical consistency and claim support reach only 21.2% and 15.9%.

• Omissions dominate, not contradictions. Inspection of failed coverage criteria shows that missing audited research targets — rather than explicit factual conflicts with the video — drive most coverage failures.

• Finding sources is easier than using them. Under Claude Code, GPT-5.6 Sol passes 98.0% of source-count criteria and 93.5% of multimodal-material criteria but only 23.9% of claim-support criteria; Gemini-3.8-Flash passes 93.0% of multimodal-material criteria yet reaches 13.9% on claim support. Trace inspection shows reports frequently collect links in a final Sources section without citing them beside the claims they support.

• Skill ablation shows orchestration matters most for citations. On 15 matched tasks with GPT-5.6 Sol and Claude Code, adding text web research to source-video processing raises the all-task mean from 31.8 to 45.5 and success from 73.3% to 100.0%. Multimodal retrieval adds only 1.1 points overall (a 5.2-point content gain offset by a 5.3-point citation loss). Full orchestration adds 7.3 points, with a paired 95% interval of [3.4, 11.1], improving 12 of 15 tasks — its largest gain is 20.7 citation points against just 0.8 content points.

• Quality comes at a cost. Full orchestration raises median steps from 103 to 167, time from 0.7 to 1.6 hours, and tokens from 2.3M to 4.7M.

• Harness effects are model-dependent. Claude Code and Codex both average 48.4 points on the three matched models, but Codex completes 99.3% of runs versus 96.0% for Claude Code. OpenClaw has the lowest mean (39.4) and lowest completion rate (84.7%).

• Video category barely matters; the bottleneck is universal. The four video families span only 2.0 points (52.8 to 54.8), while subcategories range from 46.0 to 59.1. Query forms average 41.3 (general), 52.5 (video view), and 50.4 (user demand) — the authors note these remain descriptive since subcategories hold one to five tasks.

• Interleaving correlates with, but does not guarantee, quality. Median post-write search ranges from 33.3% (GPT-5.6 Sol) to 61.5% (Claude Opus 5) under Claude Code; GLM-5.3 reaches 54.2% with 15.5 median write blocks. Eight configurations meet the interleaving rule, but trajectory length alone does not predict quality.

• Run reliability is imperfect. Across 900 model–task attempts, 835 reports are evaluable and 65 are unsuccessful or unavailable; 19 of the latter are detected unfilled interface skeletons counted as failures.

• Evaluation is stable. Only 0.89% of applicable criteria receive non-unanimous decisions on average across three scorings of each R2 report.

Methodology in Plain English

Building the tasks. The authors manually sourced and downloaded videos from YouTube and Bilibili across four families chosen to reflect why people research videos: knowledge and viewpoints (explain and verify), consumer and business (compare and decide), physical procedures (reconstruct and diagnose), and creative and digital (workflows and transferable practice). Each query comes from a pool combining a general report request with two focused views — a video view (theories, cases, objects, claims) and a user demand view (a concrete decision, diagnosis, reproduction, or application need). A task is accepted only when video-only analysis is insufficient and generic web research cannot substitute for the source video. The fixed benchmark is 50 tasks over 19 subcategories, averaging 128 criteria each.

Building the rubrics without peeking at submissions. The core design worry is that if you write criteria by looking at what agents produce, you reward one system's source choices and organization. So ARC builds an independent fact library first: isolated agents produce a timestamped Video Evidence Map and, separately, a Web Evidence Map covering webpages, documents, images, and related videos. The two maps are then connected via six relation types (support, contradiction, update, explanation, comparison, application). Only after that fact library exists are candidate binary criteria generated, screened for being atomic, evidence-grounded, objectively worded, unambiguous, unique, and directionally balanced, then assembled under the three dimensions and 23 directions with caps so high-volume directions cannot dominate the score.

Calibration and auditing. ARC generates three reports with Claude Opus 4.8 under Claude Code — R1 (video only), R2 (full tool set plus the video-research skill), and R3 (fact-library-assisted) — and requires R1 < R2 < R3 with a minimum R3−R1 gap of 20% on every retained task, confirming the rubric responds to added evidence access. Static checks strip schema, identifier, evidence-binding, and logic errors before any behavioral test; human reviewers return semantic errors to a repair stage; up to three rounds are allowed before the evidence, rubrics, and evaluator versions are frozen.

Scoring. Each leaf is binary. Direction rates are atom pass rates, except for three high-volume directions (fact coverage, causal analysis, comparison) which are saturating — credited as min(1, rate / 0.18) — and capped at 15 atoms, following the logic that the standard report budget can only hold so much evidence. A failed gate direction zeros that direction's contribution. Dimensions combine direction rates weighted by scored atom count, and the final score weights content, citation, and presentation at 0.50, 0.30, and 0.20. A guarded N/A procedure handles omitted facts: a failed citation criterion is excused only if its linked coverage checks were all scored zero and a direction-level guard finds the content gain from stating the fact exceeds the citation gain from removing it — preventing agents from being penalized twice for the same omission.

Judging. GPT-5.6 Sol judges semantic and visual criteria at temperature zero; gate criteria are judged three times with majority vote. Deterministic programs, including browser interaction probes, handle everything directly observable — file structure, sources, assets, navigation, citation jumps, playback, controls. All 18 configurations use the same judge, prompts, and evaluator version.

Why This Matters

Impact on research. The paper identifies a genuine blind spot: rubrics and benchmarks for open-ended multimodal reports must be built independently of the systems being evaluated, or they encode those systems' idiosyncrasies. The ARC protocol — build the fact library first, generate criteria second, calibrate third — is a transferable template for any open-ended generation task where there is no single reference answer. The finding that agents score 92.4% on video faithfulness but 15.9% on claim support also reframes where multimodal agent progress actually stands: perception is largely solved relative to evidence integration.

Real-world applications (as framed by the paper's motivation):

• Helping viewers locate the specific moment in a long video that answers their question.
• Verifying claims made in a video against primary sources, standards, and updated context.
• Reproducing a demonstrated procedure by linking each step to its timestamp and its supporting documentation.
• Supporting practical consumer and business decisions — comparison and alternative discovery — from video content.

Industry relevance. The ablation results give product teams a concrete cost–benefit picture: orchestration and citation work is what moves scores (20.7 citation points), while simply adding multimodal retrieval barely helps (1.1 points) and can even hurt citation quality. The step, time, and token deltas (103→167 steps, 0.7→1.6 hours, 2.3M→4.7M tokens) quantify the price of that quality gain. Harness choice matters too, but model-dependently — Codex completes 99.3% of runs versus 96.0% for Claude Code at identical average score.

Future Directions

• Better audiovisual use. Video faithfulness is already 92.4%, but external multimodal evidence passes at only 8.7% — closing that gap requires agents to actually retrieve and integrate related videos, images, and documents rather than treating retrieval as a checkbox.

• Cross-modal reasoning and multi-fact composition. At 5.0% composition and 9.5% causal analysis, the hardest open problem is not finding facts but chaining several of them across modalities into a supported argument.

• Efficiency of long-horizon research. Full orchestration costs 4.7M tokens and 1.6 hours per report for 7.3 points; finding trajectories that transfer evidence productively rather than merely running longer is an open question the paper explicitly flags.

• Citation support as a first-class target. The Sources-section-without-inline-citation failure pattern is a specific, diagnosable behavior that current agents exhibit under every harness tested — the authors list citations as a named future work direction.

Target Audience

Researchers and engineers building or evaluating multimodal deep-research agents; benchmark designers who need evaluation protocols for open-ended, no-single-answer generation tasks; teams deploying video-analysis products that must produce auditable output rather than summaries; and applied scientists working on citation grounding and long-horizon agent trajectory analysis. Readers seeking a first introduction to LLM agents will find the setup accessible but the evaluation machinery dense.

Authors’ abstract

Deep research agents synthesize open-web evidence into long reports, but current benchmarks do not evaluate research centered on a source video. We introduce Video-Centered Deep Research Report Generation. Given a video and user request, an agent identifies research questions, retrieves external multimodal evidence, and produces an interactive report linked to relevant video moments. This task has direct practical value because it helps users locate relevant moments, verify claims, reproduce demonstrated procedures, and make informed decisions from video content. We present a skill-based workflow and OmniDR-Bench for this task. For evaluation, we propose Omni Atomic Rubric Trees (Omni-ARTs). Each tree contains three dimensions, 23 shared directions, and task-specific atomic binary criteria. The directions test audiovisual grounding, multimodal evidence use, and interactive presentation. Audited Rubric Construction (ARC) builds these trees through independent evidence collection, automated checks, behavioral calibration, and iterative human review. OmniDR-Bench contains 50 Chinese and English tasks across 19 realworld subcategories. It includes 6,414 criteria. Across 18 agent configurations, the best system scores 63.3 out of 100. Current agents still need broader content coverage, stronger causal and logical analysis across modalities, and more accurate citation support. We further analyze models, harnesses, skill configurations, video categories, and research trajectories. The code and data are released at https://github.com/marinero4972/OmniDR-Bench.