HERBench: A Benchmark for Multi-Evidence Integration in Video Question Answering

AI summary

Overview

Research area: Computer Vision, specifically Video Question Answering (VideoQA) evaluation for Video Large Language Models (Video-LLMs).

Technical level: Advanced. The paper assumes familiarity with multimodal LLM architectures, frame-selection policies, object tracking pipelines (RF-DETR, DeepSORT), and benchmark design.

Scope: This paper introduces HERBench, a 26,806-question VideoQA benchmark constructed so that every question structurally requires integrating at least three non-overlapping visual cues, together with the Minimum Required Frame-Set (MRFS) metric and a diagnostic evaluation of 13 Video-LLMs.

What This Paper Is About

Current VideoQA benchmarks can often be answered using a single salient frame or language priors, so high scores do not prove that models actually reason across time. The authors build a benchmark where multi-evidence integration is unavoidable: each of the 26,806 five-way multiple-choice questions is constructed to require at least three distinct, non-overlapping cues drawn from separate video segments. The goal is to measure evidential demand directly and to separate two failure modes—failing to retrieve the right frames versus failing to fuse them once retrieved.

Key Contributions

1. HERBench benchmark: 26,806 five-way multiple-choice questions across 12 compositional tasks and four reasoning families, constructed to structurally enforce k ≥ 3 distinct, non-redundant visual cues per question.
2. Minimum Required Frame-Set (MRFS) metric: An automated, model-centric measure of the smallest number of frames a model must fuse to answer correctly, computed with adaptive bisection in O(log x) model calls per item, enabling cross-benchmark comparison under a canonical protocol.
3. Diagnosis of two bottlenecks: By disentangling frame selection from multi-evidence reasoning, the authors identify (i) a retrieval deficit, where adaptive selectors still overlook key evidence and trail oracle keyframes, and (ii) a fusion deficit, where models fail to integrate complementary information even when all necessary frames are provided.
4. Oracle-frame and human-validation design: The benchmark provides curated ground-truth frames and reports human accuracy on those frames, which allows retrieval failures to be measured separately from fusion failures.

Main Findings

• Overall model performance is low: Across 13 state-of-the-art MLLMs, mean accuracy is 38.2%, with the best model (Ovis-2.5-9B) reaching 42.1% and the lowest (LLaMA-4-Scout-17B) at 31.4% — a narrow band only 11–22 percentage points above the 20% random-guess baseline.
• Task-family breakdown: Models average 66.8% on Referring & Tracking tasks but only 26.8% on Temporal Reasoning & Chronology, 34.8% on Global Consistency & Verification, and 26.3% on Multi-Entity Aggregation & Numeracy.
• Multi-cue aggregation collapses: [AC] (Action Counting) and [MEGL] (Multi-Entities Grounding & Localization) average 23.0% and 23.2% respectively, barely above chance. [TSO] (Temporal Shot Ordering) scores fall as low as 0.1%.
• Single-entity tracking is comparatively intact: Ovis-2.5-9B reaches 79.7% on [AGBI] and 76.2% on [AGAR], suggesting models can follow one described entity but not combine distributed evidence.
• Cross-benchmark evidential demand: Under the canonical MRFS protocol (Qwen2.5-VL backbone, AKS selector, x = 16 frames), HERBench attains the highest mean MRFS at 5.49, above Video-MME (5.31), MINERVA (5.14), MMBench-Video (4.41), LongVideoBench (4.07), AGQA balanced (3.42), MVBench (3.52), CVRR-ES (2.77), NExT-QA (2.61), and TemporalBench (2.21).
• Retrieval is a bottleneck: Averaged over 1200 questions (100 per task) for three models, Oracle Frames reach 47.8 (InternVL3.5-14B), 41.0 (Qwen3-VL-8B), and 47.9 (Ovis-2.5-9B), exceeding learned selectors such as AKS (42.7, 36.2, 42.6) and BOLT-ITS (41.1, 38.4, 42.1).
• Even oracle frames do not solve it: With correct evidence frames supplied, accuracy stays below 50%, showing that access to the right information is insufficient.
• Errors concentrate on one frame: Under oracle-only frames, correct predictions show more balanced importance allocations with mean Top-1 shares near 0.5, while incorrect predictions frequently approach 0.8, indicating that errors come from misallocating weight to a single frame.
• Humans handle the oracle setting well: Annotators achieved 88.8% accuracy with full-video access and 95.7% in the oracle-frame setting, confirming the curated frame set is generally sufficient to resolve the question.
• Benchmark composition: 336 unique videos averaging 395 s (range 60–2100 s), with 12 tasks organized into four families: Temporal Reasoning & Chronology, Referring & Tracking, Global Consistency & Verification, and Multi-Entity Aggregation & Numeracy.

Methodology in Plain English

The authors build videos into a rich spatiotemporal scaffold using three parallel streams. The first stream runs object tracking: RF-DETR-L detections are linked with DeepSORT, and the top 20% of entities are kept via a composite TrackRank score favoring appearance rarity, trajectory length, and frame coverage. Selected tracks yield decoupled "A-cards" describing appearance and "B-cards" describing behavior and trajectory, generated from temporally separated portions of the trajectory, so that identifying a person and reasoning about what they did require different moments of the video. The second stream performs shot segmentation, using a multimodal LLM to summarize each shot into a scene card, which supports chronological ordering and fidelity-checking tasks. The third stream integrates human-verified raw event logs to anchor counting, absence-detection, and sequence tasks in confirmed ground truth.

Questions are then compiled from these representations by oriented task programming. Quality control includes token-level A/B-card disentanglement checks, manual leakage review, discarding items solved by at least 3 of 4 blind LLMs, and expert verification on a stratified 15% sample to confirm k ≥ 3 compliance and answer uniqueness. To measure evidential demand, the authors introduce MRFS: they feed a fixed model the top-k frames under a fixed selector and find the smallest k that yields a correct answer, searching by adaptive bisection. Items that are correct with no frames at all are excluded as text-only solvable. For the experiments, all models receive the same budget of 16 uniformly sampled frames. To isolate fusion from retrieval, the authors also run a study supplying only manually curated ground-truth frames, and compute per-frame leave-one-out importance deltas to see how weight is distributed across frames.

Why This Matters

Impact on research: The paper argues that benchmark scores alone do not measure grounded temporal reasoning. By controlling the evidential requirement and providing MRFS as a measurable quantity, it offers a way to compare benchmarks on how much multi-moment integration they actually demand, and it makes retrieval failures separable from fusion failures — two problems that require different research responses.

Real-world applications:

• Video surveillance and monitoring, where answering a question such as who entered a region and when requires combining several moments rather than one frame.
• Video search and retrieval over long archives, where queries depend on events scattered across the timeline.
• Assistive and accessibility tools that answer questions about long recordings for users who cannot watch the whole video.
• Robotics and embodied agents that must accumulate observations over time before acting.

Industry relevance: Any organization deploying Video-LLMs for long-form video analysis needs to know whether a model is reasoning over distributed evidence or exploiting shortcuts. HERBench's oracle-frame design and MRFS metric provide an evaluation instrument for that question, and its findings suggest that scaling model size or context length alone does not address the fusion deficit.

Future Directions

• Improve frame selection: Closing the gap between adaptive selectors (AKS, BOLT-ITS) and Oracle Frames, since learned selectors still overlook key evidence.
• Improve multi-evidence fusion: Addressing the tendency of models to concentrate attention on one frame even when all required evidence is present, which the authors identify as a primary source of failure independent of retrieval.
• Retrieval-aware querying: The conclusion points toward future improvements in retrieval-aware querying and distributed evidence fusion as complementary research directions.
• Open questions the paper raises: The authors note that because HERBench is partially instantiated through an oriented task-programming pipeline, some residual systematic artifacts may remain, and they state that the benchmark ordering under MRFS remains stable across alternative backbones and selectors (with details deferred to the Supplementary Material), leaving broader robustness of the metric as a topic for further study.

Target Audience

Researchers and engineers working on Video-LLMs, VideoQA, long-video understanding, and multimodal evaluation design. It is most useful for readers who already understand transformer-based multimodal models and benchmark methodology, and who want a principled way to test multi-evidence integration rather than single-cue performance. Practitioners selecting or auditing video-capable models for long-form content will also find the retrieval-versus-fusion diagnostic directly applicable.

Authors’ abstract

Video Large Language Models (Video-LLMs) are improving rapidly, yet current Video Question Answering (VideoQA) benchmarks often admit single-cue shortcuts, under-testing reasoning that must integrate evidence across time. We introduce HERBench, a benchmark designed to make multi-evidence integration unavoidable: each question requires at least three non-overlapping cues drawn from distinct video segments. HERBench contains 26,806 five-way multiple-choice questions across 12 compositional tasks. To make evidential demand measurable, we introduce the Minimum Required Frame-Set (MRFS), the smallest number of frames a model must fuse to answer correctly, and show that HERBench imposes higher evidential demand than prior benchmarks. Evaluating 13 state-of-the-art Video-LLMs yields only 31-42% accuracy, only modestly above the 20\% random-guess baseline. We disentangle this failure into two critical bottlenecks: (1) a retrieval deficit, where frame selectors overlook key evidence, and (2) a fusion deficit, where models fail to integrate information even when all necessary evidence is provided. HERBench thus provides a principled benchmark for studying robust multi-evidence video understanding.