MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMs

AI summary

Overview

Research area: Multimodal large language models (MLLMs) and video understanding evaluation benchmarks.

Technical level: Intermediate — no new model architecture is proposed, but familiarity with multimodal LLMs, video benchmarks, and evaluation methodology helps.

Scope: The paper introduces MVU-Eval, the first benchmark designed to evaluate how well multimodal LLMs understand and reason across multiple videos simultaneously rather than a single video.

What This Paper Is About

Nearly all existing video-understanding benchmarks feed a model one video at a time, but real tasks like multi-camera autonomous driving, cross-angle sports analysis, or summarizing several related clips require reasoning across several videos at once. The authors build MVU-Eval, a benchmark of 1,824 question-answer pairs covering 4,959 videos, spanning eight perception and reasoning skills that specifically demand cross-video integration. They then evaluate 25+ state-of-the-art open- and closed-source MLLMs on it to measure how far current models are from human-level multi-video understanding.

Key Contributions

1. The first multi-video understanding benchmark. MVU-Eval is the first evaluation suite where every question requires integrating information from multiple distinct videos, filling a gap left by single-video benchmarks such as Video-MME, MVBench, and LongVideoBench.

2. Eight core competencies under two progressive protocols. Four perception tasks (Object Recognition, Spatial Understanding, Counting, Comparison) and four reasoning tasks (Knowledge-Intensive Reasoning, In-Context Learning, Retrieval-Augmented Generation, Temporal Reasoning), each with sub-tasks tied to real-world scenarios like autonomous driving and sports analytics.

3. A hybrid, rigorously filtered data pipeline. Video pairs are sampled by task-specific rules (e.g., Jaccard caption similarity for RAG, shared objects for counting), questions are generated automatically or from templates, and roughly 54% of raw samples are discarded through two quality-control rounds — one removing answer leakage, one using human annotators — leaving 1,824 of 4,187 original pairs.

4. Extensive model evaluation with actionable diagnostics. 25+ MLLMs (Gemini 2.5 Pro, GPT-4o, Qwen2.5-VL, InternVL3, VideoLLaMA3, and others) are benchmarked, with ablations on frames, resolution, video count, input format, and visual-modality removal.

Main Findings

• Performance is far below human level. The best model, Gemini 2.5 Pro, reaches only 58.4% overall accuracy, while five human experts average 93.6%. Random guessing scores 26.0%. Except for the Qwen2.5-VL series, most open-source models fall below 50%.

• Strengths are uneven across tasks. No single model dominates every subtask: Qwen2.5-VL-72B leads on Object Recognition, Qwen2.5-VL-32B on Spatial Understanding, and Gemini 1.5 Pro on Counting. Qwen2.5-VL-72B actually beats Gemini 2.5 Pro on Retrieval-Augmented Generation (48.1% vs 43.7%).

• Scaling and context length both help — up to a point. Larger models in the Qwen2.5-VL (3B→72B) and InternVL3 (8B→38B) families consistently perform better. Increasing frames improves VideoLLaMA3-7B up to 32–48 frames but degrades at 64; increasing resolution helps up to 720px but degrades at 960, both due to token overload.

• Every part of the multi-video setup matters. Removing videos or visual information causes sharp drops. For VideoLLaMA3-7B, accuracy falls from 47.5% (multi-video) to 24.9% (single video), 34.6% (one frame per video), 41.0% (text descriptions only), and 16.0% (no video) — the last being below random chance, because models refuse to answer without visual evidence.

• More videos means harder questions. Accuracy declines steadily as the number of involved clips grows, confirming that questions genuinely depend on analyzing all inputs. Notably, 93.8% of questions involve two to six videos, with a maximum of 13.

• Input format alone changes results. Feeding Qwen2.5-VL-7B a merged video (45.2%) or per-video frame sets (44.6%) performs notably worse than native multi-video input (51.9%).

• Failure modes are specific and revealing. For perception, models detect objects but struggle with object status (e.g., whether a shovel is in use) and cross-camera spatial relationships. For reasoning, they fail at domain-specific knowledge, filtering irrelevant content, and explaining causal/temporal "why," even when they can describe "what" happened. Some models (e.g., LLaVA-Video-7B) also fail to follow the instructed answer format.

Methodology in Plain English

The authors start by selecting pairs (or groups) of videos according to rules tailored to each task — for example, videos sharing the same objects for counting questions, or one anchor video plus similar ones for retrieval tasks. Video clips come from established datasets including Kinetics-400, nuScenes, ScanNet, FineDiving, YouCook2, Vchitect-2.0, and DREAM-1K, plus 130 human-curated samples from real multi-video editing use cases, with faces and copyrighted content excluded.

Questions are then written either by MLLMs with rejection sampling or via hand-crafted templates that inject plausible distractors (like swapping difficulty levels between similar athletic moves). Two quality-control passes follow. The first catches leakage — cases where a model can guess the answer from option wording alone (e.g., "the classroom in Video 1") or from formatting cues like one long correct option among short wrong ones; options are rewritten until no-video accuracy drops to chance. The second pass uses three strong MLLMs to filter out easy questions and then human annotators to verify each question is answerable, challenging, and requires checking all videos. Answer options were also rebalanced, since generated options skewed toward A and B.

Evaluation is zero-shot and accuracy-based: 32 frames per video, longer side capped at 720 pixels, with rule-based regex parsing to extract answer letters even when models produce extra reasoning text.

Why This Matters

Impact on research. MVU-Eval reframes video evaluation around a capability that real deployments already demand but benchmarks have ignored: fusing information across unaligned, asynchronous video streams. It gives the field a shared, human-verified yardstick and shows that current architectures, training data, and token budgets are not yet sufficient for the task.

Real-world applications:

• Autonomous driving: combining simultaneous camera feeds from different angles on the same vehicle to judge traffic conditions and hazards.
• Sports analytics: cross-referencing multiple camera angles to classify actions, rate difficulty, and judge scores.
• Video search and summarization: synthesizing several retrieved clips into a coherent answer for a user query.
• Multi-camera surveillance and robotics: aggregating partially occluded views to count objects or track them across coverage gaps.

Industry relevance. The evaluation covers commercially important models from Google, OpenAI, Alibaba, and others, and the results are directly actionable for teams deciding which model to deploy for multi-stream video, how many frames to feed, and what resolution to use. The token-overload findings in particular inform practical inference-cost tradeoffs.

Future Directions

• Cross-video visual alignment. Most videos in the benchmark are asynchronous and not aligned on a shared timeline; methods for temporal synchronization across streams are needed.
• Cross-video spatial understanding. Models need to identify the same object across different viewpoints and use it as an anchor for geometric reasoning, a capability current models lack.
• Scalable multi-modal fusion. New architectures are required to handle high-cardinality video inputs without hitting token limits or suffering the performance degradation observed beyond 32 frames or 720px resolution.
• Broader modalities and scale. The authors flag two explicit limitations to address: the benchmark uses relatively short clips rather than movie-length sequences, and it evaluates only visual input, leaving audio integration for future work.

Target Audience

Researchers and engineers working on multimodal LLMs, video understanding, and benchmark design will find the most value here, as will practitioners building multi-camera or multi-stream video applications who need to choose models and inference settings. The paper is also useful for graduate students entering video-language research, since the task taxonomy and ablation methodology offer a clear template for constructing rigorous multimodal evaluations.

Authors’ abstract

The advent of Multimodal Large Language Models (MLLMs) has expanded AI capabilities to visual modalities, yet existing evaluation benchmarks remain limited to single-video understanding, overlooking the critical need for multi-video understanding in real-world scenarios (e.g., sports analytics and autonomous driving). To address this significant gap, we introduce MVU-Eval, the first comprehensive benchmark for evaluating Multi-Video Understanding for MLLMs. Specifically, our MVU-Eval mainly assesses eight core competencies through 1,824 meticulously curated question-answer pairs spanning 4,959 videos from diverse domains, addressing both fundamental perception tasks and high-order reasoning tasks. These capabilities are rigorously aligned with real-world applications such as multi-sensor synthesis in autonomous systems and cross-angle sports analytics. Through extensive evaluation of state-of-the-art open-source and closed-source models, we reveal significant performance discrepancies and limitations in current MLLMs' ability to perform understanding across multiple videos. The benchmark will be made publicly available to foster future research.