CrossVid: A Comprehensive Benchmark for Evaluating Cross-Video Reasoning in Multimodal Large Language Models

AI summary

Overview

Research area: Multimodal large language models (MLLMs), video question answering, and benchmark design for video reasoning.

Technical level: Intermediate. Readers should be familiar with vision-language models, video QA benchmarks, and standard evaluation metrics (accuracy on multiple-choice and open-ended tasks).

Scope: This paper introduces CrossVid, a large-scale benchmark (5,331 videos, 9,015 QA pairs, 10 tasks) built to measure how well multimodal LLMs can reason jointly across multiple related videos rather than a single clip.

What This Paper Is About

Existing video benchmarks mostly test a model on one video at a time, or at best on multiple camera angles of the same scene. Real-world tasks, however, often require watching several related videos together — comparing cooking techniques, tracking a plot across film clips, or triangulating an object's position from two drone feeds. CrossVid is the first benchmark built specifically to test this "cross-video reasoning" (CVR) ability, and it shows that current MLLMs, even the strongest ones, fall far short of human performance on it.

Key Contributions

1. A new benchmark for cross-video reasoning. CrossVid organizes evaluation into four high-level dimensions (comparative analysis, temporal understanding, multi-view reasoning, and free-form QA) that expand into ten concrete tasks requiring a model to integrate evidence spread across a group of semantically related videos.

2. A large, multi-format dataset. The benchmark contains 5,331 video clips drawn from six public sources (Animal Kingdom, MovieChat-1K, YouCook2, VisDrone, Charades, Assembly101) and 9,015 QA pairs spanning single-choice, multiple-choice, closed-ended generation, and open-ended generation formats. Each query requires understanding roughly 770 seconds of video on average.

3. A semi-automated annotation pipeline with human verification. Frames are captioned by Qwen2.5-VL-72B, QA pairs are drafted by DeepSeek-R1 using task-specific prompts, and ten expert annotators filter, refine, and quality-check the results. A temporal-realignment trick deliberately breaks low-level visual continuity in the sequencing task to prevent shortcut learning.

4. A broad empirical study of 22 MLLMs. The paper evaluates closed-source and open-source models from 7B to 78B parameters, including Mixture-of-Experts and "thinking"-enabled architectures, plus ablation studies on input frame count and chain-of-thought prompting.

Main Findings

• Cross-video reasoning is far from solved. The best model, Gemini-2.5-Pro, reaches only 50.4% average accuracy, while human performance is 89.2%. On multi-view reasoning, the top model scores 40.7% versus 93.7% for humans, and on the functional step alignment task Gemini-2.5-Pro manages just 13.4% against 85.2% for humans.

• Closed-source models dominate open-source ones. Every closed-source model outperforms every open-source model on overall average accuracy. On temporal understanding, the weakest closed-source model (GPT-4o at 35.5%) still beats the best open-source model (LLaVA-OV-72B) by 6.2 percentage points.

• Explicit reasoning ("thinking") mechanisms help. Models with built-in reasoning modules, such as Gemini-2.5-Pro and GLM-4.1V-9B-Thinking, consistently rank at the top of their size classes, suggesting that structured multi-step reasoning is valuable for cross-video tasks.

• More frames help, but not uniformly. Increasing input frames from 32 to 256 raised Qwen2.5-VL-72B's overall accuracy from 33.8% to 39.5%, with open-ended culinary QA improving by 15.1%. Beyond a point, however, irrelevant frames introduce noise — a plot-inference case showed accuracy dropping when extra atmospheric shots distracted the model from the causal chain.

• Chain-of-thought prompting gives mixed but generally positive results. CoT helped most models on temporal and multi-view tasks, but only larger open-source models gained overall, and gains on individual tasks were inconsistent.

• Four dominant failure modes. Manual error analysis identified: key frame loss (too few frames per video when many videos share a budget), single-video understanding errors, cross-video comparison errors (each video is understood but not compared), and format errors (failing to follow output constraints like timestamps).

Methodology in Plain English

The researchers first collected video clips from six existing public datasets, choosing sources that naturally support comparison — cooking videos with step labels, film clips with genre tags, drone footage from two synchronized viewpoints. They then had a strong vision-language model (Qwen2.5-VL-72B) write short captions for extracted frames, and fed those captions to DeepSeek-R1 with carefully written, task-specific prompts that instructed it to generate questions depending on relationships among the videos, not just within one. Ten expert annotators then removed bad questions (those about a single video, or too subjective), rewrote ambiguous ones, answered them independently to verify ground truths, and ran a final quality-control pass.

For evaluation, the team fed all videos in a group plus the question into each model in a single turn, sampled frames evenly, and used a zero-shot setup with no fine-tuning. They reported accuracy across all tasks, ran ablations on how many frames to feed and whether to prompt for step-by-step reasoning, and manually categorized reasoning errors.

Why This Matters

Impact on research. CrossVid shifts video QA benchmarking from "watch one video, answer one question" to "watch a group of videos, aggregate and compare." It provides a concrete, reproducible target that exposes a capability gap most benchmarks miss, and its error taxonomy gives model developers a diagnostic vocabulary for what specifically breaks.

Real-world applications.

• Cooking and instructional media: comparing technique differences across recipe videos for training or quality control content.
• Autonomous driving and drone operations: fusing synchronized views from multiple cameras to reason about object positions and distances.
• Media and entertainment analysis: comparing plot, character, and theme across film clips for recommendation or editorial tools.
• Industrial assembly and inspection: spotting procedural errors across recordings of the same task performed multiple times.

Industry relevance. Video-capable MLLMs are being deployed in assistants, content moderation, robotics, and analytics pipelines, where queries naturally span multiple recordings. A benchmark showing that the best commercial model scores around 50% signals that current products are not yet reliable for multi-video reasoning, which is directly useful for product planning, evaluation procurement, and prioritizing research investment.

Future Directions

• Frame selection over frame quantity. The ablation shows that adding frames helps up to a point and then hurts; smarter key-frame retrieval that filters irrelevant content is a clear next step.

• Architectures designed for cross-video fusion. Current failures often stem from comparing evidence across videos rather than understanding each video individually, suggesting that dedicated cross-video attention or memory mechanisms may be needed.

• Scaling and training for CVR. Since larger and thinking-enabled models do better, exploring fine-tuning or reinforcement learning on cross-video reasoning data could close part of the human-model gap.

• Extending task coverage and modalities. The current benchmark focuses on visual and caption-based reasoning; integrating audio, subtitles, and longer multi-hour contexts would test whether the observed limitations persist in richer settings.

Target Audience

Researchers and engineers working on multimodal LLMs, video understanding, and benchmark design; model developers who need a stress test for multi-video capabilities; and applied teams in media, robotics, or industrial inspection who are evaluating whether current video-capable models can handle tasks that require reasoning across several recordings rather than one.

Authors’ abstract

Cross-Video Reasoning (CVR) presents a significant challenge in video understanding, which requires simultaneous understanding of multiple videos to aggregate and compare information across groups of videos. Most existing video understanding benchmarks focus on single-video analysis, failing to assess the ability of multimodal large language models (MLLMs) to simultaneously reason over various videos. Recent benchmarks evaluate MLLMs' capabilities on multi-view videos that capture different perspectives of the same scene. However, their limited tasks hinder a thorough assessment of MLLMs in diverse real-world CVR scenarios. To this end, we introduce CrossVid, the first benchmark designed to comprehensively evaluate MLLMs' spatial-temporal reasoning ability in cross-video contexts. Firstly, CrossVid encompasses a wide spectrum of hierarchical tasks, comprising four high-level dimensions and ten specific tasks, thereby closely reflecting the complex and varied nature of real-world video understanding. Secondly, CrossVid provides 5,331 videos, along with 9,015 challenging question-answering pairs, spanning single-choice, multiple-choice, and open-ended question formats. Through extensive experiments on various open-source and closed-source MLLMs, we observe that Gemini-2.5-Pro performs best on CrossVid, achieving an average accuracy of 50.4%. Notably, our in-depth case study demonstrates that most current MLLMs struggle with CVR tasks, primarily due to their inability to integrate or compare evidence distributed across multiple videos for reasoning. These insights highlight the potential of CrossVid to guide future advancements in enhancing MLLMs' CVR capabilities.