MovieRecapsQA: A Multimodal Open-Ended Video Question-Answering Benchmark

AI summary

Overview

• Research area: Computer Vision / Video Question Answering (VideoQA), multimodal large language model (MLLM) evaluation, and benchmark construction.
• Technical level: Intermediate. The concepts (open-ended QA, LLM-as-a-judge, video-language models) are explained in the paper, but familiarity with VideoQA benchmarks and evaluation metrics helps.
• Scope: The paper introduces an open-ended, multimodal VideoQA benchmark built from movie recap videos, along with a reference-free, atomic-fact-based evaluation metric, and benchmarks seven MLLMs plus human participants against it.

What This Paper Is About

Understanding real-world video such as movies requires combining what is seen on screen with what is said in dialogue, but most existing VideoQA benchmarks avoid this by using multiple-choice questions, which allow shortcuts that do not require actually understanding the video. The core problem is that open-ended answers are hard to evaluate, since reference answers are not single words and word-overlap metrics correlate poorly with human judgment of factuality. The paper's goal is to build a benchmark of free-form, multimodal video questions whose answers can be verified against facts rather than against a single gold reference, and to use it to measure how well current models actually reason over long-form narrative video.

Key Contributions

1. MovieRecapsQA, a long-form multimodal VideoQA benchmark with approximately 8,200 open-ended questions (8,231 total) across 60 videos, labeled by the modality required to answer (dialogue-centric, vision-centric, multimodal) and by reasoning category.
2. The first reference-free open-ended VideoQA benchmark, to the authors' knowledge, using recap-derived atomic facts to ground an LLM judge in assessing factuality, relevance, and coherence without a unique reference answer.
3. Multiple input-video settings per question, enabling the same question to be evaluated with (a) the full-length movie, (b) the full approximately 11-minute recap video (visual only), (c) approximately 14 minutes of aligned movie scenes, and (d) approximately 1.2 minutes of aligned recap video scenes, in all cases with any associated movie dialogue text provided.
4. A benchmark of seven MLLMs and human annotators, revealing a substantial human-model gap, over-reliance on dialogue, and systematic failures on fine-grained visual perception.

Main Findings

• Semantic metrics do not discriminate between models. ROUGE-L ranged only 0.22 to 0.28, BERTScore 0.63 to 0.69, and BARTScore 0.03 to 0.05 across all models, with per-model variance of at most 0.03.
• Reference-based LLM judges misalign with human preference. HELMET Correctness rated MiniCPM-o (1.27) above the best human (1.26) and average human (0.98), and scored LLaVA-NeXT-Video (0.98) on par with the average human.
• The reference-free metric produces human-aligned separation. Factuality scores spanned 2.96 to 3.99 across models, compared with 4.01 for average human and 4.59 for best human, with higher per-model variance (1.97 to 2.35 on Factuality) than reference-based metrics (0.89 to 1.42).
• Vision-centric questions score lowest on Factuality. Proprietary models dropped from 3.63 (dialogue) to 3.15 (vision); open-source models from 3.21 to 3.05; humans from 4.17 to 3.84.
• Models stay on topic but get facts wrong. Relevance stayed stable across modalities for proprietary (3.84 to 3.63) and open-source (3.52 to 3.61) models, while humans scored above approximately 4.0, meaning models attend to the right scene but cannot extract the precise visual facts.
• Removing visual input improves proprietary model factuality. Dialogue-only input raised Factuality by +0.49 on dialogue questions, +0.73 on vision-centric questions, and +0.70 on multimodal questions, with the largest gains exactly where visual cues should matter most.
• Proprietary models lose their advantage with frames only. Under frames-only input on vision-centric questions, open-source models scored higher than proprietary ones (3.15 versus 3.06), and open-source models showed smaller drops, suggesting proprietary gains come from stronger language priors rather than better visual understanding.
• The bottleneck is visual perception, not reasoning type. Dialogue-rewarding categories were easiest (CRD 3.53, NPA 3.41 proprietary Factuality), while Setting and Technical Analysis (STA) was hardest (2.98 proprietary, 2.83 open-source) even though humans showed no such drop (4.15), making that gap 1.17 to 1.32 — roughly double that of CRD or NPA. On STA, dialogue-only input improved proprietary Factuality by +0.63 while frames-only declined by -0.19.

Methodology in Plain English

The researchers tapped movie recap videos — YouTube videos that narrate a full film in roughly 10 minutes using visual-only clips and a voiceover. This source is useful because the voiceover (RecapSummary) maps tightly onto specific moments of the movie, so narrated events can be linked to specific footage and dialogue.

Collection and alignment. They selected 60 films released between 1980 and 2024 that have publicly available scripts, searched the top 10 most popular recap channels via the YouTube API, took the top five results per movie, and manually filtered for correct movie-storyline alignment. They segmented both the full movie and the recap video into scenes using SceneDetect, embedded the first and last three seconds of each scene with SlowFast, and matched scenes by cosine similarity, then applied a lightweight statistical step to enforce semi-chronological ordering. This yielded alignment among the recap video, the full movie, and the movie subtitles.

Question generation. Each RecapSummary was split into segments, and an LLM (GPT-4.1) extracted atomic facts from each segment. A second prompt generated question-answer pairs grounded in one or more of those facts. Because the facts were verbose and produced overly easy questions, they also generated simplified questions that abstract away identifying details such as character names. The final set keeps the verbose answer paired with the simplified question. The result is 8,231 questions across 60 videos, with 16,462 aligned atomic facts.

Evaluation. Baseline metrics included semantic metrics (ROUGE, BERTScore, BARTScore) and reference-based LLM judges (G-Eval, HELMET Fluency and Correctness). The proposed reference-free metric extracts claims from a model's answer and prompts a judge (GPT-4.1 mini) to score Factuality and Relevance on a 0-5 scale against the question's atomic facts and the segment subtitles. Coherence was excluded from the main results because answers are too short to show internal contradictions.

Experiments. Seven MLLMs were run zero-shot with standardized prompting: proprietary GPT-4o, Gemini 2.5 Flash, Claude 3.5 Sonnet, and Amazon Nova Lite; open-source LLaVA-NeXT-Video, MiniCPM-o, and Qwen 2.5-VL. Video-capable models received the recap segment as an MP4 clip; frame-based models received uniformly sampled frames, with the number varying per model up to its context limit. Ablations compared full multimodal input, frames-only, and dialogue-only. A human study collected answers from five participants on 118 randomly sampled questions spanning all three modalities, scored with the same metrics.

Why This Matters

The paper argues that the metrics used to evaluate open-ended video QA shape what the field can see. Conventional semantic metrics and reference-based LLM judges do not reliably indicate factual correctness, whereas a fact-grounded, reference-free metric separates models and humans and exposes genuine weaknesses. The paper also argues that current MLLMs do not robustly integrate video and dialogue: removing frames can improve factuality, so progress on visual encoders alone is insufficient.

Potential real-world applications implied by the work:

• Video and media search or question answering, where a system must answer free-form questions about long content such as films, lectures, or recordings.
• Robot and embodied reasoning over visual and linguistic modalities, which the paper cites as a generalization beyond movie understanding.
• Accessibility tools, such as systems that describe or answer questions about narrative video content for users who cannot see it.
• Automated content verification and summarization, where fact-grounded checking against a textual representation is cheaper than verifying against raw video.

Industry relevance: the findings matter to teams building or evaluating multimodal assistants, since the paper shows that vision-centric performance is a weak point and that stronger language priors can mask it — a caveat for anyone reporting benchmark gains on video tasks. It also matters for evaluation infrastructure, because reference-free fact-based scoring avoids collecting gold answers and avoids the cost of using raw video as judge context. The authors release the dataset as derived metadata only (question-answer pairs, atomic facts, time stamps, YouTube identifiers, links to IMDb and OpenSubtitles), not the copyrighted video, audio, frames, or subtitle text, with an opt-out policy for recap creators.

Future Directions

• Architectures that explicitly ground answers in both video and dialogue, rather than letting subtitles serve as the primary source of truth.
• Training objectives that reward visual-textual consistency instead of mere fluency, given that models stay relevant but fail on visual facts.
• Fully exploring the benchmark's alternative input settings — the full-length movie, the full recap video, and the approximately 14-minute aligned movie scenes — since the reported model benchmark uses the shortest setting (approximately 1.2 minutes of aligned recap scenes).
• Extending the data pipeline to full movie scripts, which the authors note offer a richer textual source than subtitles, and which they intentionally enabled by selecting films with publicly available scripts.

Target Audience

Researchers and engineers working on video question answering, multimodal large language models, and long-form video understanding will benefit most, along with anyone designing benchmarks or evaluation metrics for open-ended generation. Evaluation practitioners and teams auditing claims about multimodal model capability will find the metric analysis directly useful, and the benchmark is also relevant to those studying video-grounded reasoning in adjacent areas such as robotics and media analysis.

Authors’ abstract

Understanding real-world videos such as movies requires integrating visual and dialogue cues. Yet existing VideoQA benchmarks struggle to capture this multimodal reasoning and, given the difficulty of evaluating free-form answers, largely resort to simple multiple choice questions. We introduce a novel open-ended multimodal VideoQA benchmark, MovieRecapsQA, created using movie recap videos -- a distinctive type of YouTube content that summarizes a film via a voiceover description of key clips from the movie (recap video). From the transcribed voiceover (recap summary) of 60 recap videos, we generate $\approx$8.2K questions along with the necessary ``facts'' expected in each answer; the former facilitates the creation of questions that require mutimodal reasoning and the latter allow the construction of a reference-free evaluation metric that can be applied to open-ended responses. To our knowledge, this is the first reference-free open-ended VideoQA benchmark. The benchmark allows each question to be evaluated in different input video settings: given (a) the full-length movie, (b) the full ($\approx$11 min) recap video (visual only), (c) $\approx$14 min of aligned movie scenes, i.e, movie scenes relevant to the question, and (d) $\approx$1.2 min of aligned recap video scenes. In all cases, the text of any associated movie dialogue is provided. Each question is categorized by the modality required to answer it -- visual, dialogue, or both -- enabling fine-grained evaluation of multimodal capabilities. We benchmark (setting (d)) seven state-of-the-art MLLMs and find that (i) only our reference-free metric produces meaningful human-aligned model separation; (ii) vision-centric questions yield the lowest scores across all models; (iii) removing visual input often \textit{improves} model factuality; and (iv) the primary bottleneck is visual perception, not visual reasoning.