DramaVQA: Probing Detailed Understanding of Ultra-Long TV Dramas

AI summary

Overview

Research area: Computer Vision / Multimodal Large Language Models (MLLMs) — specifically long-form and ultra-long video question answering, plus benchmark construction.

Technical level: Advanced.

Scope: This paper introduces DramaVQA, a ~20K-question benchmark built from multi-episode TV dramas (average full-drama length 38.9 hours) designed to test whether multimodal models can answer detailed questions about videos lasting tens of hours.

What This Paper Is About

Existing video benchmarks mostly cover short clips or hour-scale videos, so there is no rigorous way to measure whether multimodal models can actually understand content that runs for tens of hours, such as a full television series. The authors build DramaVQA from 12 long-form TV dramas and pair it with a scalable, LLM-assisted but human-verified annotation pipeline, then test a wide range of proprietary, open-source, and agentic models at four different video-length settings. The goal is to establish a hard, reproducible testbed that exposes where current models break down when context must span dozens of episodes.

Key Contributions

1. A benchmark an order of magnitude larger than prior long-video datasets. DramaVQA contains approximately 20,000 QA pairs over 12 TV dramas (466 hours of the 606 total hours collected; the other three series are reserved for a training split), compared with 900 QAs in Video-MME-L and 1,549 in LVBench, and averages 1,666.7 questions per video.

2. A scalable LLM-assisted annotation pipeline with strict human checking. Structured metadata (subtitles via PaddleOCR plus Qwen3-VL-32B, a character/face library, speaker labels, and six-level hierarchical captions from 10–15s L1 units up to a single L6 full-drama summary) feeds LLM QA generation, followed by rule-based filtering, two-pass human checking, text-only filtering with GPT-5, Gemini-2.5-Pro, and Seed1.8 Pro, and a global uniqueness check using bge-large-en/zh-v1.5 embeddings plus BM25.

3. A multi-level temporal structure enabling four evaluation tracks. Because every retained question is verified unique across the whole drama, the same question can be evaluated on Medium (averaging 7.6 minutes), Long (43.1 minutes), Extra-Long (9.3 hours), and Ultra-Long (the full drama, averaging 38.9 hours) inputs.

4. A large auxiliary corpus and a reinforced training recipe. The authors plan to release over 400K human-validated QA pairs with precise timestamps that failed only the uniqueness test, and show that reinforcement learning on their training split improves LongVideo-R1 by at least 3.6% on all four tracks.

Main Findings

• Ultra-long questions collapse to near-chance accuracy. Random guessing scores 16.2% on all tracks, while Qwen3.6-27B reaches 70.8% full-set on Medium with 512 frames but only 35.1% on Ultra-Long. Gemini-3.5-Flash scores 77.5 / 61.2 / 43.9 / 37.1 on Medium / Long / Extra-Long / Ultra-Long on the mini set.

• Accuracy degrades monotonically with input length. Every tested model drops sharply from Medium to Long, and lands close to the random baseline on Extra-Long and Ultra-Long. Human performance on Medium is 81.0%.

• Even the easiest track is unsolved. The best full-set Medium score is 70.8% (Qwen3.6-27B, 512 frames), well below the 81.0% human reference, which the authors attribute to dense captions enabling hard questions, 6–8 competitive options (range 2–8, average 6+), and removal of language-prior-solvable questions.

• Text-only models confirm language priors were stripped. A language-only Qwen3-VL-32B baseline scores 21.5% [22.2%] and Qwen3.5-27B 23.1% [23.3%], close to the 16.2% random baseline.

• Stronger base models help, but gains shrink with length. At 512 frames without thinking, Medium rises from 42.5% (Qwen3-VL-8B) to 53.3% (Qwen3-VL-32B) to 70.8% (Qwen3.6-27B), while Ultra-Long only moves 24.2% → 26.2% → 35.1%.

• Thinking helps short tracks far more than long ones. Qwen3-VL-8B with 512 frames improves from 42.5% to 58.9% on Medium and 31.1% to 41.0% on Long, but only from 24.2% to 25.2% on Ultra-Long.

• Retrieval and agentic methods buy accuracy with large compute cost. Naive-RAG (Qwen3.8) scores 60.1 / 56.7 / 36.9 / 31.5 on the mini set, but mini inference time rises to 293.0 / 1,664.6 / 1,955.5 / 1,955.0 seconds as the video grows, versus roughly 12–40 seconds for native MLLMs.

• Training on the DramaVQA data helps agentic models. LongVideo-R1 post-tuned on the training split improves by at least 3.6% on all four tracks and raises average medium-caption calls from 2.38 to 4.65, low-caption calls from 3.12 to 4.66, and reasoning steps per question from 7.41 to 10.94.

• Compositional reasoning is a separate, severe weakness. On 1,510 multi-point and 1,200 multi-hop questions (256 frames), Qwen3-VL-32B and GLM-4.6V-106B sit at or below chance on TTC-2 (46.4% and 47.4% versus 50.0% random) and CECI (43.4% and 48.2% versus 50.0% random); the best CECI score is 59.0% (Qwen3.6-27B), and all models do worse on multi-hop questions than on their paired single second-hop questions.

• Three diagnosed failure modes. Sparse frame sampling misses keyframes, models fail to track story logic and cause-and-effect, and the absence of audio makes "who said what" questions hard.

• Category difficulty is uneven. Temporal understanding and counting are harder than visual perception and event understanding, out of seven top-level categories and 28 sub-categories.

Methodology in Plain English

The authors first clean each drama by removing opening titles and credits so episodes form one continuous stream. They then extract structured metadata: subtitles read off the screen with OCR and cleaned up by a vision-language model, a character library built from public cast photos and refined by scanning video at 1 FPS (with filtering by similarity threshold 0.8 and quality threshold 0.45), and speaker labels obtained by clustering voice embeddings with the ERes2Net model and then having humans check every line. On top of this they build a six-level caption tree, starting from 10–15 second scene units detected with PySceneDetect and CLIP similarity, and repeatedly clustering and summarizing upward until one summary describes the entire drama.

Questions are then generated in a funnel. An LLM writes candidate single-choice questions from roughly 1.2-minute L2 captions across seven question types; a filter removes self-answering or clip-relative questions; two LLMs try to answer from text alone and any question they get right is dropped; human labelers watch the corresponding video and check or fix each item in two passes; three stronger LLMs (GPT-5, Gemini-2.5-Pro, Seed1.8 Pro) must all answer incorrectly from text alone; and finally a global search over the whole drama removes any question that could be answered from elsewhere. The funnel shrinks the pool dramatically — from 103.1 candidate QAs per L2 segment down to 27.3, 18.8, and 5.4 after successive steps, ending at 1.7 per segment for the benchmark.

Evaluation keeps the original on-screen subtitles so models must read text from pixels, draws red boxes and name labels on detected faces, and requires a single option as output, with up to five attempts and a 1/X credit for no valid answer. Models see 256 or 512 uniformly sampled frames, and a mini subset of about 5% of the test set is used for proprietary models.

Why This Matters

Research impact. DramaVQA reframes long-video evaluation: instead of asking whether a model can squeeze more frames into a context window, it asks whether the model can locate and reason over evidence scattered across 38.9 hours. The near-chance results on the longest tracks, plus the separate multi-hop analysis, give the community concrete evidence that context scaling alone is not the answer and that memory-based or agentic designs are needed.

Real-world applications:

• Streaming and broadcast platforms that want automatic episode recaps, "previously on" generation, or spoiler-aware search across entire seasons.
• Content moderation and compliance teams that must track a character, claim, or event across a full catalogue rather than a single clip.
• Post-production and dubbing workflows, where editors need to retrieve exact moments and verify who said what across many episodes.
• Accessibility tooling, such as audio description or subtitle generation that requires knowing which character is on screen and what happened earlier.

Industry relevance. The paper quantifies the accuracy-efficiency trade-off directly, showing that retrieval and agentic pipelines gain accuracy but can cost thousands of seconds per question unless captioning is shared across many questions from the same video — a practical constraint for any deployed video assistant. The planned release of 400K+ timestamped QA pairs also gives industry teams a supervised fine-tuning and RL resource, and the authors demonstrate that value by improving LongVideo-R1 on all four tracks.

Future Directions

• Memory-centric architectures. The authors argue for moving beyond context-window expansion toward compact, searchable memory formats and "video-driven world models"; comparing such designs against the RAG and agentic baselines here is the obvious next experiment.
• Tri-modal audio-visual-text models. The lack of audio input is identified as a direct cause of errors on dialogue-based questions, so adding speech is a clear extension.
• Better evidence seeking under a compute budget. Agentic models still fail to locate the needed segment, and naive RAG is extremely slow, leaving open how to schedule retrieval and captioning cheaply at batch scale.
• Compositional and multi-hop reasoning. With CECI peaking at 59.0% and every model underperforming on multi-hop versus single-hop questions, extending the 1,510 multi-point and 1,200 multi-hop questions into a full-fledged evaluation axis is an open problem.

Target Audience

Researchers and engineers working on multimodal LLMs and video-language understanding, benchmark and dataset builders who need to know how annotation pipelines are designed and quality-controlled at this scale, and agentic-AI or retrieval-system developers interested in long-term memory and accuracy-efficiency trade-offs. Product and R&D teams in streaming, media production, or video search will also find the baseline numbers and cost measurements directly useful, while newcomers to the field can use the paper's four-track structure and category breakdown as a survey of where current video models stand.

Authors’ abstract

Despite recent progress in long-form video understanding, current benchmarks mostly focus on short clips or hour-scale videos. This leaves a major gap in evaluating ultra-long video understanding, where videos often last tens of hours. To bridge this gap, we introduce DramaVQA, a large-scale benchmark built upon long-form TV dramas. Containing approximately 20K question-answering (QA) pairs, the scale of DramaVQA exceeds previous long-video datasets, such as Video-MME and LVBench, by an order of magnitude. Our benchmark has two main features that make it stand out from existing datasets. First, we establish a scalable, LLM-assisted annotation pipeline that uses multimodal large language models (MLLMs) to generate complex questions, followed by strict human checking to ensure high data quality. Second, DramaVQA introduces a multi-level time structure; queries are designed across different video lengths, with the hardest questions requiring overall understanding across the entire drama. Extensive evaluations show that state-of-the-art MLLMs face major challenges when tested on these very long videos, highlighting the need for agentic AI algorithms capable of managing long-term memory. The project is available at https://github.com/xinyuehuo/DramaVQA