DataVista: Diagnosing Multimodal LLMs on Data Video Understanding

AI summary

Overview

• Research area: Multimodal large language model (MLLM) evaluation; data visualization understanding; video understanding benchmarks.
• Technical level: Intermediate — assumes familiarity with multimodal benchmarking and chart question answering, but the framing is accessible.
• Scope: The paper introduces DataVista, the first benchmark for "data video understanding," and reports a systematic diagnostic evaluation of 19 MLLMs against a human expert baseline across three progressive capability levels.

What This Paper Is About

Data videos — videos that combine animated charts with narration and scene transitions, as produced by outlets like Bloomberg Graphics, NYT The Upshot, and Vox — require viewers to read numbers off moving charts, connect evidence across different charts and moments in time, and understand how narrative and visual design build an argument. Existing benchmarks cover either general videos (Video-MME, MVBench) or static charts (ChartQA, VisJudge-Bench), so this specific combination had not been systematically measured. The authors build DataVista to diagnose where multimodal models succeed and fail on this task, and to establish how far current systems are from human expert performance.

Key Contributions

1. A new benchmark: DataVista contains 961 real-world data videos collected from YouTube and 6,775 multiple-choice evaluation questions (both single-select and multi-select), spanning five topic domains (Economy, Society, Science, Politics, Culture) and 22 subcategories.
2. A three-level capability framework: Questions are organized into a progressive hierarchy — data perception (Level 1), temporal reasoning (Level 2), and narrative understanding (Level 3) — subdivided into 10 fine-grained question types, each with its own distractor design principles.
3. A human-AI collaborative annotation pipeline with quality control: Gemini-3-Flash is used for pre-screening, structured pre-extraction, and text-only filtering, while five annotators write questions and four independent reviewers verify them; 80.1% of candidate questions pass review unchanged, 15.4% pass after revision, and 4.5% are rejected.
4. A diagnostic evaluation of 19 MLLMs: Five closed-source and 14 open-source models are compared across capability levels, input conditions (frames, subtitles, native video with audio), frame budgets, and manually categorized error modes.

Main Findings

• A large human–model gap remains. Human experts reach 91.7% overall accuracy (97.1% on Level 1, 89.0% on Level 2, 86.8% on Level 3). The best model, Gemini-3.1-Pro, reaches 70.0% overall — 21.7 percentage points below humans. GPT-5 reaches 67.4%, Gemini-3-Flash 66.3%, and Claude-Sonnet-4.6 63.6%; GPT-4o reaches 57.1%.

• Open-source models trail but are competitive at the top. The strongest open-source model, Qwen3-VL-32B-Instruct, reaches 57.3%, 12.7 points behind Gemini-3.1-Pro. Its Thinking mode raises overall accuracy to 62.9%, with gains concentrated in temporal reasoning rather than narrative understanding. The weakest evaluated model, InternVL3-2B, reaches 31.1%.

• Temporal reasoning is the most discriminative level. Gemini-3.1-Pro leads Qwen3-VL-32B by 25.3 percentage points on Level 2, versus 10.3 points on Level 1 and 4.2 points on Level 3. The gap between models and humans widens with level: the best model hits 81.8% on Level 1 (15.3 points below humans) but only 59.2% on Level 3 (27.6 points below).

• Causal reasoning and narrative structure are the weakest question types. With frames and subtitles, the five representative models average 73.6% on Cross-Temporal Change and 62.6% on Cross-Chart Comparison, but only 39.2% on Causal Reasoning and 53.9% on Argument Synthesis; even the best model reaches only 44.2% on Causal Reasoning. On Narrative Structure, all five models fall between 46.9% and 51.0%, while their Cross-Temporal Change scores range from 56.0% to 87.8%.

• Frames mostly help basic reading, not narrative understanding. Adding frames to question-only input raises mean Level 1 accuracy across five models from 39.4% to 71.4% (+32.0 points), compared with +14.7 on Level 2 and +5.1 on Level 3 (48.0% to 53.1%). Frames alone beat questions-with-subtitles (46.7%) by 24.7 points on Level 1.

• Subtitles help most where cross-chart evidence must be linked. Adding subtitles to frames raises mean Level 2 accuracy from 52.6% to 59.1% (+6.5 points), versus +2.9 on Level 1 and +2.4 on Level 3. Across all 19 models, subtitles raise overall accuracy by a median of 3.1 percentage points, with the largest gain of 5.1 points for Qwen3-VL-32B. For GPT-5, subtitles raise accuracy on videos longer than 9 minutes from 59.4% to 66.9%, shrinking the gap to the shortest-video group from 8.9 to 1.4 points.

• More frames have diminishing returns above the perception level. Raising the frame budget from 8 to 32 raises mean Level 1 accuracy by 11.6 percentage points, versus 4.7 on Level 2 and 2.1 on Level 3. At 100 frames, Level 1 can still improve, while Levels 2 and 3 show no consistent gain. The leading model also changes with budget: GPT-5 leads at 8 and 16 frames, Gemini-3.1-Pro at 32 and 50.

• Native video with audio does not close the gap. Given complete videos with original audio, Gemini-3.1-Pro scores 73.5% and Gemini-3-Flash 68.9% overall; on Level 3 they reach 59.4% and 55.9% against a human score of 86.8%.

• Errors cluster in evidence judgment, not arithmetic. Of 120 manually inspected incorrect Gemini-3.1-Pro responses (50 frames, with subtitles), 47.5% were perceptual (36.7% value/text reading, 5.0% confused visual mappings, 3.3% reversed trends or rankings, 2.5% chart-structure errors), 42.5% were reasoning errors (37.5% evidence judgment, 4.2% cross-chart linking, 0.8% pure arithmetic), and 10.0% were instruction-understanding errors (5.0% shifting the specified time range or target object, 3.3% mapping a reasonable conclusion to the wrong option, 1.7% overlooking an evidence-source restriction).

• Partial credit reveals incompletely justified answers. On Argument Synthesis, models reach 88.0% option-level F1 but only 53.9% exact match, indicating many responses select some but not all of the supporting evidence.

Methodology in Plain English

The authors started by gathering 5,604 candidate videos from YouTube via keyword search and channel-level scraping across media, data-journalism, educational, and independent-creator channels. A model (Gemini-3-Flash) pre-screened candidates for three conditions: the core information is genuinely data-driven rather than narration-only, at least one clearly visible and sustained data visualization appears, and the video has a real communicative intent (excluding pure chart displays, software tutorials, and tool demos). Human annotators then confirmed or corrected each classification, leaving 967 videos that passed screening.

For question design, the authors borrowed the structure of Bloom's cognitive taxonomy, visualization literacy frameworks, and data-storytelling design spaces to define three levels. Level 1 covers reading values and recognizing chart elements. Level 2 covers tracking change over time, comparing across charts (including arithmetic derivation), identifying causal chains, and picking out the evidence supporting the central argument. Level 3 covers narrative structure, visual communication intent, counterfactual reasoning about design choices, and holistic quality judgment along Fidelity–Expressiveness–Aesthetics dimensions. Each question type has explicit rules for distractors — for example, Level 1 numeric questions must include a distractor within ±15% of the correct answer, and Level 2 cross-chart questions must include at least one option that would be correct if only a single chart were consulted, blocking that shortcut.

Five annotators with data-video or visualization research experience wrote the questions (roughly 1–2 hours per video), assisted by model-extracted chart segments, key data points, narrative stages, and argument chapters that they verified segment by segment. To prevent models from answering without watching, a text-only filter removed any candidate whose answer the model could predict from the question stem and options alone; retained questions were 22.9–32.2 percentage points harder under this test across five model families. Four separate reviewers then checked wording, video evidence, and distractors, with disagreements resolved against the source video.

For evaluation, all models received the question, the options, and 50 uniformly sampled frames, without audio, in two matched conditions (with and without subtitles). Scoring is exact match; multi-select answers must match the ground-truth option set exactly. Six human evaluators watched full videos with audio and answered independently, two per question, achieving 88.6% exact agreement overall (95.5%, 85.7%, and 81.3% across the three levels).

Why This Matters

Impact on research. The paper separates capabilities that prior benchmarks conflate: reading a chart, integrating evidence across charts and time, and understanding narrative intent. Its finding that more frames and more subtitles barely move Level 3 accuracy is a concrete signal that scaling visual input is not the bottleneck for narrative understanding — a useful constraint for anyone designing the next generation of video-language models or training data.

Real-world applications:

• Data journalism QA: Newsrooms producing chart-driven video (the kind made by Bloomberg Graphics, NYT The Upshot, and Vox) could use such evaluations to check whether automated tools correctly interpret their own published work.
• Automated accessibility and description: Generating accurate audio descriptions or summaries of data videos requires exactly the skills DataVista tests — reading values off animated charts and knowing which evidence supports the video's claim.
• Business analytics and reporting: Automated briefing or report-generation systems that consume chart-heavy recorded presentations need cross-chart comparison and evidence judgment, the two areas where models scored lowest.
• Content moderation and claim verification: Detecting whether a video's charts actually support its stated argument maps directly onto the Argument Synthesis and Causal Reasoning tasks, where accuracy is 53.9% and 39.2%.

Industry relevance. Organizations building multimodal agents over slide decks, dashboards, recorded meetings, or financial broadcasts can use DataVista as a diagnostic to locate failure modes before deployment. The error taxonomy — particularly the 37.5% evidence-judgment share — tells vendors that their models often read charts correctly but misjudge whether a fact supports a claim, which is the more consequential failure in decision-support settings.

Future Directions

• Closing the evidence-judgment gap. The dominant reasoning error is not arithmetic (0.8%) but judging whether a correctly read fact supports a specified claim (37.5%). Training or prompting approaches that explicitly ground claims in chart evidence are an open problem.
• Rethinking narrative understanding. Since frame counts and subtitles yield only +5.1 and +2.4 points on Level 3, and native video with audio still leaves Gemini-3.1-Pro at 59.4% versus 86.8% for humans, new architectures or representations for narrative structure — not larger visual budgets — appear necessary.
• Extending coverage and reducing contamination risk. The benchmark draws from YouTube with roughly 68.5% of videos published in or after June 2025. Broader language and source coverage, and continued decontamination as models train on more recent web video, remain ongoing concerns.
• Operationalizing evaluation. The authors state they will deploy an official automated evaluation service and public leaderboard upon publication, which would allow standardized tracking of progress and comparison of new models under identical input settings.

Target Audience

Researchers and engineers working on multimodal LLMs, video-language understanding, and chart/visualization question answering will get the most from this paper, particularly those who need a diagnostic rather than a single aggregate score. It is also directly relevant to visualization researchers interested in how data storytelling is comprehended, to data-journalism practitioners evaluating automated tools, and to product teams building agents that must reason over chart-heavy recorded content. Readers new to multimodal benchmarking will find the three-level framework and the distractor design principles a useful conceptual map, though the experimental sections assume comfort with exact-match metrics and ablation-style input conditions.

Authors’ abstract

Data video is a media form that integrates data visualization with video narrative, widely adopted in news reporting and business analysis. Compared with general video understanding, data video understanding places greater emphasis on accurately reading data from animated charts, integrating evidence across charts and time, and understanding how narrative organization and visual design communicate information. Yet existing benchmarks target either general videos or static charts, and data video understanding has not been systematically evaluated. We present DataVista, the first benchmark for data video understanding, containing 961 real-world data videos and 6,775 evaluation questions organized under a three-level progressive capability framework (data perception, temporal reasoning, narrative understanding) with 10 fine-grained question types across five topic domains. Systematic evaluation of 19 mainstream MLLMs shows that the best-performing model, Gemini-3.1-Pro, achieves 70.0% overall accuracy, still far below human expert performance, with models performing worst on Causal Reasoning and Narrative Structure. Increasing frame counts and adding subtitles mainly benefit data perception and temporal reasoning, with limited gains in narrative understanding. Further analysis of model responses identifies typical failure modes in chart reading, evidence judgment, and instruction understanding. The benchmark is available at https://github.com/HKUSTDial/DataVista.