MINERVA-Cultural: A Benchmark for Cultural and Multilingual Long Video Reasoning

AI summary

Overview

• Research area: Computer vision / multimodal video-language evaluation, specifically cultural and multilingual long-video reasoning benchmarks.
• Technical level: Intermediate. The paper is readable without deep technical background, but familiarity with Video-LLMs, benchmarks, and evaluation metrics helps.
• One-sentence scope: The paper introduces MINERVA-Cultural, a 2,400-question, 540-video, 18-locale benchmark of human-authored, native-language questions and reasoning traces over long culturally specific videos, and uses it to show that state-of-the-art Video-LLMs perform far below human level, with most failures traced to cultural visual perception.

What This Paper Is About

Existing long-video understanding benchmarks are predominantly Western-centric in their visual content and dominated by English, which biases how models are evaluated. The authors build a benchmark of long videos (one minute to over an hour) drawn from 18 global locales, where every question, answer, and multi-step reasoning trace is written in the locale's native language by locally situated experts rather than translated. The goal is to measure, and ultimately reduce, the gap between how well models handle culturally familiar content versus culturally distant content.

Key Contributions

1. MINERVA-Cultural benchmark: 2,400 questions across 540 videos spanning 18 global locales and their native languages, with open-ended questions requiring at least two reasoning skills (Temporal Ordering, Goal Reasoning, Event Occurrence, Reading, Listening, Spatial Perception, Temporal Event Localization, Counting, Cause and Effect, Numerical Reasoning, Object Recognition, Counterfactual Reasoning) plus a mandatory Visual Cultural Understanding skill. Each question comes with a human-authored reasoning trace in the native language.
2. A benchmarking and analysis suite for state-of-the-art models: evaluation of two open Video-LLMs (Qwen-2.5-VL, Qwen-3-VL) and five closed models (Claude-Sonnet-4, GPT-5-mini, GPT-5, Gemini-2.5-Flash, Gemini-2.5-Pro) against a human baseline, plus analyses of modality effects, frame count, and test-time compute.
3. A graph-based error diagnostics method: conversion of human reasoning traces into evidence-style Directed Acyclic Graphs and a three-stage "Iterative Error Isolation" loop that traverses the graph, tags errors, generates corrective hints, and re-evaluates until the question is solved, yielding fine-grained error attribution.
4. A human-centric curation pipeline: a four-stage process (culture-specific video selection, 10% sample calibration, final curation and audit, human evaluation) run by locally situated Curators and Auditors, with approximately five experts per locale on average, 50% of the total data reviewed during auditing, and 50% of questions independently answered by Auditors.

Main Findings

• Large human-model gap: Human performance is 95.22% aggregate, while the top model, Gemini-2.5-Pro, reaches 45.07%. Aggregate scores are 12.75 (QWEN-2.5-VL), 21.50 (QWEN-3-VL), 23.36 (Claude-Sonnet-4), 35.84 (Gemini-2.5-Flash), 36.64 (GPT-5-mini), 42.20 (GPT-5), and 45.07 (Gemini-2.5-Pro).
• Cultural disparity across locales: Models do best in locales such as Korean (ko-KR) and British English (en-GB). For South Indian languages, the top model's accuracy drops to 28.00% for Telugu (te-IN) and 31.60% for Tamil (ta-IN), which the authors present as evidence of Western and English-centric bias.
• Audio matters: Adding audio to video-only input yields a 4.32% average performance increase with Gemini-2.5-Pro, with larger gains for Chinese (zh-TW, +8.15%) and Indonesian (id-ID, +7.09%).
• Test-time compute saturates: Average performance across six locales rises from 35.9% at a 128-token thinking budget to a peak of 45.9% at 2,000 tokens, then plateaus, still far below human performance.
• More frames help but plateau: Accuracy increases monotonically as input frames scale from 1 to 512, confirming tasks need temporal reasoning, but gains diminish and a large human gap persists.
• Errors are mostly perceptual and cultural: Temporal Localization, Spatial Grounding, Spurious Object/Event, and Attribute Misidentification errors together account for approximately 75% of all failures; the authors call these "cultural visual perception" errors and find them more prevalent than reasoning errors.
• Low-resource languages suffer more perception errors: Low-resource locales (ar-EG, ta-IN) show 1.4 times more cultural visual perception errors than high-resource locales (en-GB, ja-JP, es-MX, ru-RU).
• Model profiles differ: Gemini-2.5-Pro commits around 7% fewer cultural perception errors than GPT-5, despite a slight tendency to make more Spurious Object/Event errors.
• Iteration reveals hidden errors: Running up to five iterations of Iterative Error Isolation solves 99.7% of questions and uncovers approximately 22% of Gemini-2.5-Pro's total errors beyond the first iteration, including 78 additional reasoning errors that earlier perception failures would have masked.
• Reasoning structure statistics: Questions require an average of 5.0 atomic evidences, over 63% of these are grounded in specific video timestamps, and graph depth statistics are mean 2.5 with standard deviation 1.3. True divergences from the human reasoning path are rare, occurring in 2% of questions.
• Diagnostic setup details: The error analysis covers questions scored 0 by the LLM Judge: 490 of 878 for Gemini-2.5-Pro and 524 of 878 for GPT-5, across six locales (es-MX, ja-JP, en-GB, ru-RU, ta-IN, ar-EG).

Methodology in Plain English

The authors first defined a cultural taxonomy per locale, expanding six broad domains (Sports, Cuisine, Festivals, Tourism, Rituals, Education) into locally recognized sub-categories. They then mined a large pool of videos and had Auditors manually screen them against a checklist: primarily native language, meaningful audio-visual content, authentic cultural scenes, longer than one minute, and complex enough to support at least 4-5 distinct multi-step reasoning questions. Local experts (Curators) wrote question-answer-reasoning triples in the native language, while separate Auditors checked that answers were objective and unambiguous by answering questions without seeing the ground truth; disagreements triggered revision until consensus, and questions without consensus were discarded. A calibration phase on a 10% sample standardized question difficulty and correctness before full-scale annotation.

For evaluation, because questions are open-ended, standard string matching was replaced with an LLM Judge (Gemini-2.5-Flash) scoring responses on a three-point scale (0, 1, 2) for semantic alignment with the ground truth. A separate pool of locally situated human evaluators, who never saw ground-truth answers or traces, provided the human baseline; they were allowed open-web search but prohibited from using LLMs. The error analysis converts each human reasoning trace into an evidence DAG with nodes as atomic evidences and edges as prerequisites, then iteratively traverses it, tags each missing evidence using a taxonomy of perception (Temporal Localization, Spatial Grounding, Attribute Misidentification, Spurious Objects/Events), knowledge, and reasoning errors, and re-queries the model with corrective hints until the chain is completed.

Why This Matters

Impact on research: The benchmark makes a previously under-measured problem measurable: that model accuracy is unevenly distributed across languages and cultures, and that the dominant failure mode on culturally situated video is perception of cultural elements rather than logic. The evidence-graph plus Iterative Error Isolation pipeline reframes evaluation as an evidence classification task and offers step-level diagnosis rather than a single accuracy number.

Real-world applications:

• Multilingual video search and recommendation systems serving non-English and non-Western markets.
• Assistive or accessibility tools that must interpret culturally specific audio-visual content, such as local festivals, cuisine, or rituals.
• Content moderation and metadata generation for regionally specific video platforms.
• Cross-cultural training or educational tools that need to interpret locally situated video material correctly.

Industry relevance: The authors are at Google DeepMind (with one co-author at UC Berkeley), and the evaluated models are frontier commercial systems from Google, OpenAI, Qwen, and Anthropic. The results identify concrete weaknesses in current products across 18 markets and provide a diagnostic tool for targeting them.

Future Directions

• Extending beyond 18 locales. The authors state their locale set is not exhaustive and that the methodology is designed to be extensible; expanding coverage, especially for underrepresented languages, is a natural next step.
• Disentangling cultural from general perception errors. The paper acknowledges that some errors may stem from general visual perception limits rather than cultural gaps, and calls disentangling the two an open challenge.
• More robust error diagnostics. The diagnostic pipeline relies on a prompted LLM (Gemini-2.5-Pro) for classification and hint generation; the authors flag potential model bias and name more robust LLM-based diagnostic tools or human-in-the-loop verification of LLM-judged errors as important future work.
• Process-oriented, interpretable models. The detailed human reasoning traces are presented as an opportunity to build models that are evaluated and trained on the reasoning process, not just the final answer.

Target Audience

Researchers and engineers working on video-language models, multimodal reasoning, and benchmark design will get the most from this paper, particularly those concerned with evaluation fairness across languages and cultures. It is also relevant to product teams deploying video understanding in non-English markets, and to researchers studying chain-of-thought or multi-step reasoning evaluation, since the evidence-graph error isolation method generalizes beyond this dataset.

Authors’ abstract

Recent advancements in video models have shown tremendous progress, particularly in long video understanding. However, current benchmarks predominantly feature western-centric data and English as the dominant language, introducing significant biases in evaluation. To address this, we introduce MINERVA-Cultural, a challenging benchmark for multicultural and multilingual video reasoning. MINERVA-Cultural comprises high-quality, entirely human-generated annotations from diverse, region-specific cultural videos across 18 global locales. Unlike prior work that relies on automatic translations, MINERVA-Cultural provides complex questions, answers, and multi-step reasoning steps, all crafted in native languages. Making progress on MINERVA-Cultural requires a deeply situated understanding of visual cultural context. Furthermore, we leverage MINERVA-Cultural's reasoning traces to construct evidence-based graphs and propose a novel iterative strategy using these graphs to identify fine-grained errors in reasoning. Our evaluations reveal that SoTA Video-LLMs struggle significantly, performing substantially below human-level accuracy, with errors primarily stemming from the visual perception of cultural elements. MINERVA-Cultural will be publicly available under https://github.com/google-deepmind/neptune?tab=readme-ov-file\#minerva-cultural