ExpVid: A Benchmark for Experiment Video Understanding & Reasoning

AI summary

Overview

Research area: Computer Vision / Multimodal Large Language Models (MLLMs) applied to scientific video understanding; AI for science.

Technical level: Intermediate.

Scope: A publicly released benchmark (arXiv:2510.11606v1, 13 Oct 2025) of 7,800 question–answer pairs over 390 peer-reviewed laboratory experiment videos that tests 19 MLLMs across fine-grained perception, procedural understanding, and scientific reasoning.

What This Paper Is About

Multimodal large language models are increasingly proposed as aids for scientific work, but nobody has systematically checked whether they can actually follow real laboratory footage. Existing video and science benchmarks focus on general actions, narratives, or static outcomes (such as medical images) rather than the fine-grained, long-horizon manipulations of wet-lab experiments. The authors build ExpVid to measure that gap directly.

Key Contributions

1. A new benchmark. ExpVid is presented as the first benchmark that systematically evaluates MLLMs on scientific experiment videos across three hierarchical levels: fine-grained perception, procedural understanding, and scientific reasoning.
2. A scalable annotation pipeline. The authors design a vision-centric annotation method that builds multi-level tasks from videos, ASR transcripts, and paired peer-reviewed papers, followed by multi-disciplinary PhD-level expert validation and refinement.
3. An empirical study of 19 MLLMs. The paper benchmarks 19 leading open-source and proprietary models on ExpVid and reports per-task, per-level analysis of where they succeed and fail.
4. Diagnostic findings as a roadmap. The results identify specific weaknesses — disambiguating fine details, tracking states and quantities over time, and linking procedures to scientific conclusions — that the authors frame as directions for building trustworthy laboratory AI assistants.

Main Findings

• Benchmark scale: ExpVid contains 7,800 QA pairs over 390 videos spanning 10 task types, with an average video duration of 489.0 seconds. Videos average 489s (median 505s) in the source collection, with long-tail outliers beyond 2,000s.
• Task hierarchy: Level-1 covers four recognition tasks (Tool, Material, Quantity, Operation); Level-2 covers four procedural tasks (Step Ordering, Sequence Generation, Completeness Verification, Step Prediction); Level-3 covers two fill-in-the-blank tasks (Experimental Analysis, Scientific Discovery).
• Frontier models beat the non-expert human baseline. On Level-1, Gemini-2.5-Flash with thinking reaches 60.2 versus a human average of 37.6. On Level-2, GPT-5 scores 57.5 versus a human average of 42.1.
• Closed-source models lead, and the gap widens with complexity. Best Level-1 averages: Gemini-2.5-Flash (thinking) 60.2, Gemini-2.5-Pro 59.2, versus best open-source InternVL3-78B at 50.9 and Intern-S1 at 49.9.
• The gap is largest in high-order reasoning. On Level-3, GPT-5 leads at 56.4 (Experimental Analysis 55.4, Scientific Discovery 57.4), while the best open-source model, Intern-S1, reaches 39.6 — nearly 17 points short.
• Open-source models can match on structured, but not generative/predictive, tasks. InternVL3-78B scores 87.1 on Step Ordering, above GPT-5's 85.1, but falls behind on Sequence Generation (45.5 vs. 66.9) and Step Prediction (15.5 vs. 51.8).
• Uneven capabilities within a single model. Qwen2.5-VL-72B-Instruct scores 86.3 on Step Ordering yet only 0.3 on Step Prediction.
• Scaling helps consistently for open-source models. The InternVL family improves across all three levels from InternVL3-8B (L1 39.4, L2 23.9, L3 27.2) to InternVL3.5-38B (44.0, 36.0, 31.9) to InternVL3-78B (50.9, 41.9, 37.7).
• "Thinking" is not consistently better. Reasoning mode sometimes degrades results; in analyzed error cases, Gemini-2.5-Flash with Thinking_Budget=8,192 drifted toward plausible but non-visual workflows, while the NoThinking mode stayed video-grounded.
• Vision is essential to the design. Removing frames consistently hurts Level-1 and Level-2 performance, making some tasks (such as Step Prediction) unsolvable. For Level-3, accuracy is near zero without frames and rises as frames are added; InternVL3.5 peaks around 128 frames then declines, while MiMo-VL and Kimi-VL keep improving up to 256 frames.
• No human baseline is reported for Level-3. The 15 undergraduate participants without specialized biomedical backgrounds reported being unable to complete the open-ended cloze questions.

Methodology in Plain English

The authors sourced videos, ASR transcripts, and matching peer-reviewed papers from JoVE, a peer-reviewed video journal. Transcripts were scored by DeepSeek-R1 on five criteria (Continuity, Alignment, Clarity, Integrity, Focus) on a 0–5 scale; only videos scoring at least 4 overall with no dimension below 3.5 were kept, producing 5,879 videos (37.2%). Durations were restricted to the interquartile range of 378–728s, and experts selected 30 top-ranked experiments from each of 13 disciplines, yielding 390 videos with ASR transcripts averaging 1,026 words.

Videos were then split into three temporal levels: roughly 10k action-level clip–text pairs averaging about 8 seconds, roughly 3.5k stage-level segment–text pairs averaging about 48 seconds, and full experiment videos averaging about 8 minutes (in some cases concluding slides, figures, and data-analysis segments were removed to block shortcuts).

Questions were built semi-automatically to be vision-centric: LLMs extract candidate entities and actions from transcripts but contextual narration that could reveal the answer is deliberately excluded, and distractors are designed to be visually or semantically plausible. PhD-level experts in biology, medicine, chemistry, and related fields reviewed, corrected, and rebalanced every item on a custom online annotation platform. Annotation required an average of 0.3 hours per Level-1 question, 0.5 hours per Level-2 question, and 1.2 hours per Level-3 question.

Evaluation used Top-1 accuracy for Level-1 and for the Level-2 multiple-choice tasks, Jaccard similarity for Sequence Generation, and per-blank accuracy for Level-3 (judged with the lightweight model Phi-3-mini). Inference settings included 8 frames for Level-1, 32 frames for Level-2, uniformly sampled frames resized to 224×224, a maximum of 8192 tokens, and temperature fixed at 0.1.

Why This Matters

Experimental science depends on precise, stepwise procedures, and the paper shows that today's best models are far better at coarse recognition than at tracking fine detail and connecting procedures to conclusions. This matters for anyone planning to deploy MLLMs in a lab, since it identifies specific capabilities that are not yet reliable.

Real-world applications:

• Lab training and onboarding: Models could help newcomers follow protocols, if fine-grained tool, material, and quantity recognition improves.
• Protocol verification and audit: Automatically checking whether a recorded experiment matches its written protocol, a task ExpVid's Completeness Verification explicitly probes.
• Reproducibility and peer review: Linking video evidence to published claims, which is exactly the Level-3 Scientific Discovery task.
• Autonomous or assistive lab automation: Robotic and agentic systems that need to perceive state changes across multi-step workflows.

Industry relevance: The results give model developers a diagnostic target — the authors report a persistent gap between proprietary and open-source models that widens with reasoning complexity, and show that scaling helps within the InternVL family. The paper also reports that vision-centric training data appears to matter: models trained with long-video data (Kimi-VL, MiMo-VL) kept improving up to 256 frames, while InternVL3.5 saturated around 128.

Future Directions

• Broaden the science coverage. ExpVid currently focuses on wet-lab experiments and excludes physics, purely computational work, and large-scale engineering tests, which the authors list as underexplored.
• Evaluate reasoning processes, not just outcomes. Level-3 measures answers but does not illuminate the chain-of-thought linking experiments to conclusions.
• Capture missing human baselines. Only non-expert human performance is reported (15 undergraduates), and no human baseline exists for Level-3; expert baselines would clarify the remaining headroom.
• Improve generative and predictive procedural reasoning. Step Prediction and Completeness Verification are the weakest tasks across nearly all models, including cases such as Qwen2.5-VL-72B-Instruct's 0.3 on Step Prediction.
• Reconcile thinking modes with visual grounding. The paper's error analysis suggests reasoning mode can drift from video evidence, raising the question of how to keep step-by-step reasoning anchored to what is actually shown.

Target Audience

Researchers working on multimodal large language models and long-video understanding; benchmark designers in AI evaluation; AI-for-science and laboratory-automation researchers; and practitioners evaluating whether current MLLMs are reliable enough for scientific or procedural video analysis. Readers looking for a dataset to build on rather than a new model architecture will find the code and benchmark at https://github.com/OpenGVLab/ExpVid, released under CC BY-NC-SA 4.0.

Authors’ abstract

Multimodal Large Language Models (MLLMs) hold promise for accelerating scientific discovery by interpreting complex experimental procedures. However, their true capabilities are poorly understood, as existing benchmarks neglect the fine-grained and long-horizon nature of authentic laboratory work, especially in wet-lab settings. To bridge this gap, we introduce ExpVid, the first benchmark designed to systematically evaluate MLLMs on scientific experiment videos. Curated from peer-reviewed video publications, ExpVid features a new three-level task hierarchy that mirrors the scientific process: (1) Fine-grained Perception of tools, materials, and actions; (2) Procedural Understanding of step order and completeness; and (3) Scientific Reasoning that connects the full experiment to its published conclusions. Our vision-centric annotation pipeline, combining automated generation with multi-disciplinary expert validation, ensures that tasks require visual grounding. We evaluate 19 leading MLLMs on ExpVid and find that while they excel at coarse-grained recognition, they struggle with disambiguating fine details, tracking state changes over time, and linking experimental procedures to scientific outcomes. Our results reveal a notable performance gap between proprietary and open-source models, particularly in high-order reasoning. ExpVid not only provides a diagnostic tool but also charts a roadmap for developing MLLMs capable of becoming trustworthy partners in scientific experimentation.