Sci-VBench: Evaluating Knowledge- and Reasoning-Intensive Video Generation in Science Domains

AI summary

Overview

Research area: Computer vision / generative video evaluation, with a focus on knowledge- and reasoning-intensive text-to-video generation in scientific domains.

Technical level: Intermediate. The evaluation protocol and metric definitions are accessible to a general machine-learning reader, but interpreting the benchmark items themselves requires domain knowledge in the relevant scientific subject.

Scope: The paper introduces Sci-VBench, a benchmark of 1,253 expert-authored video-generation prompts spanning 60 subjects across four disciplines, together with a released per-example evaluation specification and an empirical comparison of 16 text-to-video models.

What This Paper Is About

Video generators now produce convincing-looking clips, but "looking plausible" is not the same as correctly depicting a scientific process — a video can violate a conservation law or reverse a causal relationship while still appearing realistic. Existing benchmarks mostly measure perceptual quality, prompt alignment, or generic physical commonsense, and the one concurrent scientific benchmark, VideoScience-Bench, covers only physics and chemistry through 200 prompts. Sci-VBench's goal is to test whether models can generate videos whose underlying mechanism is scientifically correct, and to do so with evaluation criteria that non-experts and automated judges can apply without recruiting domain experts.

Key Contributions

1. A multidisciplinary benchmark of 1,253 expert-annotated examples spanning 60 subjects under four core disciplines (Natural Science, Healthcare, Humanities & Social Sciences, and Engineering). Each example was authored by a domain expert and independently reviewed by a second expert.

2. Released per-example evaluation specifications. Each prompt is paired with a high-level reference guide (target concept, minimal mechanistic assumptions, phase-based expected phenomena) and a 1–5 anchored rubric. Prompt Grounding, Scientific and Causal Correctness, and Spatiotemporal Consistency receive per-example rubrics; Low-level Perceptual Fidelity uses a single rubric shared across all examples.

3. A validated rubric-based evaluation protocol. The authors measure agreement between expert ratings and both non-expert human raters and MLLM-as-Judge systems, with and without the evaluation specification, and compare against prior automatic evaluators (VideoScore, VideoScore2, VideoReward, ETVA).

4. A benchmark of 16 frontier text-to-video models — eight proprietary and eight open-source — on the benchmark, plus an error analysis and a prompt-rewriting ablation.

Main Findings

• Perceptual quality is saturated, mechanism is not. VBench-based Vision Tools (VT) scores are nearly flat across all 16 models (3.79–4.12), while automatic Scientific and Causal Correctness ranges from 1.24 to 3.34 and human SCC from 1.12 to 3.06. Human Low-level Perceptual Fidelity ratings of the same construct spread wider (2.66–3.76), so the authors describe the proxy as saturated as far as the VBench-based measure can resolve, not saturated in absolute terms.

• The proprietary–open-source gap is concentrated on reasoning. Gemini-Omni-Flash attains the best automatic average (3.38) and human average (3.18), followed by HappyHorse-1.1 and Seedance-2.0; these three surpass Veo-3.1-Fast, Veo-3.1, and Sora-2 (2.98–3.00). MiniMax-H3 leads the open-source group but reaches only 1.63 on SCC, described as half the proprietary best. On Spatiotemporal Consistency open-source models are not behind: Wan2.2-5B attains the highest automatic score of any model (2.79).

• No model is uniformly strong across disciplines. Gemini-Omni-Flash leads three of the four disciplines (3.33 on Engineering, 3.13 on Natural Science, 2.95 on Humanities & Social Sciences), while Sora-2 leads Healthcare (2.97). Open-source profiles are similarly uneven, so models with nearly identical overall averages can differ in which domain mechanisms they preserve.

• Evaluation specifications improve non-expert agreement. Pearson correlation with expert ratings rose from 83.2 to 84.7 on Low-level Perceptual Fidelity, from 74.1 to 83.3 on Prompt Grounding, from 68.2 to 82.5 on Scientific and Causal Correctness, and from 70.3 to 79.6 on Spatiotemporal Consistency. With the specification, non-experts agreed with experts more closely than any MLLM-as-Judge instantiation tested, on every dimension (overall averages: 82.5 with specification, 74.0 without).

• Rubric-conditioned MLLM judging beats prior automatic evaluators. The strongest judge, Qwen3.5-397B-A17B, reached an overall average correlation of 63.4, versus VideoScore (47.5), VideoScore2 (51.6), VideoReward (47.4), and ETVA (47.9). Judging alignment generally increased with evaluator scale (Qwen3.5-397B-A17B 63.4, Gemma-4-31B 60.4, Qwen3.5-9B 55.7).

• Expert reference labels are stable. A second independent expert re-rated 300 sampled labels and reached Cohen's κ = 0.842 with the first. A separate audit of 200 randomly sampled examples, in which a second annotator wrote an independent specification, produced quadratic weighted κ = 0.79 (Prompt Grounding), 0.75 (Scientific and Causal Correctness), and 0.73 (Spatiotemporal Consistency).

• Prompt rewriting helps unequally. Rewriting testmini prompts with Gemini-3-Flash and regenerating with Wan2.2-5B and HunyuanVideo-1.5 gave the largest gains on Scientific and Causal Correctness (+23.3% and +51.7%), smaller gains on Prompt Grounding (+12.4% and +26.7%), and the smallest on Spatiotemporal Consistency (+7.5% and +16.5%). HunyuanVideo-1.5 reached 2.20 on SCC, above the best verbatim open-source score in the main table (1.63) but below the strongest proprietary system (3.34).

• Three recurring error categories. The authors classify failures as poor adherence to instructions, inaccurate simulation of scientific principles (aesthetics prioritized over physical realism), and deficiencies in temporal coherence and visual quality.

• testmini is a faithful proxy. Open-source models were evaluated on both the full benchmark and testmini; per-model full-benchmark averages differ from testmini by at most 0.07.

Methodology in Plain English

Building the benchmark. The authors first ran a user study with 133 undergraduate and graduate students, asking them to curate video prompts that require expert-level reasoning in their own field. After manually analyzing those prompts alongside videos generated by Sora and Wan 2.2, the authors selected 60 subjects whose core concepts are both expert-level and verifiable from video evidence alone. 61 expert annotators (11 undergraduates, 45 graduate students, and five of the authors) were recruited, matched to subjects by expertise, and trained on an annotation protocol. Every example was authored by one annotator and independently reviewed by another.

Writing the prompts. For each subject, an annotator picked a target concept from canonical textbooks and course materials that has a mechanism-governed visual realization — reaction dynamics in chemistry, conservation-driven interactions in engineering, intervention response in healthcare. They then wrote a minimal prompt that states only the observable initial setup and any explicit intervention or task objective, deliberately omitting the mechanistic trajectory, so the model must infer the mechanism rather than repeat a described outcome.

Writing the evaluation criteria. Alongside each prompt, annotators wrote a high-level reference guide (target concepts, minimal assumptions, phase-based expected storyline, visibility constraints, plus brief clarifications for non-expert verifiers) and a detailed 1–5 anchored rubric tied to observable evidence. The rubric defines what earns full credit, what counts as partial correctness, and what warrants low scores.

Quality control. A second domain expert reviewed every example for prompt clarity, visual testability of the mechanism, and mutual consistency of prompt, reference guide, and rubric. The reviewer revised failing examples and an author verified each revision.

Two splits. The full split contains all 1,253 examples; testmini is a fixed subset of 150 examples (37 Engineering, 16 Healthcare, 78 Natural Science, 19 Humanities & Social Sciences) for rapid, cost-constrained iteration. Open-source models were run on both splits; proprietary models were run on testmini only, because generating the full benchmark through commercial APIs is prohibitively expensive.

Scoring. Expert human ratings on a 1–5 scale serve as the reference labels. For automation, Low-level Perceptual Fidelity uses VBench's six Video Quality metrics, normalized and averaged into a composite VT score linearly mapped to the 1–5 range. The other three dimensions use a rubric-conditioned MLLM-as-Judge pipeline instantiated with Qwen3.5-397B-A17B, which receives the prompt, the video, the reference guide, and the rubric for one dimension only, states a brief justification, then gives a 1–5 score; every video–dimension pair is scored in three independent runs and the mean is reported.

Reliability testing. Every testmini video was rated by an expert. Reliability analysis used the 1,500 videos generated by the ten systems released on or before January 2026. Two separate non-expert cohorts scored videos either with or without the evaluation specification, and multiple MLLM judges were compared against the expert labels using instance-level Pearson correlation.

Why This Matters

Impact on research. The paper argues that progress in visual realism has not translated into reliable modeling of scientific and causal dynamics, and that generic perceptual metrics cannot resolve this difference. Releasing reusable per-example rubrics shifts expert evaluation from a per-study recruitment problem to a portable artifact, and the agreement measurements give evidence that such rubrics produce consistent scores across raters. The authors note that prior work has found persistent reliability gaps in model-based evaluation for AI-generated videos and complex scientific tasks, motivating structured specifications of this kind.

Real-world applications:

• Evaluating AI-generated instructional and educational science videos, where a superficially correct animation of a wrong mechanism would mislead learners.
• Screening generative models before use in scientific communication, medical or clinical visualization, and public health messaging, where causal accuracy matters.
• Benchmarking and procurement decisions for organizations choosing among proprietary and open-source video generators for expert-domain work.
• Providing reusable domain-expert rubrics that small teams can apply without hiring subject-matter specialists for every evaluation round.

Industry relevance. The reported gap concentrates on Prompt Grounding and Scientific and Causal Correctness rather than on consistency or appearance, including an SCC range of 1.24 to 3.34 across the 16 evaluated models. Combined with the prompt-rewriting result — where explicit wording lifts SCC substantially but leaves a large residual distance to the strongest proprietary system — this points to generator capability, not prompt specification, as the binding constraint for expert-domain applications.

Future Directions

• Closing the mechanism gap. The authors state that the observed failure modes point to concrete opportunities to improve instruction adherence, mechanistic consistency, and spatiotemporal stability in expert-domain generation, but do not propose a specific method for doing so.

• Extending beyond the current subject set. Sci-VBench deliberately excludes expert concepts whose correctness cannot be verified from video evidence alone; whether and how those excluded concepts could be evaluated remains open.

• Improving automatic judges on perceptual and temporal dimensions. MLLM judges aligned more strongly on the reasoning-centric dimensions than on Low-level Perceptual Fidelity and Spatiotemporal Consistency, whose fine-grained visual and temporal artifacts the authors describe as difficult for general-purpose MLLM judges.

• Addressing cost barriers to full-benchmark evaluation. Proprietary systems were evaluated only on the 150-example testmini split because full-benchmark generation through commercial APIs is prohibitively expensive; whether the strong testmini agreement (within 0.07 for open-source models) holds as models change is an open question.

Target Audience

Researchers and engineers working on video generation and video-generation evaluation; benchmark designers interested in rubric-based and expert-knowledge-grounded evaluation protocols; and domain experts in the natural sciences, healthcare, engineering, or the humanities and social sciences who want to assess whether generative video tools represent their field's mechanisms faithfully. Model developers comparing proprietary and open-source systems on reasoning-heavy tasks will also find the per-dimension breakdown and error taxonomy directly useful.

Authors’ abstract

We introduce Sci-VBench, a comprehensive benchmark for evaluating knowledge- and reasoning-intensive video generation across scientific domains. It contains 1,253 expert-annotated examples spanning 60 subjects across four core disciplines: Natural Science, Healthcare, Humanities & Social Sciences, and Engineering. Each example requires models to generate temporally rich videos that demand scientific reasoning and knowledge-grounded synthesis, going beyond surface-level visual plausibility. We further establish a rubric-based evaluation protocol. Our analysis shows that, under this protocol, both non-expert human evaluators and MLLM-as-Judge systems can achieve relatively high agreement with expert judgments, supporting reproducible evaluation at scale. We benchmark 16 frontier proprietary and open-source models and find that, while automatic perceptual-quality scores cluster tightly across systems, performance on Prompt Grounding and Scientific and Causal Correctness varies substantially, with a pronounced proprietary-open-source gap. These findings show that advances in visual realism have not yet translated into reliable modeling of scientific and causal dynamics.