RISE-Video: Can Video Generators Decode Implicit World Rules?

AI summary

Overview

Research area: Computer vision / generative video evaluation — specifically benchmarking reasoning capabilities of Text-Image-to-Video (TI2V) models.

Technical level: Intermediate. The paper assumes familiarity with diffusion-based video generation, multimodal large language models (LMMs), and standard benchmark evaluation design, though the core argument is accessible.

Scope: The paper introduces RISE-Video, a 467-sample, eight-category benchmark with a four-metric evaluation protocol and an automated LMM judging pipeline, used to evaluate 11 state-of-the-art TI2V models on implicit world-rule reasoning.

What This Paper Is About

Contemporary video generators produce visually impressive output, but it is unclear whether they actually understand the implicit rules of the world — physics, social norms, procedures, spatial relations — or merely render plausible-looking pixels. Existing benchmarks like VBench largely measure perceptual quality and temporal coherence, leaving higher-level reasoning untested, especially in the TI2V setting. RISE-Video addresses this gap by shifting evaluation from surface aesthetics to cognitive reasoning, and by providing an automated judge that approximates human assessment at scale.

Key Contributions

1. A reasoning-centric TI2V benchmark. RISE-Video contains 467 meticulously curated, human-expert-annotated samples across eight reasoning dimensions: experiential, commonsense, temporal, societal, perceptual, spatial, subject-specific, and logical reasoning.

2. Four complementary evaluation metrics. The benchmark defines Reasoning Alignment, Temporal Consistency, Physical Rationality, and Visual Quality, plus two aggregate scores — a Weighted Score (weights 0.4 / 0.25 / 0.25 / 0.1) and an Accuracy score that requires full marks on all four dimensions.

3. An automated, scalable LMM judging pipeline. Manually designed, reasoning-aware questions and prompts drive GPT-5 as the primary judge (with gpt-5-mini for Visual Quality only), plus specialized protocols for Schematic Puzzle tasks that are ill-suited to standard LMM-as-a-Judge evaluation.

4. A systematic empirical study of 11 TI2V models. The evaluation spans six closed-source and five open-source systems, revealing consistent reasoning deficits and validating the pipeline's alignment with human judgment.

Main Findings

• Reasoning is unsolved. Even the best model, Hailuo 2.3, reaches only 22.5% Accuracy. The next best are Veo 3.1 (22.3%) and Sora 2 (21.3%).
• Closed-source models lead across the board. Open-source models underperform on both reasoning capability and visual quality; CogVideoX1.5-5B has the lowest overall Weighted Score (49.5%) and Accuracy (1.9%).
• Reasoning Alignment is the discriminating metric. Hailuo 2.3 scores 76.6% on Reasoning Alignment, exceeding the second-ranked Wan 2.6 (70.0%) by 6.6%.
• Sora 2 is strongest on Temporal Consistency. It records 92.2% on that metric, indicating better preservation of non-instructed elements.
• Perceptual tasks are the easiest; logical tasks are the hardest. Models score comparatively well on Perceptual Knowledge (Wan 2.6 reaches 90.7 Weighted Score) but consistently poorly on Logical Capability.
• Rule-based game scenarios fail broadly. In the Gold Miner scenario, no evaluated model captures the stone along the current hook trajectory; Veo 3.1 shows hook-shape inconsistencies, and Kling 2.6 moves the object without physical contact.
• Experiential reasoning shows a split. Only Veo 3.1 and Hailuo 2.3 infer the necessary action of unscrewing a bottle cap when a person drinks water; other models omit this step.
• Some models barely react to instructions. Kling 2.6 often produces near-static content and preserves original appearance in chameleon camouflage and rose capillary-action tasks rather than performing the required commonsense transformation. Veo 3.1 fails to make the chameleon match the surrounding branch.
• Temporal discontinuities persist at the top. Sora 2 and Veo 3.1 exhibit abrupt changes between consecutive frames.
• GPT-5 is the most human-aligned judge overall. Across metrics, GPT-5 achieves MAE/STD of 0.11/0.23 (RA), 0.51/0.85 (TC), 0.80/0.76 (PR), and 0.22/0.36 (VQ, judged by gpt-5-mini). Qwen3-VL-235B has a lower error on Temporal Consistency (0.42/0.80) but shows a high-score bias that limits discrimination between genuinely good and severely defective outputs. Gemini-3-Flash performs worst (e.g., MAE 1.52 on Physical Rationality).
• Gpt-5-mini is sufficient for visual quality. It aligns more tightly with human ratings than both Gemini-3-Flash and Qwen3-VL-235B on the Visual Quality dimension.

Methodology in Plain English

The researchers built a test set of 467 image-and-instruction pairs, each designed to require a specific kind of implicit knowledge — knowing that a vase breaks when struck, that an orange must be peeled before eating, that events can unfold in reverse, and so on. Each sample was labeled by human experts.

Each sample is scored on four things. Reasoning Alignment checks whether the generated video gets the underlying knowledge right; the LMM judge answers a set of hand-written yes/no questions per sample, producing a 0–1 score. Temporal Consistency checks that only instruction-required changes occur, with the judge rating 1–5 and explicitly ignoring the instructed change. Physical Rationality checks gravity, collisions, object permanence and similar, rated 1–5, and is applied only to physically grounded scenes, excluding abstract planar puzzles. Visual Quality checks sharpness, texture, artifacts and lighting, rated 1–3, with super-resolution applied to low-clarity inputs so the judge does not mistake native resolution for generative blur.

Frame sampling is tailored per metric: 2 fps uniform sampling for full-progression tasks, lower rates for final-state tasks, and 6 uniformly sampled frames excluding the first and last for Visual Quality. For hard-to-verbalize "Schematic Puzzles" — maze navigation, symmetry generation, and board games — the team built specialized alternatives: color-based trajectory tracking for mazes (satisfied constraints {0,1,2} map to scores {0, 0.5, 1}), grid-level HSV cell comparison for symmetry (accuracy 1 − (FP+FN)/N, discretized at a 0.85 threshold), and reference-image-assisted LMM comparison for board games.

Two overall scores combine the metrics: a Weighted Score (0.4/0.25/0.25/0.1) and an Accuracy score where a sample counts as correct only if all four dimensions achieve full marks. To validate the automated judge, five independent expert annotators rated samples, and the team computed Mean Absolute Error and Standard Deviation against that consensus for three judge models.

Why This Matters

The paper argues that current TI2V systems have learned to look right without learning to reason right, and that this gap is invisible to benchmarks that only measure aesthetics and coherence. By making implicit reasoning measurable and automatable, RISE-Video gives the field a way to track progress on a capability that visual-fidelity metrics systematically miss.

Real-world applications:

• Model selection and procurement. Teams deploying video generation can use the benchmark's per-category scores to pick the right model for a domain — for instance, temporal stability versus experiential reasoning.
• Training signal design. The category-level breakdown (Logical Capability as a consistent bottleneck) identifies where data and objectives need work.
• World-simulator and embodied-AI development. Rule-consistent video generation underpins simulation environments for robotics and planning, where physical and procedural errors are costly.
• Content and media production. Procedural, societal, and commonsense reasoning failures translate into unusable footage for instructional, educational, or culturally sensitive content.

Industry relevance: The authors are affiliated with Shanghai Jiao Tong University, Tencent Youtu Lab, Xidian University, Beijing Normal University, and The Chinese University of Hong Kong. The benchmark spans six closed-source production systems (Hailuo 2.3, Wan 2.6, Sora 2, Veo 3.1, Kling 2.6, Seedance 1.5-pro) and five open-source models, and the LMM-based pipeline is positioned as a cost-effective substitute for large-scale human evaluation — directly relevant to teams that need repeatable evaluation at production scale. Code and data are released publicly.

Future Directions

1. Improving Logical Capability. The authors identify logical reasoning as a consistent bottleneck across all 11 models, making it the clearest target for new architectures or training data.
2. Closing the open-source gap. Open-source models trail substantially on both reasoning and visual quality; whether this is a scale, data, or training-recipe problem remains open.
3. Extending the judging pipeline. The validation covers three LMM judges (GPT-5, Qwen3-VL-235B, Gemini-3-Flash); the authors point to Qwen3-VL-235B's high-score bias as evidence that judge calibration needs continued study, and to gpt-5-mini's strong perceptual performance as a cost-efficiency lever.
4. Reasoning-aware model design. The paper calls for TI2V architectures and training objectives explicitly built around rule-consistent reasoning rather than visual realism alone.

Target Audience

Researchers and engineers working on video generation, multimodal evaluation, and world models will get the most from this paper. It is also valuable for benchmark designers interested in LMM-as-a-Judge methodology and human-alignment validation, for teams selecting or deploying TI2V models in production, and for graduate students entering generative video research who want a clear map of where current systems fail. Readers need some background in diffusion-based generation and multimodal evaluation to follow the metric design, but the headline results are readable without it.

Authors’ abstract

While generative video models have achieved remarkable visual fidelity, their capacity to internalize and reason over implicit world rules remains a critical yet under-explored frontier. To bridge this gap, we present RISE-Video, a pioneering reasoning-oriented benchmark for Text-Image-to-Video (TI2V) synthesis that shifts the evaluative focus from surface-level aesthetics to deep cognitive reasoning. RISE-Video comprises 467 meticulously human-annotated samples spanning eight rigorous categories, providing a structured testbed for probing model intelligence across diverse dimensions, ranging from commonsense and spatial dynamics to specialized subject domains. Our framework introduces a multi-dimensional evaluation protocol consisting of four metrics: \textit{Reasoning Alignment}, \textit{Temporal Consistency}, \textit{Physical Rationality}, and \textit{Visual Quality}. To further support scalable evaluation, we propose an automated pipeline leveraging Large Multimodal Models (LMMs) to emulate human-centric assessment. Extensive experiments on 11 state-of-the-art TI2V models reveal pervasive deficiencies in simulating complex scenarios under implicit constraints, offering critical insights for the advancement of future world-simulating generative models.