PVBench: Quantifying Physical Plausibility in Video Generation with Object-Level Physical Measurements

AI summary

Overview

Research area: Computer vision — evaluation and benchmarking of generative video models (text-to-video and image-to-video), specifically physical plausibility.

Technical level: Intermediate. Readers should be comfortable with video-generation models (diffusion transformers, flow matching, T2V/I2V settings) and with the idea of using specialized vision models (depth, pose, segmentation, tracking) as measurement instruments.

Scope (one sentence): The paper introduces PVBench, a benchmark that scores generated videos on 46 dimensions across semantic, perceptual, and physical layers — measuring 25 physical dimensions largely through numerical physical quantities extracted by specialized vision models rather than through language-model yes/no judgments.

What This Paper Is About

Video generators now produce visually convincing clips, but "looking real" is not the same as obeying geometry, kinematics, dynamics, and conservation laws — generated objects still clip through each other, vanish, or appear at wrong scales. Existing benchmarks measure this poorly: either they rely on coarse MLLM or human yes/no ratings that cannot localize which physical quantity is violated or by how much, or they cover only a handful of easy physical categories. PVBench's goal is to turn physical plausibility into something numerically computable and testable, using object-level measurements such as 6D pose, metric scale, surface normals, kinematics, and conservation-law residuals.

Key Contributions

1. A systematic physical taxonomy. 46 evaluation dimensions organized into 7 domains, spanning three assessment layers (semantic, perceptual, physical), of which 25 dimensions are physical and split into object-intrinsic properties, relations between objects and space, and object dynamics across space and time.
2. A same-scene T2V/I2V paired prompt set. 600 prompt groups, each with a one-to-one text-to-video and image-to-video variant of the same scene, 30% involving embodied agents (robot arms, humanoids, quadrupeds, multi-agent systems); each prompt binds its target dimensions at construction time and requires the key physical quantity to be visible in frame.
3. A physical-quantity-dominated evaluation protocol. Specialized vision models (including VGGT-Ω, Depth Anything 3, StableNormal, SAM 3, SpatialTracker V2, Any6D, FlowSeek, VFIMamba, ReasonX, DINOv3, CLIP, FineVQ, and OpenCV/Farneback) extract numerical quantities that are checked against physical laws and mapped to 1–5 scores; only dimensions that cannot be computed (materials, phase change) fall back on MLLM scoring. Of the 25 physical dimensions, 17 are purely numerical, 4 hybrid, and 4 pure MLLM.
4. Human-validated calibration. An annotator study with 10 annotators per sample (inter-annotator agreement ICC(2,k) = 0.89) confirms that automatic scores track human judgments, plus threshold calibration for two key evaluator hyperparameters.

Main Findings

• Physical scores lag far behind semantic and perceptual scores. Across 6 T2V and 6 I2V configurations, physical-layer domain scores sit well below semantic ones — for example Seedance 2.5 (T2V) reaches 92.2 on condition and semantic fidelity (CSF) but 47.2 on material properties and field states (MFS) and 30.0 on mechanical plausibility and conservation laws (MC).
• Visual quality is not physical plausibility. SANA-Video 2.0 (T2V) scores 58.2 on perceptual and temporal quality (PTQ) but only 39.0 on behavioral and causal commonsense (BC), showing polished frames and plausible action causality are separable.
• Materials and conservation laws are the weakest areas. Averaged over the six T2V models the full physical-layer score is 38.5; removing MC raises it to 43.4 and removing MFS raises it to 41.9, whereas removing GS lowers it to 35.0 and removing KP to 33.6 — so MC and MFS are what drag the physical score down.
• Ranking summary. Seedance 2.5 performs best overall, followed by MiniMax H3; MC is the lowest-scoring domain for nearly every configuration (e.g. 18.7 for SANA-Video 2.0 T2V, up to 30.0 for Seedance 2.5 T2V).
• Numerical measurement beats MLLM scoring. On the same physical dimensions, physical-quantity scores show higher global entropy (2.25 vs. 1.12) and inter-dimension variance (1.05 vs. 0.14), a much lower full-mark ratio (15.5% vs. 73.7%), and far higher agreement with humans (Spearman 0.84 vs. 0.14).
• Automatic scores agree with humans. Domain-level Spearman correlations are 0.660 (GS), 0.805 (MFS), 0.732 (KP), 0.782 (MC), total 0.772; Kendall coefficients are 0.528, 0.692, 0.619, 0.651, total 0.643. Per-dimension correlations range widely, from 0.125 for object permanence to 0.867 for material properties.
• Complex scenes hurt most models. On a merged high-difficulty subset (embodied agents, occlusion, visual clutter, camera motion), 5 of 6 T2V models drop by an average relative 7.1% and 5 of 6 I2V models drop by an average relative 10.9%; Seedance 2.5 T2V is the exception with a gain of +2.3.
• A first-frame prior helps semantics, not physics. For MiniMax H3, I2V beats T2V on CSF (+4.8), BC (+0.6) and PTQ (+7.1) and on MFS (+3.4), but falls on GS (-3.7) and MC (-5.2), and is roughly flat on KP (-0.8).
• Results are robust to VLM backend choice. Swapping among Qwen3.5-9B, Qwen3.8-27B, and InternVL3.5-8B gives pairwise Spearman correlations from 0.71 to 0.84; Qwen3.8-27B is the default.
• Thresholds matter and were tuned. SAM3 detection confidence of 0.10 gives Spearman 0.601 at 84.3% coverage (vs. 0.132 at 0.50); trajectory object area of 0.1% gives 0.858 at 99.2% coverage (vs. 0.666 at 0.50%).

Methodology in Plain English

The authors start from the claim that "does this video look physical?" is the wrong question, and instead ask: can we measure the underlying physical state and check it against known laws?

Building the test set. They write prompts in three ways: 5 targeted prompts for each of the 46 dimensions; 10 hybrid prompts for each of 23 related dimension pairs; and a high-difficulty supplement that injects embodied agents plus controlled occlusion, camera motion, and other visual confounders. Drafts are produced with GPT-5.6 Sol, then screened by hand for physical plausibility, visual feasibility, and unambiguous observability. For the I2V version, the reference image is synthesized by GPT-image-2 from the prompt's first-frame description, so both variants start from the same scene. Total: 600 prompt groups.

Measuring instead of judging. For each quantifiable physical dimension, a specialized vision model reports a number rather than an opinion — depth, surface normals, 6D pose, metric and relative scale, displacement/velocity/angular velocity, momentum and energy residuals, and so on. The number is compared against the physical law it should satisfy, and then mapped through a manually set threshold into a 1–5 score, with thresholds validated against human ratings. Each dimension is averaged over its samples first, then dimensions are equally weighted so that dimensions with more samples do not dominate. Final scores are linearly mapped to a percentile scale.

Keeping context. Semantic and perceptual layers reuse or adapt established paradigms so that a physical score can be read alongside semantic fidelity and image quality. One dimension (IF-10, reference-condition fidelity) only applies to I2V, so T2V evaluates 45 dimensions while I2V evaluates all 46, with equal-weight aggregation over the dimensions actually evaluated.

Validating. Three T2V models, 25 physical dimensions, and 5 prompts per dimension were scored independently by 10 trained annotators each and averaged, then correlated with the automatic scores. They also ablate the VLM backend, calibrate two evaluator thresholds, and run a leave-one-domain-out ablation.

Experiment scale. Six models — MiniMax H3, SANA-Video 2.0, Wan2.1, Wan2.2, Seedance 2.5, LingBot-Video — in both T2V and I2V configurations, each generating all 600 prompt groups at 720p, 10 seconds, fixed random seed: roughly 7,200 videos, evaluated on a single H20 GPU. Normally each configuration yields 951 dimension-level evaluations.

Why This Matters

Impact on research. PVBench reframes physical plausibility from a subjective semantic impression into a set of measurable object-level quantities with pass/fail criteria against physical laws. Its ablation showing numerical scores are far more discriminative and human-aligned than MLLM scores (Spearman 0.84 vs. 0.14, full-mark ratio 15.5% vs. 73.7%) is a direct argument that future benchmarks should prefer computable measurements over language-based judging. It also raises the coverage bar: prior physical benchmarks cover between 1 and 10 of the 25 physical dimensions PVBench covers (VideoPhy-2 and PhyGenBench 4, Physics-IQ 6, PhyWorldBench 10, WorldModelBench 4, WorldScore 1).

Real-world applications:

• Robot learning and embodied AI: metric scale and 6D pose determine whether a generated clip can supply real dimensions for manipulation and simulation initialization, rather than just inspiration.
• Content creation and VFX: catching clipping, vanishing objects, and wrong-scale artifacts before they reach a final render.
• Scene reconstruction and simulation setup: cross-frame geometric consistency, depth, and normals are exactly the quantities pipelines need to consume a video as data.
• World-model development: physical-plausibility scores give a measurable target for models positioned as learned simulators of the physical world.

Industry relevance. The benchmark spans both open-source and closed-source systems, and the spread is large and diagnostic: leading models nearly saturate semantic fidelity while scoring roughly half as well on material and mechanical dimensions. For teams shipping video generators or building on them, that gap identifies where investment is actually needed — mechanics, forces, and conservation — rather than in the perceptual quality that is already largely solved.

Future Directions

• Ground-truth references via simulation. The authors note the evaluation mostly checks consistency with physical laws and, apart from a few dimensions, lacks ground truth; they plan to build references for conditional image-to-video (CI2V) models using physics engines and other simulation tools.
• Full ablation of the specialized evaluators. Only the VLM backend was ablated (Qwen3.5-9B, Qwen3.8-27B, InternVL3.5-8B); a systematic ablation of the specialized measurement models is explicitly left open.
• Beyond MLLM fallback. Material properties and phase-transition dimensions still rely on MLLM scoring, and the authors expect more robust tracking and finer physical quantities to improve coverage and reduce that dependence.
• Scaling the benchmark. The prompt set is intended to expand as compute and open-source models grow, with per-dimension weak spots (notably object permanence at Spearman 0.125 and motion continuity at 0.240, where samples are often skipped or unobservable) as natural targets for better measurement.

Target Audience

Researchers and engineers working on video generation and world models who need a rigorous way to evaluate physical correctness; benchmark designers interested in replacing MLLM-as-judge with instrumented measurement; robotics and embodied-AI practitioners who want to know whether generated video is accurate enough to use as data; and technical leads comparing open- and closed-source video models beyond visual quality. Readers new to the area will find the taxonomy and plain-language framing of "physical plausibility as computable quantity" accessible, though the appendix-level detail on evaluators and thresholds rewards some background in 3D vision.

Authors’ abstract

Recent video generation models produce visually compelling outputs, yet their physical plausibility remains poorly measured. Physical plausibility asks whether generated objects obey geometry, kinematics, dynamics, and conservation laws. Existing benchmarks rely on coarse visual-language model judgments or target embodied world models rather than video generation itself. We introduce PVBench, a physical video benchmark that quantitatively measures the plausibility of generated videos across 46 dimensions organized into seven domains, spanning: i) instruction and semantic fidelity, ii) behavioral and causal consistency, iii) perceptual and temporal quality, iv) geometric and spatial consistency, v) material properties and field states, vi) kinematic and spatiotemporal persistence, vii) and mechanical plausibility with conservation laws. PVBench comprises 600 carefully crafted prompts, of which 30% involve physical embodied agents such as robot arms, humanoids, quadrupeds, and multi-agent systems. For challenging physically tractable dimensions, it pairs each with specialized vision models that derive verifiable, physically grounded measurements, such as object-level 6D pose, metric scale, surface normals, kinematics, and numerical conservation-law residuals. It thus avoids relying solely on subjective scoring, while supporting both text-to-video and image-to-video generation. Finally, we validate the agreement between automatic metrics and human judgments through human evaluation. Meanwhile, we evaluate mainstream open-source and closed-source models and present several insights. By releasing the benchmark, prompts, and calibration protocol, we aim to foster physically grounded video generation.