Principia: Relational Physics Tests for Video Models

AI summary

Overview

Research area: Computer vision — evaluation of physical reasoning in generative video models and vision-language models.

Technical level: Intermediate. The benchmark design and scoring are explained in accessible terms, but familiarity with video generation, segmentation-based tracking, and basic Newtonian mechanics helps.

Scope: The paper introduces Principia, a calibration-independent benchmark that tests whether video generators and vision-language models preserve relational physical invariants between paired objects across eight Newtonian phenomena.

What This Paper Is About

Modern video generators produce clips that look realistic — scoring around 0.8 on visual-quality benchmarks such as VBench — yet it is unclear whether they actually obey physical laws. Existing physics benchmarks typically require absolute measurements (scale, frame rate, camera calibration, or recorded ground-truth parameters) that are ambiguous or unavailable in generated video, or rely on subjective human judgments of "plausibility." Principia sidesteps this by testing relations between two objects in the same scene that obey the same law, so the test holds regardless of camera, scale, or frame rate.

Key Contributions

1. Principia benchmark: 500+ real-world paired-object scenes covering 8 physics principles, with matched geometry, aligned release, and controlled conditions to allow comparison against analytic expectations. The final dataset comprises 529 scenes, derived from 401 real-world scenes captured in-house plus augmentation, all drawn from approximately 750+ recorded videos.
2. A unit-free relational consistency score: a continuous invariance score computed purely in image space, requiring no metric scale, velocity, or camera intrinsics.
3. A scalable synthetic pipeline in Isaac Sim (Principia-Synth): generates both physically correct and explicitly "anti-physics" (invariant-violating) scenarios, enabling evaluation of vision-language models and video generators, including counterfactuals that cannot be recorded in the real world.
4. A large empirical study: six state-of-the-art video generators (Omni, Veo-3.1, Wan2.2-5B/14B, Cosmos-2.5-2B/14B) and four vision-language models (Gemini-3.1-Pro, Gemini-3-Flash, Qwen-32B, Qwen-4B) are evaluated, showing that no model exceeds 0.42 on Principia despite scoring around 0.8 on visual benchmarks, and that increasing model size does not reliably improve physical consistency.

Main Findings

• Visual quality and physical fidelity are decoupled: All six video generators cluster around 0.8 on VBench but score between 0.14 and 0.42 on Principia. The authors describe the two axes as "nearly orthogonal."
• No generator exceeds 0.5 average consistency: Wan2.2-14B achieves the best overall score at 0.419 ± 0.253, narrowly ahead of Omni at 0.409 ± 0.272 and Veo-3.1 at 0.38 (0.379 ± 0.186 in Table 3).
• Scaling helps unevenly: Cosmos-2.5 scaling from 2B to 14B improves friction (+0.24), pendulum (+0.18), and inertia (+0.13), but reduces restitution (−0.09), momentum (−0.06), and gravity (−0.01). Wan2.2 scaling from 5B to 14B improves restitution (+0.35), friction (+0.40), gravity (+0.32), inertia (+0.26), projectile (+0.22), pendulum (+0.17), and spring (+0.03), while momentum slightly decreases (−0.05). Larger scale does not uniformly improve physical reasoning, and each invariant stresses different failure modes.
• Compute does not predict fidelity: Cosmos-2.5-14B uses more compute per video than Wan2.2-14B (74 vs 66 minutes) yet scores 0.22 lower on overall consistency.
• Momentum is the hardest phenomenon for generators: Nearly all models perform poorly on the momentum task, which the authors attribute to the difficulty of modeling complex multi-object interactions.
• Vision-language models perform near chance at detecting violations: No VLM exceeds 0.7 average agreement; the best (Gemini-3-Flash) reaches 0.661 overall, and the paper reports best-case accuracy of only 67%. The authors suggest architectural rather than scale-related limitations.
• Distinct per-model failure profiles: Omni is strong on inertia and friction but weak on restitution, gravity, and momentum; no model dominates across all phenomena.

Methodology in Plain English

The core idea is that when two objects in a scene obey the same physical law, their motions must satisfy a predictable relationship — an equality, a ratio, or an ordering — and that relationship holds no matter how the camera is positioned or how the scene is scaled.

The researchers designed eight experimental setups, each paired with a specific invariant:

• Gravity / restitution: two identical balls dropped from different heights; rebound ratios r₁/h₁ and r₂/h₂ should match, and drop heights should scale as the square of fall times.
• Friction: two blocks of different mass slide down identical ramps with the same angle and surface; because acceleration on an incline with friction is mass-independent, they should arrive at the same time.
• Rotational inertia: a solid and a hollow cylinder matched in mass and radius roll down a no-slip ramp; the time ratio follows from their moments of inertia.
• Momentum: balls released from different heights, or colliding with blocks of different masses, must produce predictable ordering in block displacement.
• Projectile motion: two spheres launched from different heights — the higher launch gives greater range.
• Pendulum and mass-spring: period and equilibrium extension follow square-root and linear laws respectively.

To make the dataset trustworthy, they enforced matched geometry (for example, the rotational-inertia cylinders were manufactured in-house to match in mass, height, and outer radius within 5%), synchronized mechanical release, validated contact surfaces, and minimized external forces, then discarded videos failing manual and automated checks (lateral drift, asymmetric release, large-angle pendulum motion, ball spin, surface slip). Objects are tracked with SAM3 from hand-annotated initial points, and all measurements are in pixel space.

The consistency score is S_φ = 1 − |F(o₁) − F(o₂)| / (|F(o₁)| + |F(o₂)|), which equals 1 when the invariant holds exactly; S_φ = 0.95 corresponds to roughly a 10% relational asymmetry. Projectile and momentum invariants are qualitative ordering tests, scored as the fraction of scenes where the ordering is satisfied. Videos that fail basic qualitative motion checks are filtered out first using a Directional Consistency Score (DCS), which ranges from −1 to 1 and compares each frame's displacement direction against the expected direction. For generation, each model is conditioned on a text prompt plus the first frame of a real recording, with the experimenter, suspension strings, and release mechanisms inpainted out using Nano Banana 2.

Why This Matters

Impact on research. The paper argues that visual realism is a poor proxy for physical understanding, and that evaluation based on absolute measurements is fragile in generated video. By reframing physics evaluation as calibration-independent relational consistency, Principia offers a contamination-robust protocol — the score is computed within each generated video, so test-set leakage cannot help a model. It also generalizes a prior single-phenomenon two-object gravity protocol to eight phenomena spanning translational, rotational, collisional, and oscillatory dynamics, and extends evaluation to both generators and vision-language models.

Real-world applications:

• Video generation and VFX pipelines: studios and tool builders can use relational checks as automated quality gates before shipping physically implausible output.
• World-model and robotics training data: if generators are used to synthesize training data for agents that predict the consequences of actions, invariant violations translate into unsafe or nonsensical policies.
• Media authenticity and verification: detecting relational physics violations offers a signal for flagging synthetic or manipulated footage that looks realistic but behaves impossibly.
• Benchmark and evaluation design: the unit-free scoring template can be adapted to other domains where calibration is unavailable.

Industry relevance. The finding that larger models do not reliably improve physical consistency — and that open-weights models (Wan2.2-14B) can exceed closed-frontier systems (Veo-3.1, Omni) — challenges the assumption that scale alone delivers world-model capability. The authors state that closing the gap will likely require new training signals or architectural biases for relational invariants.

Future Directions

• Beyond macroscopic Newtonian mechanics: Principia does not address fluid dynamics, soft-body deformation, or thermodynamic phenomena — extending the relational approach to these regimes is an open problem.
• Separating physics failures from generation failures: current scoring does not automatically distinguish genuine physics violations from hallucinated objects, missing objects, or severe deformations, so better disentanglement is needed.
• Improving models, not just measuring them: the synthetic Principia-Synth pipeline is proposed as a resource for developing methods that improve physical consistency, potentially through new training signals or architectural inductive biases.
• Scaling visual diversity: the first frames are staged setups that current image generators cannot reproduce, but they can be cheaply edited to vary backgrounds and lighting, enabling extension to new visual configurations without re-recording. The authors release first-frame inputs and prompts, while holding back the full real-world videos to preserve benchmark integrity.

Target Audience

Researchers and engineers working on video generation, world models, and vision-language models; benchmark designers interested in calibration-free evaluation; robotics and simulation researchers who rely on generated video for training or prediction; and computer vision students looking for a concrete case study in how to design physically grounded, unit-free evaluation protocols.

Authors’ abstract

Evaluating physical reasoning in video models is difficult because absolute motion measurements depend on frame rate, object scale, and camera calibration, all of which are often ambiguous or unavailable in generated video. We propose a different approach. When two objects in the same scene obey the same physical law, their motions must satisfy predictable relationships, and these relationships hold independent of calibration. We introduce Principia, a benchmark that evaluates Newtonian physics through relational consistency between paired objects. Principia spans eight phenomena - gravity, restitution, friction, rotational inertia, projectile motion, momentum, pendulum, and mass-spring oscillation - across translational, rotational, collisional, and oscillatory dynamics, using real-world scenes recorded under controlled protocols. We also introduce a calibration-independent consistency score that quantifies physical violation directly in image space. Across thousands of generations from six state-of-the-art video generators, no model exceeds 0.42 on Principia despite all scoring around 0.8 on VBench. Vision-language models are evaluated on their ability to detect relational physics violations, with the best model achieving only 67% accuracy and most performing near chance level.