PhysInOne: Visual Physics Learning and Reasoning in One Suite

AI summary

Overview

• Research area: Computer Vision — large-scale synthetic video/3D datasets for physics-grounded world models, video generation, and embodied AI.
• Technical level: Advanced. The paper assumes familiarity with 3D/4D scene representations (NeRF-style deformation fields, Gaussian-based dynamic models), diffusion/flow-matching video generation, and physically based simulation (MPM, SPH).
• Scope in one sentence: PhysInOne is a synthetic dataset of 153,810 dynamic 3D scenes and 2 million annotated videos covering 71 everyday physical phenomena, benchmarked across four tasks: physics-aware video generation, future frame prediction, physical property estimation, and motion transfer.

What This Paper Is About

Models trained on today's large video and 3D datasets can produce visually convincing output but frequently violate basic physics — objects falling upward or abruptly changing velocity. The authors argue this is largely a data problem: existing physics-focused datasets contain only dozens to thousands of examples and cover a narrow set of phenomena. PhysInOne addresses this by generating a large, heavily annotated synthetic corpus of physically simulated scenes in which every clip is governed by stated physical laws, then using it to test how far today's foundation models fall short.

Key Contributions

1. A large-scale physics dataset. PhysInOne contains 2 million videos across 153,810 dynamic 3D scenes, covering 71 basic physical phenomena in mechanics, optics, fluid dynamics, and magnetism. Each scene is rendered from 12 fixed cameras plus 1 moving monocular camera (13 videos per scene), with ground-truth annotations spanning geometry, semantics, motion, physical properties, and text.

2. Multiobject, multiphysics scene construction. The authors combine the 71 basic phenomena into 3,284 physical activities (71 single-physics, 943 double-physics, and 2,270 triple-physics activities) and instantiate them in scenes containing multiple interacting objects, varied materials, and 528 complex 3D backgrounds — features absent from prior physics datasets.

3. A new quantitative metric for physical motion fidelity (PMF). PMF compares the discrete Fourier transform energy of a generated video against a physically accurate reference video produced from the same initial frames and text prompt, capturing kinematic deviation rather than pixel-level similarity. Higher PMF indicates closer agreement with real physical motion.

4. Four task benchmarks exposing current model limits. The paper benchmarks physics-aware video generation, long- and short-term future frame prediction, physical property estimation, and motion transfer, showing where fine-tuning helps and where state-of-the-art methods fail.

Main Findings

• Fine-tuning on PhysInOne improves physical plausibility for some models. SVD improved from PMF 2.753 to 3.147 with SFT (FVD 203 to 143). Wan2.2-5B improved from 2.041 to 2.978 with SFT (FVD 258 to 190). CogVideoX reached 2.877 before fine-tuning and 2.869 after LoRA, essentially unchanged.
• Supervised fine-tuning was the strongest of the three techniques overall, with Wan2.2-5B SFT reaching the highest reported human rating (5.95) and the highest PMF among fine-tuned Wan2.2-5B variants. Final Layer Tuning on Wan2.2-5B performed worse than the untuned model on PMF (2.227 vs. 2.041) and worse on FVD (341 vs. 258).
• Human ratings track PMF. The paper reports that human perception of physical dynamics correlates with the new PMF metric, though it does not report a numeric correlation value.
• Different physics domains are not equally hard. Most evaluated generation models scored higher in magnetism and fluid and lower in mechanics and optics, which the authors highlight as a challenge for future work.
• Long-term prediction degrades badly at novel viewpoints. On 103 test scenes (~2.6 seconds ahead, ~78 future frames), DefGS reached PSNR 22.85 at seen viewpoints but 17.95 at novel ones; TiNeuVox fell from 21.49 to 15.20. MAGI-1 reported 23.14 and ExtDM 19.55 for seen viewpoints only.
• Short-term prediction is more accurate but shows the same view gap. Predicting the next 10 frames, FreeGave reached PSNR 27.09 / 20.80 (seen / novel) and DefGS 26.02 / 20.92. MAGI-1 reached 26.75 and ExtDM 22.14 at seen viewpoints.
• Property estimation is plausible but inaccurate on complex scenes. Across elastic solids, plasticine, Newtonian fluids, non-Newtonian fluids, and granular substances (20 test scenes, 4 per category), both PAC-NeRF and GIC produced large error spreads. On resimulation, GIC scored PMF 5.938 / PSNR 26.90, ahead of PAC-NeRF at 5.617 / 24.12.
• Motion transfer preserves appearance but not physics. On 273 dynamic scenes, MotionPro (PMF 3.484, PSNR 20.28) and GoWithTheFlow (PMF 3.309, PSNR 18.98) retained high visual fidelity but failed to transfer complex physical motions such as moving cars or a falling ball.

Methodology in Plain English

The authors start from a university-level physics textbook and identify 71 everyday phenomena across mechanics, optics, fluid dynamics, and magnetism, deliberately excluding thermodynamics and acoustics because those are rarely visible without extra sensors.

They collect 2,231 objects (~163 categories) from Sketchfab, FAB, and BlenderKit, including solid, interactable, destructible, deformable, granular, and liquid objects, plus 623 materials across plastic, metal, wood, stone, and fabric, and 528 indoor/outdoor backgrounds.

They then combine phenomena into 71 single-, 943 double-, and 2,270 triple-physics activities, and instantiate each activity in many concrete scenes with different objects, materials, and backgrounds — averaging 46.84 scenes per activity, with 3.9, 6.3, and 7.8 objects per scene for single-, double-, and triple-physics scenes respectively.

Dynamics are simulated with Chaos Physics in Unreal Engine 5 for most phenomena, Taichi's Material Point Method (MPM) for deformable and granular objects, and Smoothed Particle Hydrodynamics (SPH) via Doriflow for liquids. Each scene is rendered at 1120×1120, 30 FPS (60 FPS for the 8,780 scenes containing laser activities), averaging 5.2 seconds long. Text descriptions average about 64 English words per scene and were manually written and proofread using Qwen3. The data is split 8:1:1, with all assets of a given kind confined to one split to prevent leakage.

For evaluation, the authors sample 83,650 text-video pairs for fine-tuning and 772 pairs for a test set called test-small; long-term prediction uses 103 scenes (test-mini) and property estimation uses 20 scenes (test-tiny).

Why This Matters

Impact on research. PhysInOne is presented as the largest dataset of its kind, orders of magnitude beyond prior visual physics datasets, and as a benchmark for physics-grounded world models in generation, simulation, and embodied AI. Its controlled setup — identical initial frames and prompts for reference and generated videos — gives a quantitative alternative to purely qualitative physics QA benchmarks.

Real-world applications:

• Physics-aware video generation and film/animation production, where generated clips must obey plausible motion.
• Future-aware robot planning and embodied AI, where continuous short-term frame prediction supports manipulation and control.
• Physical property estimation (system identification / inverse physics) for resimulation, scene editing, and robot manipulation.
• Controllable motion transfer for image animation and virtual prototyping.

Industry relevance. The paper works with widely used open video foundation models (SVD-XT, CogVideoX-1.5-5B, Wan2.2-5B) and standard adaptation techniques (LoRA, SFT, FLT), so the fine-tuning recipe maps directly onto practical pipelines. The failures it documents — novel-viewpoint degradation and poor intrinsic property estimation — mark concrete gaps for teams building simulators, world models, and robotics stacks. The work was supported in part by the Research Council of Hong Kong under Grants 15219125 and 15225522.

Future Directions

• Closing the novel-viewpoint gap: current methods perform reasonably on trained viewpoints but collapse on unseen angles, which the authors frame as a core open problem in 3D physical motion modeling.
• Improving property estimation for complex objects against intricate backgrounds, where both PAC-NeRF and GIC showed large errors.
• Extending motion transfer beyond simple optical-flow patterns so that multiphysics, multiobject interactions can be transferred between scenes.
• Expanding coverage beyond the current four physics areas and 71 phenomena, and continuing to benchmark newer video generation models on the dataset (the authors state they will update results on the project website).

Target Audience

Researchers and engineers working on video generation and world models, 3D/4D scene reconstruction and simulation, physical scene understanding, and embodied AI or robotics. It is also relevant to dataset builders interested in synthetic data pipelines and to practitioners who need a benchmark for whether a model's output is physically plausible rather than merely visually convincing.

Authors’ abstract

We present PhysInOne, a large-scale synthetic dataset addressing the critical scarcity of physically-grounded training data for AI systems. Unlike existing datasets limited to merely hundreds or thousands of examples, PhysInOne provides 2 million videos across 153,810 dynamic 3D scenes, covering 71 basic physical phenomena in mechanics, optics, fluid dynamics, and magnetism. Distinct from previous works, our scenes feature multiobject interactions against complex backgrounds, with comprehensive ground-truth annotations including 3D geometry, semantics, dynamic motion, physical properties, and text descriptions. We demonstrate PhysInOne's efficacy across four emerging applications: physics-aware video generation, long-/short-term future frame prediction, physical property estimation, and motion transfer. Experiments show that fine-tuning foundation models on PhysInOne significantly enhances physical plausibility, while also exposing critical gaps in modeling complex physical dynamics and estimating intrinsic properties. As the largest dataset of its kind, orders of magnitude beyond prior works, PhysInOne establishes a new benchmark for advancing physics-grounded world models in generation, simulation, and embodied AI.