Seeing the Unseen: Unified Visible--Invisible Motion for Physically Consistent Video Generation

AI summary

Overview

Research area: Computer Vision — physics-grounded image-to-video generation, combining learned physical simulation with diffusion-transformer video synthesis.

Technical level: Advanced. The paper assumes familiarity with graph neural networks, diffusion/flow-matching models, 3D scene lifting, and video latent representations. The core ideas are still explainable without the math.

Scope: A two-stage framework that learns object-interaction dynamics with a particle-based graph simulator and injects the resulting 3D trajectories into a pretrained video generator through a dual "visible–invisible" motion representation.

What This Paper Is About

Video generators produce visually convincing clips but frequently violate physics: objects pass through each other, collisions fail to trigger, or motion follows no causal logic. The authors argue this happens because these models learn pixel-level temporal patterns rather than the underlying physical processes. Their goal is to learn physical dynamics from data and then feed that knowledge into a video generator in a form rich enough to describe not just what moves in the input frame, but also what becomes visible later as objects collide, rotate, or are revealed.

Key Contributions

1. Particle-based Graph Dynamics Simulator (PGDS): A lightweight, learnable simulator that represents a lifted 3D scene as a graph of particles and predicts 3D motion trajectories, learning physical interactions from data without hand-specified physical parameters.

2. Visible–Invisible Motion Field (VIMF): A motion representation that combines a reference-visible tracking video (points observable in the input frame) with a future-emergent tracking video (points initially occluded or out of view that appear later), giving more complete guidance than 2D optical flow, tracking videos, layouts, or sparse keypoints.

3. An injection scheme into a diffusion transformer: The two tracking videos are encoded by a frozen 3D VAE, concatenated with the noisy video latent along the channel dimension, and added via a zero-initialized strategy so the pretrained backbone's behavior is preserved while motion guidance is absorbed progressively.

4. A physics-aware video generation benchmark: Seven interaction categories — four primitive (free fall, domino, collision, translation) and three compositional (collision+domino, free fall+domino, free fall+collision) — with 50 samples per category, 350 total, including multi-object compositions.

Main Findings

• Best scores across nearly all benchmark scenarios: On the authors' benchmark, the full model reaches SA/PC of 3.976/4.013 (Domino), 4.160/4.394 (Collision), 4.347/4.328 (Free Fall), 3.854/4.016 (Collision+Domino), 3.985/4.017 (Free Fall+Domino), 4.327/4.589 (Free Fall+Collision), and 4.013/4.578 (Translation). Its own backbone, Wan2.2-TI2V-5B, scores 2.650/3.033 on Domino and 2.917/3.067 on Collision.

• Baselines degrade on compositional tasks: Most compared models score reasonably on Translation (single-object motion without interaction) but drop sharply on compositional scenarios. NewtonGen and PhysCtrl are marked "–" for all scenarios except Translation, since they support only simple motion settings.

• Domino tasks are hardest for competitors: Dense layouts and frequent occlusions cause baselines to propagate contact incorrectly or produce unstable motion; the authors report their method generating coherent interaction patterns in these cases.

• External validation on VideoPhy2: On the compatible subset, the full model scores SA 4.01 / PC 4.13 versus Wan2.2-5B at 3.14/3.46 and CogVideoX-5B at 3.21/3.65.

• Ablation confirms the "invisible" half matters: A visible-only variant using the same diffusion backbone scores 3.47/3.78 on the VideoPhy2 subset, below the full model's 4.01/4.13. In the injection comparison (Table 6), the visible-only variant is also lower than the full model in all four tested scenarios, with the largest gaps on Domino and Collision.

• Injection strategy comparison: Converting the same PGDS trajectories into other representations — optical flow for Go with Flow, 10 sparse tracking points for Wan-Move, visible tracking videos for Diffusion as Shader — yields lower SA and PC than the full model on Domino, Collision, and Free Fall.

• PGDS trajectory accuracy (Physion++): Average velocity RMSE 0.0250 m/s, strict Acc-V 95.92%, relaxed Acc-V 98.37%, and kinetic-energy bias −3.5%, across seven scenarios (Dominoes, Collide, Support, Link, Contain, Roll, Drop).

• Error grows slowly with rollout horizon: At 30/60/90 steps, position MSE is 0.7/1.7/3.0 cm², center-of-mass error is 0.5/1.2/2.6% of object size, and rotation error is 1.3/3.8/7.5 degrees.

• Efficiency trade-off: In the configuration comparison, PGDS requires 0 manual physical parameters and roughly 5 minutes total runtime, versus roughly 10 minutes for simulator-based methods (PhysGen, PhysMotion, PhysGen3D), roughly 8 minutes for PhysCtrl, roughly 6 minutes for NewtonGen, and roughly 2 minutes for LVLM-based PhysT2V.

• Human preference: Pairwise human evaluation reports gains over Wan2.2-5B of +42.0% in instruction following, +38.0% in video quality, and up to +44.0% in overall preference.

Methodology in Plain English

The pipeline has two halves.

Half one — learn how objects behave. A single input image is lifted into 3D: depth is estimated and turned into a 3D point map, foreground objects are segmented (SAM3) and reconstructed (SAM 3D) with metric poses, everything is placed in one scene coordinate system, and a ground plane is fitted with RANSAC. The scene is then re-expressed as roughly 3000 particles. Those particles form a graph, with edges connecting neighbors within radius r = 0.08. Each particle carries a feature vector describing its material, its offset from its object's centroid, and its velocity state. Ten rounds of message passing let particles exchange information with neighbors, and a decoder outputs predicted velocities, which are integrated into trajectories. This is trained by mean squared error against ground-truth particle velocities.

Half two — turn those trajectories into video. The predicted 3D trajectories are projected into two tracking videos: one for points visible in the reference frame, one for points that only become visible later. Both are encoded with a frozen 3D VAE, concatenated onto the noisy video latent, and fed into Wan2.2-TI2V-5B. The new motion channels are zero-initialized so the pretrained model's behavior is not disrupted early in training. Training uses flow matching, with the first-frame latent kept clean and assigned a zero timestep for appearance conditioning. The patch embedding layer trains at 2×10⁻⁴ while all other parameters use 2×10⁻⁵, letting the new channels adapt quickly without damaging the pretrained generator.

Data. PGDS is trained on Physion++, roughly 16K samples from the ThreeDWorld simulator covering eight interaction types. The generation stage combines simulated data with real videos from MiraData, with motion supervision extracted using SpatialTracker2.

Why This Matters

The work targets a central weakness of current video generators: they make pretty frames but poor physics. By showing that a compact learned simulator plus a richer motion representation beats both simulator-based pipelines and LLM-commonsense approaches, it suggests a practical middle path that avoids per-scene 3D reconstruction and material tuning.

Real-world applications:

• Simulation and scientific visualization — generating plausible dynamic scenes from a single still, where contact and causality must hold.
• Education — producing illustrative physics demonstrations (domino chains, collisions, free fall) that behave correctly.
• Content production and VFX — previsualizing interaction-heavy shots without hand-authoring simulation parameters.
• Robotics and embodied AI — supplying learned dynamics priors that could inform prediction or planning in interactive scenes.

Industry relevance: The method builds directly on an open backbone (Wan2.2-TI2V-5B) and adds a lightweight module rather than retraining from scratch, which is attractive for teams wanting to improve physical fidelity of existing video pipelines at modest cost.

Future Directions

1. Broaden the physical domain: The evaluation covers structured interactions — collisions, domino effects, free fall. Extending to fluids and highly deformable objects would require additional data and simulation design, which the authors flag as future work.

2. Harden the 3D lifting stage: The pipeline depends on an intermediate lifting step; geometry estimation errors can propagate into dynamics simulation and then into generation. More robust 3D perception is an explicit open problem.

3. Scale to harder multi-object scenes: PGDS models local particle interactions rather than object-specific global geometry, and prior work suggests simulators of this type remain reliable as particle counts grow — testing that claim on denser, more cluttered scenes is a natural next step.

4. Robustness under imperfect guidance: The authors point to appendix experiments on incomplete motion guidance and upstream perception errors; systematically characterizing failure modes and adding correction mechanisms would strengthen deployment.

Target Audience

Researchers and engineers working on controllable or physics-aware video generation, learned physics simulation, and graph neural network dynamics models. It is also relevant to practitioners integrating motion control into diffusion-transformer video backbones, and to readers interested in benchmark design for physical plausibility. Readers without background in diffusion models or graph networks will find the conceptual narrative accessible, but the method sections require intermediate-to-advanced familiarity.

Authors’ abstract

Recent advances in video generation have achieved impressive visual quality, yet they often fail to produce physically consistent dynamics due to the lack of explicit modeling of underlying physical processes. Existing approaches either rely on simulators with strong assumptions or use coarse semantic guidance, and further struggle to effectively inject physical priors into generation models. In this work, we propose Seeing the Unseen, a unified framework for physics-grounded video generation. Our approach decomposes the problem into two stages: learning physical dynamics and injecting them into video synthesis. First, we introduce a lightweight Particle-based Graph Dynamics Simulator (PGDS) that learns generalizable physical interactions from data and predicts plausible 3D motion trajectories. Second, we propose a Visible--Invisible Motion Field (VIMF) that captures both motion observable in the input frame and motion that emerges over time due to object dynamics. This representation enables more complete and structured motion guidance compared to conventional physical signals. By integrating these trajectories into a diffusion-based generator, our method produces videos that are both visually coherent and physically consistent. Extensive experiments demonstrate improved motion accuracy, interaction consistency, and reduced physical artifacts compared to strong baselines.