PhyMo: Learning Physical Dynamics with Accurate and Continuous Motion from Multi-View Videos

AI summary

Overview

• Research area: Computer vision — dynamic 3D scene reconstruction and physics-aware motion learning from multi-view video, built on 3D Gaussian Splatting.
• Technical level: Advanced. The paper assumes familiarity with NeRF/3DGS representations, deformation fields, and kinematic state estimation (velocity/acceleration).
• Scope (one sentence): PhyMo is a framework that reconstructs dynamic 3D scenes from multi-view videos while recovering motion that is geometrically faithful within the observed time span and kinematically continuous when extrapolated into the future.

What This Paper Is About

Recovering how a 3D scene moves, not just how it looks, is hard because image-based supervision is ambiguous: a model can make a wrong deformation look right by tweaking color or making Gaussians transparent instead of actually moving them correctly. Existing methods either fit smooth trajectories that break down outside the observed window, or learn velocity-like states that accumulate error, and in both cases the motion is only weakly constrained by rendering loss. PhyMo tackles both failure modes at once, aiming for reconstructions that are correct in 3D space and motion states that stay temporally consistent for future-frame prediction.

Key Contributions

1. A Motion-Aware Trajectory Representation that reparameterizes canonical appearance (opacity and spherical-harmonic coefficients) as an anchor plus a residual gated by a per-Gaussian motion confidence q_i, so appearance can only adapt where the deformation field shows genuine local motion. This targets the "appearance shortcut" failure mode.
2. Identification and analysis of two concrete failure modes in 4D reconstruction: the static-background appearance shortcut (changing color c_i to raise PSNR without recovering geometry) and the dynamic-primitive disappearance shortcut (reducing opacity σ_i for fast-moving Gaussians instead of fixing their trajectory).
3. A Continuous Motion Evolution module in which a Motion Network M_θ predicts linear velocity, linear acceleration, angular velocity and angular acceleration, propagated with RK2 integration, supervised by a bidirectional kinematic continuity loss linking acceleration → velocity → displacement across neighboring timestamps.
4. State-of-the-art results on four dynamic scene benchmarks for both geometrically faithful interpolation and continuous motion extrapolation, with a substantial margin in Chamfer Distance on dynamic point clouds.

Main Findings

• Interpolation quality and geometry: On the NVIDIA Dynamic Scene dataset, PhyMo reaches PSNR 31.825, SSIM 0.914 and LPIPS 0.089, the best PSNR in Table 1. On PhysInOne it reaches PSNR 32.631, SSIM 0.971, LPIPS 0.090 and Chamfer Distance 0.046 m, versus 0.496 m for FreeGave (the next best), 0.544 m for 4DGS, 0.641 m for TiNeuVox, 0.678 m for ModecGS and 0.877 m for DefGS.
• Geometry versus appearance decoupling: The large CD gap shows that competing methods can produce plausible images while placing dynamic geometry in the wrong 3D locations — the shortcuts are visible in Chamfer Distance even when PSNR looks reasonable.
• Extrapolation performance: Observing frames in the normalized interval [0, 0.75] and predicting [0.75, 1], PhyMo leads on all four datasets: NVIDIA Dynamic Scene PSNR 30.265 (vs. 29.341 for TRACE and 29.005 for FreeGave), Dynamic Multipart 35.754 (vs. 33.481 for TRACE), FreeGave-GoPro 28.135 (vs. 28.094 for FreeGave), and PhysInOne 29.591 with CD 0.073 m (vs. 0.491 m for TRACE and 0.718 m for FreeGave).
• Ablating motion guidance hurts geometry most: Removing q_i drops interpolation PSNR to 30.164, SSIM to 0.909 and raises CD to 0.540 m (from 0.046 m), and also degrades extrapolation to PSNR 29.016 and CD 0.358 m — confirming that appearance fitting alone falls into shortcut solutions and that bad trajectories propagate into worse motion states.
• Both continuity losses matter for prediction: Removing ℓ_cont1 gives extrapolation PSNR 28.945 and CD 0.079 m; removing ℓ_cont2 gives 27.468 and 0.151 m, against 29.591 and 0.073 m for the full model. Interpolation stays near the full model, indicating the losses mainly buy temporal coherence.
• The Motion Network is essential for extrapolation: Dropping M_θ leaves interpolation nearly intact (PSNR 32.521, CD 0.046 m) but collapses extrapolation to PSNR 23.297 and CD 0.547 m.
• Robustness to the integration step: Using Δt of 1δt, 2δt and 3δt gives interpolation PSNR of 32.498, 32.631 and 32.597 and extrapolation PSNR of 29.411, 29.591 and 29.467, so moderate changes in the temporal interval do not destabilize the formulation.
• Qualitative behavior: In the Balloon scene FreeGave predicts the wrong motion direction while PhyMo follows the upward buoyancy-driven trend; in the Skating scene FreeGave and TRACE produce discontinuous future motion while PhyMo stays smoother; in the Laptop and Skating scenes competitors lose dynamic parts to opacity-driven disappearance, and in the Slope scene competitors fit color but miss the dynamic structure.

Methodology in Plain English

The scene is represented as a set of 3D Gaussians defined at time zero (canonical Gaussians), fitted to all images from all cameras at t = 0 using standard rendering losses.

A deformation field F_defo then predicts, for each Gaussian and any time t, how its position, rotation and scale change. The catch is that rendering loss alone lets the model cheat. To block that, the appearance parameters (opacity and color coefficients) are frozen as anchors, and updates are released only in proportion to a per-Gaussian "motion confidence" gate. That gate is computed from how much a Gaussian's deformation changes between two adjacent timestamps — computed with a stop-gradient sigmoid so it steers appearance without being perturbed by it. Gaussians that are not really moving cannot change their appearance; Gaussians with clear local motion unlock their appearance residuals.

Training proceeds in stages. A canonical warm-up at t = 0 comes first (the first 3,000 iterations in the reported training-dynamics figure), then the deformation field is trained while the supervised time window is expanded gradually rather than introducing every timestamp at once.

On top of the recovered trajectories, a Motion Network takes a Gaussian's position and time and outputs its instantaneous linear velocity, linear acceleration, angular velocity and angular acceleration. For each observed timestamp t, the system samples two neighbors, t_l = t − Δt and t_r = t + Δt, from the learned trajectory, predicts their motion states, and marches both forward to t using a second-order Runge-Kutta (RK2) integrator — one from the left, one from the right. Both resulting Gaussian sets are rendered and compared to the image at t.

Because matching pixels does not uniquely pin down velocity and acceleration, two continuity losses are added: one penalizes mismatch when velocity and acceleration are propagated between t_l and t_r, the other does the same for displacement and velocity. The total objective is the L1 plus SSIM rendering loss plus weighted ℓ_cont1 and ℓ_cont2. Implementation settings reported in the appendix are τ = 0.25, β = 50, q_min = 0.0001 and s_dc = 0.95.

Why This Matters

Impact on research. The paper reframes dynamic reconstruction as an ambiguity problem between appearance and motion rather than purely an optimization problem, and gives a concrete mechanism (motion-gated appearance) plus a concrete diagnostic (Chamfer Distance on dynamic point clouds, which exposes errors that PSNR hides). The bidirectional kinematic continuity constraint offers a general recipe for making physics-style state estimation less under-constrained when only pixels are available.

Real-world applications.

• Embodied AI and robot manipulation, where a robot must predict how objects will move, not just what they look like.
• World models and simulation, where learned dynamics need to stay stable beyond the training window.
• Motion planning and autonomous systems operating around moving people and vehicles.
• Content creation and visual effects, where dynamic 3D assets need correct geometry and physically plausible motion for new viewpoints and future frames.

Industry relevance. The method builds directly on 3D Gaussian Splatting, which is already attractive for its fast rendering, and the reported gains come with a modest architectural addition rather than a new scene representation — a practical profile for teams already using Gaussian-based pipelines in simulation, robotics or media tooling.

Future Directions

• Rigid-particle assumption: Each Gaussian is treated as rigid with fixed scale over time; deformable or non-rigid material behavior is not modeled in the described formulation, which the paper flags for the appendix's limitation section (whose content is not included here).
• Complex real-world physics: Because the authors deliberately use a simplified kinematic formulation instead of explicit governing equations, it is open whether the approach scales to phenomena well beyond the buoyancy, gravity, friction and wind settings in the PhysInOne scenes used here.
• Longer-horizon extrapolation: The evaluation predicts the final quarter of a normalized timeline ([0.75, 1] after observing [0, 0.75]); how far continuous extrapolation can be pushed before RK2 integration error dominates is not reported.
• Choosing the integration interval: Performance is shown to be stable across 1δt, 2δt and 3δt, but the paper does not report a principled way to select or adapt Δt per scene or per motion speed.

Target Audience

Graduate students and researchers in 3D vision working on dynamic scene reconstruction, 4D Gaussian Splatting, or physics-informed learning; practitioners building world models or simulation pipelines who need reliable motion beyond the observed frames; and engineers already using Gaussian-splatting stacks who want to add physically consistent motion without changing their underlying representation. Readers need solid background in NeRF/3DGS and optimization to follow the method sections, though the problem framing and failure-mode analysis are accessible to newcomers.

Authors’ abstract

Recovering dynamic 3D scene geometry, appearance, and the physical motion states that govern scene evolution from multi-view videos is important yet challenging. A key difficulty is that appearance can be deceptive. Under rendering supervision alone, a model may fit the observed images by changing color or opacity to compensate for inaccurate deformation, rather than recovering the true evolving geometry and motion. This issue is especially severe in scenes with fast or locally complex motion, where inaccurate trajectories can still produce plausible images over the observed frames, but lead to unreliable reconstruction and discontinuous future prediction. In this paper, we propose PhyMo, a framework for learning evolving dynamics from multi-view videos. We introduce a Motion-Aware Trajectory Representation that drives the model to recover accurate motion from deformation, rather than relying on appearance shortcuts to explain observed motion. On top of this, we propose a Continuous Motion Evolution module with bidirectional kinematic continuity constraint to promote temporally consistent evolution of displacement, velocity, and acceleration. Together, these designs yield geometrically faithful interpolation and continuous motion extrapolation. Experiments on four dynamic scene benchmarks demonstrate state-of-the-art performance.