PhysLDM: Latent Diffusion for High-Fidelity Deformable Simulation

AI summary

Overview

• Research area: Neural physical simulation / computer graphics and physical AI — specifically, learning to predict the dynamics of high-resolution volumetric deformable bodies (tetrahedral meshes) from data.
• Technical level: Advanced. The paper assumes familiarity with variational autoencoders, latent diffusion models, flow matching, differentiable simulation, and finite-element/IPC-style physics solvers.
• Scope (one sentence): The paper introduces PhysLDM, a latent-diffusion framework built on a new holistic spatiotemporal VAE that compresses whole 64-frame volumetric mesh trajectories into a compact latent and predicts them in a single diffusion pass, with zero-shot out-of-distribution generalization and end-to-end differentiability for inverse tasks.

What This Paper Is About

Simulating how soft, deformable 3D objects move and settle is a core problem in graphics and robotics, but classical high-accuracy solvers are slow and hard to differentiate, while neural surrogates struggle with long, high-resolution trajectories. Existing neural simulators mostly predict frame by frame, which lets errors accumulate and drift, and predicting an entire trajectory at native resolution is too expensive. The paper's goal is to build a compact spatiotemporal latent space for mesh-based volumetric physics and then determine whether deterministic regression or generative diffusion is the better predictor inside that space.

Key Contributions

1. A high-fidelity spatiotemporal autoencoder (ST-VAE) for volumetric deformable dynamics. It replaces the block-wise temporal compression used in video VAEs with a holistic temporal scheme that removes "staircase" artifacts, and explicitly decouples global rigid motion from local deformation. This yields roughly 2.48 mm average geometric reconstruction precision on meter-scale scenes at up to 78× token compression.
2. A systematic comparison of deterministic regression versus generative diffusion for physical simulation. The authors show that complex deformable dynamics are chaotic enough that GPU solvers themselves diverge across identical runs, and that in this regime diffusion models the distribution of plausible futures while regression collapses toward damped, non-physical averages.
3. A constitutive-model-agnostic latent diffusion model with strong zero-shot generalization. A single PhysLDM, trained purely on kinematic supervision from an Objaverse-scale simulation dataset, transfers zero-shot to unseen out-of-distribution datasets (GSO and Toys4K).
4. A differentiable simulator for inverse problems and design optimization. PhysLDM supports end-to-end and higher-order differentiation, enabling material (Young's modulus) estimation and a second-order "design the throw" task that the compared differentiable simulators either fail at or handle only at prohibitive cost.

Main Findings

• Holistic temporal compression beats standard video-VAE compression. On held-out Objaverse scenes within the training vertex range, the holistic ST-VAE reaches a deformation-change correlation (DCC) of 0.949 versus 0.665 for standard compression and 0.880 for a direct world-coordinate baseline, while cutting mean reconstruction error to 2.48 mm versus 5.32 mm and 6.50 mm respectively. DCC exceeds 0.93 on all three evaluation sets.
• Decoupling rigid motion from deformation improves precision. The direct world-coordinate baseline, which uses holistic compression but no rigid/deformation factoring, is worse on every deformation and rigid-motion metric and shows noisy geometric detail in visual comparisons.
• Reconstruction holds beyond the training cap. On meshes with 5,000 < N ≤ 10,000 vertices, the holistic ST-VAE still achieves 4.71 mm mean error on held-out Objaverse, 3.39 mm on GSO, and 5.44 mm on Toys4K. A soft mug with 11,621 nodes (more than 2× the training cap) is compressed roughly 182× with only 5.3 mm mean error.
• Physics simulators are not deterministic in this regime. Running the same scene five times from byte-identical inputs produces final poses with vertices up to 442 mm apart. Across 300 Objaverse held-out scenes run twice, 23.7% of final-frame trajectories differ by more than 1% of object size (about 1 cm) and 7.7% by more than 5% (about 5 cm).
• Diffusion outperforms regression and autoregression on physics-consistency checks. On held-out Objaverse scenes the LDM records 7.9% volume violations, 0.24% self-intersecting triangles and 28.2 mm error, versus 22.7%, 0.79% and 37.5 mm for the capacity-matched regression model, and 55.1%, 3.53% and 80.6 mm for the MeshGraphNets-style autoregressive baseline — roughly three times the world-space error of the LDM.
• Diffusion's spread tracks genuine physical ambiguity. In a soft-key drop example, the solver produces two different falling directions across two runs; every diffusion sample follows one of the two directions, while regression predicts an average matching neither.
• The latent space is what makes one-shot prediction feasible. Without it, a same-capacity DiT denoising vertex×frame tokens runs out of memory at 2,600 vertices even at batch 1 on a 140 GiB H200. With it, the training cap on the same GPU rises from about 2,500 vertices to about 14,000 for ST-VAE Base.
• Inference is fast. The LDM runs at 40 to 80 frames per second on a single GPU, including meshes above the training vertex cap.
• Inverse problems are solved cheaply. Material estimation recovers Young's modulus within 0.06 dex in 40 Adam steps (about 40 seconds per run) across seven bodies with moduli from 3×10⁴ to 3×10⁶ Pa, starting up to 1.1 dex away; DiffIPC needs hand-tuned solver parameters and 26 minutes on 48 threads, and Newton/Warp fails in the backward pass. In the second-order throw-design task, 30 Adam steps finish in 7 minutes and make rigid motion 13× more sensitive to stiffness in the ground-truth simulator, while a nested finite-difference workaround for DiffIPC is about 635× slower per gradient step.
• The dedicated rigid pathway in the ST-VAE is necessary. Routing the rigid token through the shared holistic compression instead causes translation error to explode and the reconstructed deformation to shrink drastically.

Methodology in Plain English

The authors treat simulation as trajectory prediction: given a tetrahedral mesh's rest shape, initial pose, initial linear and angular velocity, and material stiffness, predict all vertices over 64 frames.

They first split each frame's motion into two parts using a standard rigid fit (Procrustes/Kabsch): a global rotation and translation, plus a small residual deformation in a canonical frame. This keeps centimeter-scale deformation from being swamped by meter-scale rigid movement.

A three-stage pipeline follows. First, a Spatial VAE compresses each single frame into a fixed number of latent tokens (512 learned query tokens cross-attending the mesh vertices), so the representation does not depend on how many vertices the object has; a decoder plus a shallow graph neural network over the tetrahedral mesh reconstructs vertex displacements. Second, the ST-VAE compresses time: rather than merging fixed blocks of frames (which makes motion hold and then snap, the "staircase"), it prepends a small set of learned queries to each spatial token's temporal sequence and keeps only their outputs. The rigid-motion token takes a separate linear temporal path so it does not dominate the shared latent. The result is one joint latent for the whole trajectory. Third, a latent diffusion model — a diffusion transformer trained with rectified flow / flow matching — generates that latent in one pass, conditioned on the frame-0 mesh (via cross-attention) and on pose, initial velocity and stiffness (via adaptive layer normalization), with the frozen ST-VAE decoder turning latents back into vertices.

Training is purely kinematic: losses are geometric (Charbonnier position reconstruction, KL terms, a latent-anchor loss, and flow-matching velocity MSE). No constitutive/energy model is used at training or inference time. Baselines include standard temporal compression, a direct world-coordinate variant, a capacity-matched deterministic regression model, and a frame-by-frame MeshGraphNets-style autoregressive model.

Why This Matters

The work reframes neural simulation around a representation question (you need a good spatiotemporal latent before one-shot prediction is tractable) and a paradigm question (in chaotic regimes, sampling beats averaging). Showing that a single kinematically trained model generalizes zero-shot to unseen object datasets suggests these surrogates can be scaled without per-object or per-material engineering.

Real-world applications:

• Computer graphics and VFX/game pipelines: fast, high-fidelity prediction of how soft props deform and settle, at interactive frame rates (40–80 frames per second on one GPU).
• Robotics and physical AI: differentiable world models of deformable objects for planning and control.
• Engineering and material characterization: recovering Young's modulus from observed motion, and designing experiments (the "throw") that make a material's stiffness easiest to measure.
• Inverse design and manufacturing: gradient-based optimization through a simulator for soft-body design tasks that are intractable for classical solvers.

Industry relevance: the method is a drop-in fast surrogate for high-accuracy IPC-class solvers, removes the need for hand-derived gradients, and provides higher-order derivatives that existing differentiable simulators (DiffIPC, Newton/Warp) do not offer practically.

Future Directions

• Extending the one-shot 64-frame prediction into an autoregressive framework for longer horizons, which the authors list as their primary future direction.
• Testing the constitutive-model-agnostic claim further on additional energy models and on real captured trajectories (the appendices report initial steps in this direction).
• Improving the least accurate cases — geometries with many long, thin features — which the paper identifies as the hardest held-out reconstructions.
• Scaling to multi-body scenes and higher vertex counts; the paper reports a multi-body extension with up to three objects and notes that the latent shifts the vertex-dependent memory cost into the ST-VAE stage, whose training memory grows linearly with vertex count.

Target Audience

Researchers and graduate students in computer graphics, physics-based animation, neural simulation, and physical AI who already understand VAEs, diffusion models, and mesh-based simulation. The systematic regression-versus-diffusion comparison and the staircase and chaos analyses are also useful to practitioners choosing a predictive paradigm or a compression scheme for spatiotemporal physical data, and to engineers evaluating differentiable simulators for inverse problems.

Authors’ abstract

Neural simulation of high-fidelity deformable bodies is a foundational challenge in computer graphics and physical AI. Long-horizon prediction for high-resolution 3D volumetric meshes is hard: autoregressive methods are prone to error accumulation, while direct multi-frame prediction at native resolution is computationally prohibitive. This motivates a compact spatiotemporal latent representation, which is largely unexplored for mesh-based volumetric physics. Meanwhile, it remains unclear whether deterministic regression or generative diffusion is the more suitable predictive paradigm. To address these coupled challenges, we introduce PhysLDM, a unified latent-diffusion paradigm for one-shot deformable simulation. Its core is a holistic spatiotemporal VAE that avoids the "staircase" artifacts of standard temporal compression (e.g., video VAEs), achieving ~2.48 mm reconstruction precision on meter-scale scenes at up to 78x token compression. Based on this reliable latent space, we systematically compare regression and diffusion methods. Our experiments uncover a key modeling insight: complex deformable dynamics are often chaotic, and in this regime deterministic regression tends to produce non-physical averages, whereas diffusion better models their distribution. Accordingly, we employ a latent diffusion model that effectively learns from chaotic data to generate physically plausible trajectories. Trained purely kinematically on an Objaverse-scale dataset, a single PhysLDM generalizes zero-shot to unseen OOD datasets. Its differentiability further enables efficient solution of inverse problems and higher-order design optimization. To our knowledge, PhysLDM is the first high-fidelity spatiotemporal autoencoder and latent-diffusion paradigm for volumetric deformable dynamics, offering a scalable and robust approach to neural simulation. Project Page: https://yucrazing.github.io/physldm/