CloDS: Visual-Only Unsupervised Cloth Dynamics Learning in Unknown Conditions

AI summary

Overview

Research area: Computer vision and computer graphics — specifically differentiable visual computing, physics-informed machine learning, Gaussian splatting, and graph neural network (GNN) based simulation, applied to deformable cloth.

Technical level: Advanced. The paper assumes familiarity with Bayesian filtering, message-passing GNNs, 3D Gaussian Splatting, and mesh-based rendering.

Scope: The paper introduces a new problem setting (Cloth Dynamics Grounding) and a method (Cloth Dynamics Splatting, CloDS) for learning cloth dynamics from multi-view video alone, without physics supervision or known material/environmental properties.

What This Paper Is About

Most deep-learning simulators of dynamic systems need known physical properties — particles, meshes, material parameters — as supervision or inputs, which are unavailable when conditions are unknown. This paper asks whether cloth dynamics can be learned in an unsupervised way purely from multi-view video observations, where the cloth deforms heavily and occludes itself. The proposed answer is CloDS, a three-stage framework that first grounds video frames into 3D meshes and then trains a dynamics model on those grounded meshes.

Key Contributions

1. Introduces Cloth Dynamics Grounding (CDG), described as a novel intuitive-physics scenario for unsupervised learning of cloth dynamics from a series of multi-view videos, where the state space is infinite-dimensional, dynamics are complex, and self-occlusion is severe.

2. Proposes Cloth Dynamics Splatting (CloDS), described by the authors as the first known unsupervised visual-only method for learning cloth dynamics under unknown conditions. CloDS also supports video prediction and novel view synthesis of dynamic scenes.

3. Designs Spatial Mapping Gaussian Splatting (SMGS) with dual-position opacity modulation, a differentiable 2D-to-3D mapping that conditions Gaussian opacity on both world-space (relative) and mesh-space (absolute) coordinates of Gaussian centers, addressing perspective distortion and transparency artifacts from large deformations and self-occlusion.

4. Demonstrates that videos synthesized by CloDS significantly outperform current state-of-the-art video prediction models and that the learned simulator generalizes to unseen configurations.

Main Findings

• CloDS beats a mesh-supervised baseline trained on limited data. In the CDG RMSE table, CloDS consistently outperforms MGN* (the baseline trained on the first 50 mesh trajectories) on both viewed and unviewed trajectories. On viewed interpolation, CloDS reports an average RMSE of 0.1321 ± 0.073 versus 0.1380 ± 0.071 for MGN*.

• With sufficient data, CloDS approaches full mesh supervision. CloDS** (trained on all training videos) reaches performance close to MGN, which relies on full mesh supervision. On viewed interpolation the two averages are 0.1294 ± 0.023 (CloDS**) and 0.1286 ± 0.028 (MGN).

• Limited data causes overfitting. CloDS* performs better than CloDS on viewed trajectories (average interpolation RMSE 0.1309 ± 0.044 versus 0.1321 ± 0.073) but worse on unviewed ones (0.1428 ± 0.037 versus 0.1399 ± 0.046), which the authors attribute to overfitting on seen trajectories given insufficient training data.

• SMGS outperforms Gaussian-splatting baselines in dynamic novel view synthesis. SMGS reaches 36.2368 ± 1.17 PSNR (dB), 9.959 ± 0.06 SSIM ×10, and 3.53 ± 0.21 LPIPS ×1000, ahead of GaMeS (33.0249 ± 1.54, 9.937 ± 0.12, 5.21 ± 0.42), M5D-GS (29.3428 ± 2.35, 9.731 ± 0.13, 12.97 ± 1.04), 4DGS (23.2089 ± 1.86, 9.718 ± 0.06, 15.82 ± 1.27), and MSTH (23.1353 ± 1.85, 9.682 ± 0.08, 16.53 ± 1.32). Per-frame 3DGS, reported as the upper bound, reaches 39.6263 ± 3.17 PSNR.

• CloDS leads on the DVC forward (video prediction) process. CloDS reports 26.6207 ± 0.39 PSNR (dB), 9.817 ± 0.03 SSIM ×10, 0.899 ± 0.04 LPIPS ×100, and 4.78 ± 0.21 RMSE ×100, ahead of SimVP (25.4770 ± 2.04, 9.801 ± 0.07, 1.020 ± 0.09, 5.57 ± 0.45), MMVP (23.9678 ± 1.86, 9.770 ± 0.06, 1.335 ± 0.10, 6.59 ± 0.50), TAU (23.9968 ± 1.88, 9.781 ± 0.06, 1.438 ± 0.10, 6.65 ± 0.51), and MAU (23.7249 ± 1.82, 9.745 ± 0.06, 1.548 ± 0.11, 6.78 ± 0.54).

• Video prediction models accumulate edge errors. The authors report that video prediction baselines accumulate errors over time at cloth edges because of strong self-occlusion, whereas CloDS models the cloth directly in 3D and maintains better consistency in occluded regions.

• Generalization to new shapes and textures. CloDS predicts accurately on a cylindrical cloth shape and maintains strong performance when retrained with modified cloth textures. Additional results on complex lighting are said to be in Appendix H.5.

• CloDS outperforms adapted geometry-aware methods. Compared against LIP (point-cloud representation) and CS (mesh representation), both of which are noted as not fully unsupervised, CloDS substantially outperforms them. LIP's cloth shape collapses rapidly during inference; CS is better than LIP but still falls short of CloDS.

• Robustness across neural simulator backbones. Using MGN, HCMT, DHMP, and BSMS-GNN as backbones, all models trained only on nodes show performance improvement after further training with videos.

• Both opacity coordinates are necessary. Ablations in the mesh-extraction (second) stage show errors accumulate over time without μ^W and μ^M, eventually making the meshes unusable — caused by perspective errors and transparency in unseen regions respectively.

• Real-world grounding is possible but imperfect. Using multi-view videos with cloth regions extracted by SAM, CloDS grounds cloth dynamics from real-world data, though artifacts remain, attributed to camera frame rate limitations and complex real-world lighting. No quantitative real-world metrics are reported.

Methodology in Plain English

The model treats the cloth as a mesh of nodes with world-space 3D coordinates and mesh-space UV coordinates, plus connectivity. The goal is to learn a transition function that predicts the next mesh state from the current one, using only multi-view video frames as input.

Stage 1 — Build the Gaussian representation. The mesh and multi-view images from the first frame are used to construct a Gaussian component representation via SMGS, optimized with the standard 3D Gaussian Splatting loss (an L1 term plus a D-SSIM term, with λ typically 0.2). Unlike NeuroFluid's warm-up phase, which needs particle representations across multiple time steps, CloDS only needs the mesh from the first frame.

Stage 2 — Recover meshes from images. Gaussian components are anchored to mesh faces by barycentric interpolation, so their centers, rotations, and scales follow the mesh as it deforms, preserving temporal correspondence. To extract a mesh at the next time step, the method optimizes a per-step displacement Δx^W_t by backpropagation so that the rendered image matches the observed frame. The loss combines an L1 photometric term with an edge loss that preserves relative node distances and prevents excessive deformation.

Stage 3 — Train the dynamics simulator. The recursively recovered mesh sequence serves as supervision for a GNN (MGN by default), trained with a rollout strategy using a node mean-squared-error loss, with rollout length set to 8.

The distinctive technical piece is dual-position opacity modulation. In standard mesh-anchored Gaussian splatting, opacity is fixed per Gaussian, which fails under cloth's large deformations and self-occlusion. CloDS instead predicts each Gaussian's opacity with a multilayer perceptron that takes both the world-space center (relative position) and the mesh-space center (absolute position). The authors state that removing world-space coordinates reduces the method to standard GaMeS; the relative term suppresses perspective errors and the absolute term prevents the cloth from going transparent when it moves into previously unseen regions. Once trained, the GNN plus SMGS form a forward rendering process that produces continuous video.

Data. The benchmark is generated with Blender on the FLAGSIMPLE dataset by rendering multi-view cloth videos. FLAGSIMPLE's training set has 1000 trajectories of 400 steps each with unique initial states; the authors render multi-view images for 120 trajectories — 100 for training ("viewed") and 20 for testing ("unviewed"). Metrics differ by task: rollout RMSE for CDG, PSNR/SSIM/LPIPS for novel view synthesis, and PSNR/SSIM/LPIPS/RMSE for the DVC forward process.

Why This Matters

Impact on research. The paper positions CDG as an open problem in intuitive physics and pushes beyond rigid-body interaction modeling into deformable continuum mechanics. It offers a template for learning dynamics without physical supervision and shows a mesh-plus-Gaussian representation can serve as the differentiable bridge between pixels and physics. It also extends Differentiable Visual Computing, previously applied to fluids and discrete elements, into thin-structure cloth.

Real-world applications:

• Garment animation and virtual try-on, since the authors state the learned dynamics can be applied to downstream animations of various garments.
• Robotics and manipulation, where physical properties under unknown environmental conditions are inaccessible and only cameras are available.
• Virtual and augmented reality content creation, via dynamic scene novel view synthesis from ordinary multi-view capture.
• Visual effects and digital twins for cloth objects interacting with other bodies, such as the object-cloth collision setup with a moving rigid sphere.

Industry relevance. Any pipeline that needs cloth motion from video without a calibrated simulator — game engines, fashion e-commerce, film production, AR/VR — could benefit. The method also provides a route to transferring simulated mesh data to video-supervised refinement, as shown by fine-tuning MGN* on videos to obtain CloDS.

Future Directions

• Multi-object scenes under unknown conditions. The authors explicitly state that future work aims to explore visual learning of the dynamics of multiple objects in complex scenes under unknown conditions.

• Real-world robustness. The paper reports artifacts attributed to camera frame rate limitations and complex real-world lighting; improving capture consistency and dataset construction for real data is left open.

• Generalization across simulators and materials. The ablation across MGN, HCMT, DHMP, and BSMS-GNN suggests backbone choice matters; how far the framework transfers to other deformable materials or without the rigidity attribute used for collisions is not established.

• Data scaling. The gap between CloDS* and CloDS** indicates sensitivity to training-video quantity and overfitting on seen trajectories; how performance scales with more trajectories and views is not reported.

Target Audience

Researchers and graduate students in computer vision, computer graphics, and physics-informed machine learning who work on dynamic scene reconstruction, Gaussian splatting, neural simulation, or intuitive physics. Practitioners building cloth animation, virtual try-on, or video-based dynamics pipelines will find the three-stage training recipe and the dual-position opacity design most directly useful. Readers without background in GNNs and Gaussian splatting will need supplementary reading, as the paper is written at an advanced technical level.

Authors’ abstract

Deep learning has demonstrated remarkable capabilities in simulating complex dynamic systems. However, existing methods require known physical properties as supervision or inputs, limiting their applicability under unknown conditions. To explore this challenge, we introduce Cloth Dynamics Grounding (CDG), a novel scenario for unsupervised learning of cloth dynamics from multi-view visual observations. We further propose Cloth Dynamics Splatting (CloDS), an unsupervised dynamic learning framework designed for CDG. CloDS adopts a three-stage pipeline that first performs video-to-geometry grounding and then trains a dynamics model on the grounded meshes. To cope with large non-linear deformations and severe self-occlusions during grounding, we introduce a dual-position opacity modulation that supports bidirectional mapping between 2D observations and 3D geometry via mesh-based Gaussian splatting in video-to-geometry grounding stage. It jointly considers the absolute and relative position of Gaussian components. Comprehensive experimental evaluations demonstrate that CloDS effectively learns cloth dynamics from visual data while maintaining strong generalization capabilities for unseen configurations. Our code is available at https://github.com/whynot-zyl/CloDS. Visualization results are available at https://github.com/whynot-zyl/CloDS_video}.%\footnote{As in this example.