DIMO: Diverse 3D Motion Generation for Arbitrary Objects

AI summary

Overview

Research area: Computer vision and generative modeling, specifically 4D (dynamic 3D) content generation from images, with connections to video diffusion models, 3D Gaussian Splatting, and motion latent-space learning.

Technical level: Intermediate. The paper is readable at a high level (image in, many motions out), but the method sections assume familiarity with SE(3) transformations, radial basis functions, linear blend skinning, VAEs, and differentiable rendering.

Scope: The paper proposes DIMO, a per-object generative model that samples diverse 3D motions and 4D content for arbitrary objects instantly from a single image by distilling motion priors from a pretrained video model into a shared low-dimensional motion latent space.

What This Paper Is About

Existing 4D generation methods can produce only one motion per object per expensive inference pass, so getting several different motions for the same object means re-running diffusion and 4D reconstruction repeatedly, which risks losing object identity and costs a lot of compute. Meanwhile, models that do offer rich motion spaces (such as SMPL for humans) depend on category-specific templates built from labor-intensive motion capture and do not scale to general objects. DIMO's goal is to build an SMPL-like, object-specific motion model for any object — one that is trained once and then lets you sample many distinct 3D motions in a single forward pass.

Key Contributions

1. The paper presents what it describes as the first generative approach for diverse 3D motions of any general object from a single-view image, by distilling motion priors from well-trained video models instead of using motion capture.
2. It embeds each motion pattern into a shared latent space and jointly learns the diverse motion distributions, represented as structured neural key point trajectories.
3. At inference it instantly generates diverse 3D motions and 4D contents in a single forward pass, and supports applications including latent-space motion interpolation and language-guided motion generation.
4. It reports state-of-the-art performance across extensive settings and standard 4D generation benchmarks (3D motion generation, image-to-4D, video-to-4D, and text-to-4D).

Main Findings

• 3D motion generation user study (35 participants, four criteria): DIMO scored 80.0% motion diversity (MD), 62.8% image alignment (IA), 68.6% motion quality (MQ), and 60.0% 3D appearance (3D App.), versus DG4D at 11.4%/17.2%/8.6%/14.3% and 4DGen at 8.6%/20.0%/22.8%/25.7%. The study covered 7 cases (Human, Cat, Bird, Robot, and SORA-generated Kangaroo, Otter, Monster), each with five motions.
• Image-to-4D comparison: DIMO reports CLIP-I 0.9505, CLIP-F 0.9912, and 59.4% motion diversity with 10 minutes of training time, compared with Animate124 (0.8596 / 0.9756 / 8.1% / 4.5h), 4DGen (0.9011 / 0.9854 / 5.4% / 1h), STAG4D (0.9281 / 0.9868 / 8.1% / 1.5h), DG4D (0.9214 / 0.9883 / 10.2% / 10min), and SV4D (0.9372 / 0.9904 / 8.1% / 15min).
• Video-to-4D on the Consistent4D benchmark: DIMO reports LPIPS 0.112, CLIP 0.92, and FVD 625.30, compared with Consistent4D (0.160 / 0.87 / 1133.93), STAG4D (0.126 / 0.91 / 992.21), DreamGaussian4D (0.160 / 0.87 / not reported), 4DGen (0.160 / 0.87 / not reported), and SV4D (0.118 / 0.92 / 732.40).
• Text-to-4D comparison (six Animate124 examples, the same 10 GPT4-generated prompts per model): DIMO reports motion amplitude 4.319, text alignment 81.3%, and motion diversity 75.0%, versus Animate124 (0.428 / 6.3% / 9.3%) and 4D-fy (0.254 / 12.5% / 15.6%). The paper states Animate124 takes 5 hours per instance and 4D-fy 12 hours per instance, while DIMO generates text-guided 3D motions within seconds in a single forward pass.
• Speed and scale: rendering runs at 250 FPS at 512×512 resolution with roughly 80k canonical 3D Gaussians. For the 50-motion joint training setting, the first stage takes about 40 minutes on a single 40GB A100 GPU for 2.8k steps, and the second stage takes an additional 3 hours for 8k steps.
• Ablations confirm each component matters: the full model reaches LPIPS 0.126, CLIP 0.93, FVD 587.09, while removing motion factorization gives 0.134 / 0.91 / 851.83, removing the latent space gives 0.163 / 0.87 / 1077.19, removing motion pre-training gives 0.149 / 0.90 / 890.26, and removing multi-motion joint training gives 0.131 / 0.92 / 693.49.
• Applications work as intended: linear interpolation between sampled latent codes yields novel, coherent 3D motions; language prompts projected into the latent space produce feed-forward motion; and unseen test motions can be reconstructed by optimizing a latent code from scratch, converging within 300 steps.
• Diversity of objects tested: experiments span Objaverse, Animate124, Consistent4D, DAVIS, SORA-generated and self-collected data, covering 58 species with over 50 motions each, including humans, robots, bipods, quadrupeds, birds, plants, fluid and deformable objects.

Methodology in Plain English

The starting intuition is that modern video generation models already "know" a lot about how things move, so they can stand in for the motion capture data that category-specific models like SMPL rely on.

Step 1 — Harvest motions from a video model. Given one image of an object, the authors auto-generate many motion descriptions using a fine-tuned Llama3 to write a structured "meta" prompt (appearance, expression, initial action state) and GPT-4o to propose subsequent motions. Those prompts drive a text-conditioned image-to-video model to produce many short videos of the same object doing different things. An MLLM filters out low-quality videos, and clips with too little or too much motion (measured by flow magnitude) are discarded. A multi-view video model supplies novel-view geometric priors.

Step 2 — Represent motion compactly. Instead of training a separate deformation network per clip, each motion is factored into a small set of neural key points with 6DoF transformations over time. Key points live in a canonical space and influence nearby points through radial basis function weights; a motion graph links key points by K-nearest neighbors over their full trajectories so topology is preserved.

Step 3 — Put all motions in one latent space. Every motion gets a latent code drawn from a learned Gaussian per motion, using the VAE reparameterization trick. A single shared latent-conditioned motion decoder (an 8-layer MLP with a skip connection at the 4th layer) takes the latent code, the key point's canonical position, and the timestep, and predicts that key point's 6DoF transformation. Because all motions share the decoder and latent space, the network is forced to learn common motion patterns rather than overfitting one clip.

Step 4 — Attach geometry. Canonical 3D Gaussians (many more than key points) model appearance and shape; they are deformed at each timestep by the nearest key points via linear blend skinning and rendered with differentiable splatting. Training uses only photometric-style losses: RGB, mask, LPIPS, edge-aware depth smoothness, and bilateral normal smoothness, plus an As-Rigid-As-Possible term that keeps nearby key points locally rigid over time, a KL term pulling the motion distributions toward a standard Gaussian, and a Chamfer distance term linking stages.

Step 5 — Train coarse to fine. Stage one learns a coarse motion basis and latent space (512 key points, with densification, pruning, and FPS downsampling as an annealing process). Stage two adds the canonical Gaussians, initializes them in small spheres around key points, recycles stage-one trajectories as guidance, and ramps resolution from 128×128 to 512×512. Each iteration samples 4 motions × 3 views × 2 frames. A separate model is trained per object; the latent dimension is 32, training used 8 views × 21 frames per motion sequence, a 33.9° virtual camera field of view at a fixed radius of 2 m, and N_m ≥ 50 motion prompts per object from Llama3 and GPT4o plus CogVideoX5B-I2V for video generation.

Why This Matters

Research impact. The paper reframes 4D generation from "one prompt, one motion, one expensive optimization" to "one object, one trained model, a whole space of motions you can sample instantly." It also suggests a route toward a general SMPL-like parametric motion model that is not restricted to a specific skeleton or category, and it positions video models as a substitute for motion capture.

Real-world applications (as implied by the paper's capabilities):

• Content creation: turning a single product or character photo into multiple animated 4D clips that can be compared and chosen rather than re-generated.
• Animation and VFX: building per-object motion libraries that can be interpolated to produce in-between or blended motions.
• Interactive and language-driven tools: generating animations directly from short prompts such as "lift the right hand," without waiting hours per instance.
• Motion analysis and reconstruction: recovering plausible motion from held-out video by optimizing a single latent code while the rest of the model stays fixed.

Industry relevance. The reported efficiency numbers are the headline for practitioners: 10 minutes of training per object in the image-to-4D setting, 250 FPS rendering, and seconds-level text-guided generation versus hours for Animate124 and 4D-fy. Combined with the dependence on existing image-to-video models, this makes the approach a candidate for pipelines that already use video diffusion and 3D Gaussian representations.

Future Directions

• Improve the upstream video models. Because DIMO distills motion priors from video models, the authors state that improvements in those models are critical for enhancing DIMO's performance.
• Unify language conditioning with motion learning. Language-guided motion generation is currently learned in two stages (optimizing latent codes and the language projector separately); the authors identify joint single-stage learning as a key direction.
• Move toward a general parametric model. The conclusion frames the work as a first step and hopes to inspire future research on building a general SMPL-like parametric model for arbitrary dynamic objects.
• Handle objects and motions outside the current scope. Since a separate generative model is trained per object and motions originate from generated videos, extending to objects whose motion priors are weakly captured by video models — and reducing reliance on video quality filtering — remains open.

Target Audience

Researchers and graduate students in computer vision, graphics, and generative AI working on 4D generation, dynamic scene reconstruction, or motion synthesis; practitioners in animation, gaming, and content creation who need multiple plausible motions per asset without repeated expensive optimization; and anyone interested in how pretrained video models can be repurposed as motion priors for general objects rather than just category-specific templates. Readers without a graphics background can follow the overview and results, but the method sections require comfort with SE(3) pose parameterization, skinning, and latent-variable models.

Authors’ abstract

We present DIMO, a generative approach capable of generating diverse 3D motions for arbitrary objects from a single image. The core idea of our work is to leverage the rich priors in well-trained video models to extract the common motion patterns and then embed them into a shared low-dimensional latent space. Specifically, we first generate multiple videos of the same object with diverse motions. We then embed each motion into a latent vector and train a shared motion decoder to learn the distribution of motions represented by a structured and compact motion representation, i.e., neural key point trajectories. The canonical 3D Gaussians are then driven by these key points and fused to model the geometry and appearance. During inference time with learned latent space, we can instantly sample diverse 3D motions in a single-forward pass and support several interesting applications including 3D motion interpolation and language-guided motion generation. Our project page is available at https://linzhanm.github.io/dimo.