Animus3D: Text-driven 3D Animation via Motion Score Distillation

AI summary

Overview

Research area: Computer vision and graphics — text-driven 3D animation, diffusion-model-based optimization, and generative 3D representations (Gaussian Splatting).

Technical level: Advanced. The paper assumes familiarity with diffusion models, score distillation sampling (SDS), denoising diffusion implicit models (DDIM), and 3D Gaussian Splatting.

Scope: The paper proposes a new score-distillation objective, Motion Score Distillation, plus regularization and refinement modules, to animate a static 3D asset into a moving, text-specified animation while preserving its original appearance.

What This Paper Is About

Given a static 3D object and a text prompt describing motion (e.g., "a walking lion"), the paper aims to automatically produce an animated version of that object. Existing methods distill motion from pretrained text-to-video diffusion models using the standard SDS objective, but this produces either barely perceptible motion or jittery, distorted results that damage the object's appearance. Animus3D reformulates the distillation objective specifically for motion so that the optimization moves from a well-defined static starting point toward a dynamic target, rather than from pure noise.

Key Contributions

1. The Animus3D framework — an end-to-end text-driven 3D animation pipeline that takes a static 3D Gaussian Splatting asset and a motion prompt, and outputs a temporally coherent animated sequence of Gaussians with preserved appearance.

2. Motion Score Distillation (MSD) — a new distillation objective that replaces SDS's "predicted noise minus random Gaussian noise" gradient with the difference between a dynamic noise prediction and a static noise prediction. The static distribution is modeled by a LoRA-enhanced video diffusion model fine-tuned on renders of the frozen canonical object, establishing an explicit static source distribution.

3. Appearance-preserving faithful noise estimation — instead of sampling stochastic noise, the method uses DDIM inversion to obtain deterministic noise estimates for both the static and dynamic branches. This directly targets "motion-appearance entanglement," the problem where SDS's gradient, which entangles motion and appearance, forces appearance loss to be absorbed as geometric distortion because only the motion field is optimized.

4. Temporal and spatial motion regularization — a TV-3D loss penalizing large per-Gaussian positional changes between consecutive frames, and an as-rigid-as-possible (ARAP) loss encouraging locally rigid deformations, both operating directly in 3D Gaussian space.

5. A motion refinement module — a post-hoc stage that interpolates the frame count (T → 2T−1), renders the animated asset, and refines the video with a larger pretrained rectified-flow text-to-video model using an SDEdit-style noising/denoising loop, then back-propagates an L1 loss into the motion field. This overcomes the fixed temporal resolution of the underlying video diffusion model and yields longer, finer-grained motion.

Main Findings

• Quantitative gains over baselines: Compared with AYG and TC4D, Animus3D achieves the best scores on CLIP-Image (93.04 vs. 91.75 / 90.99), CLIP-Text (51.05 vs. 44.31 / 50.02), FID (88.50 vs. 105.33 / 179.14), and FVD (204.1 vs. 647.6 / 340.0). Notably, the CLIP-Image score — intended to measure appearance preservation — improves while motion also increases, suggesting the method avoids the usual fidelity-versus-motion tradeoff.

• Strong user preference: In a 17-participant study, Animus3D was preferred for overall quality (65.9%), appearance preservation (69.5%), motion dynamism (75.5%), motion-text alignment (55.3%), and motion realism (70.1%).

• Baselines fail in characteristic ways: AYG produces small, flickering motions (attributed to a simple SDS variant); TC4D produces mostly trajectory-based translation with stiff skeleton movement plus noticeable distortion and appearance change.

• Both MSD components are necessary: Ablations show that removing dual distribution modeling makes large motion difficult to produce, while removing faithful (DDIM-inversion) noise causes severe appearance degradation and background artifacts. Plain SDS either yields negligible motion at low CFG (7.5) or meaningless motion and heavy distortion at high CFG (100).

• A static text prompt is not enough: Conditioning the video model on a "static, not moving" prompt does not reliably produce truly static video and introduces local distortions; the LoRA-fine-tuned static denoiser is required.

• Both regularizers matter: Removing TV-3D causes large frame-to-frame displacements; removing ARAP causes non-rigid distortion, with objects sometimes breaking apart.

• Motion refinement smooths fixed-length artifacts: Because the base video model has a fixed frame count, the original animation has discontinuous motion across the loop boundary; refinement converts this into longer, smoother sequences.

• Scope of generalization: The method handles prompts like "playing," "walking," "flying," and "swimming," and works on general objects (demonstrated on the red flag of Hong Kong), not just animals.

Methodology in Plain English

The pipeline starts with a static 3D object already reconstructed as a set of 3D Gaussians (the "canonical space"). This is frozen and never modified. A separate motion field, implemented as a multi-resolution HexPlane with an MLP decoder, is queried at each timestamp and outputs per-Gaussian offsets in position and rotation, producing an animated sequence of Gaussians that can be rendered into a video from any camera.

The core trick is how the animation is supervised. In standard SDS, you render an image, add random noise, ask a diffusion model to predict that noise, and use the difference as a gradient — implicitly transporting the rendered distribution toward a text-described target. The paper argues this is a poor fit for motion, because there is no explicit static starting point and because the gradient entangles motion with appearance, so any appearance drift gets paid for in geometric distortion.

MSD fixes both issues. First, it defines the starting point explicitly: a video diffusion model is lightly fine-tuned with LoRA on renders of the frozen static object, producing a denoiser that genuinely models "this object, not moving." The motion gradient is then the difference between what the dynamic (text-conditioned) denoiser predicts and what this static denoiser predicts — a direct estimate of the motion direction. Second, instead of sampling random noise at each step, both branches use DDIM inversion, which produces deterministic noise consistent with the actual rendered content, so the gradient carries motion signal without corrupting appearance.

On top of this gradient, two regularizers constrain the motion field: TV-3D keeps each Gaussian's position from jumping between adjacent frames, and ARAP keeps neighboring Gaussians moving in a locally rigid way, preventing stretching and tearing. Finally, a refinement stage renders a higher-frame-count version of the animation, passes it through a bigger pretrained video model in the style of SDEdit (add noise, denoise), and fits the motion field to the refined video with an L1 loss, yielding longer and smoother motion than the base model's fixed frame budget allows.

The system is built on threestudio with ModelScopeT2V as the base video model, renders at 256 resolution with 16 frames, and runs on a single 24GB GPU: roughly 3k LoRA iterations, 5k motion distillation iterations, and 100 refinement iterations.

Why This Matters

Research impact. The paper reframes SDS for motion as an explicit two-distribution transport problem (static source, dynamic target), rather than the single-distribution formulation inherited from text-to-3D. This connects to recent theoretical analyses of SDS as optimal transport and demonstrates that the choice of source distribution — not just the target — materially determines what the optimization can produce. The "motion-appearance entanglement" diagnosis and the deterministic-noise remedy are transferable insights for other SDS-based motion and editing tasks.

Real-world applications.

• Game and film asset animation, where studios have static 3D libraries and want quick text-specified motion drafts without rigging or motion capture.
• E-commerce and product visualization, generating animated turntables or demonstrations from a single static scan or CAD-derived reconstruction.
• Simulation and embodied-AI content pipelines needing diverse animated 3D assets at low cost.
• AR/VR content creation, where short animated props and characters are needed with minimal authoring effort.

Industry relevance. The method requires no multi-view dynamic training data and no motion-capture footage — only a static asset plus a pretrained video model — which fits into existing 3D reconstruction and content pipelines. It runs on a single 24GB GPU, making it accessible to small studios. The main commercial barrier is speed: score distillation methods take hours per asset.

Future Directions

• Generating new content, not just deforming existing geometry. The stated failure case is that the model cannot produce fluid ejected from inside a rocket; it distorts the object instead. Particle generation and modeling strategies are suggested as a remedy.

• Reducing optimization cost. Several hours per asset is a known weakness of score distillation. Amortized training and more efficient spatial data structures are proposed as mitigations.

• Generalizing the static distribution model. The LoRA is fine-tuned per object on its own static renders. Whether this can be amortized or replaced with a more general static prior is an open question.

• Rethinking the refinement stage. The refinement currently relies on a larger pretrained video model and an SDEdit-style loop. Making it more principled, or integrating it into the distillation objective rather than applying it post-hoc, could yield further gains.

• Extending beyond a single object. The framework animates one canonical asset; handling multi-object scenes, interactions, and articulated composition remains unexplored.

Target Audience

Researchers and graduate students working on diffusion-based 3D generation, score distillation, and 4D content creation will get the most from the technical content, particularly the reformulation of SDS and the appearance-preservation argument. Practitioners in graphics, VFX, and game development building asset-animation pipelines will find the framework-level design and its single-GPU feasibility relevant. Readers without background in diffusion models or Gaussian Splatting will find the method sections dense, though the motivation and results sections remain accessible.

Authors’ abstract

We present Animus3D, a text-driven 3D animation framework that generates motion field given a static 3D asset and text prompt. Previous methods mostly leverage the vanilla Score Distillation Sampling (SDS) objective to distill motion from pretrained text-to-video diffusion, leading to animations with minimal movement or noticeable jitter. To address this, our approach introduces a novel SDS alternative, Motion Score Distillation (MSD). Specifically, we introduce a LoRA-enhanced video diffusion model that defines a static source distribution rather than pure noise as in SDS, while another inversion-based noise estimation technique ensures appearance preservation when guiding motion. To further improve motion fidelity, we incorporate explicit temporal and spatial regularization terms that mitigate geometric distortions across time and space. Additionally, we propose a motion refinement module to upscale the temporal resolution and enhance fine-grained details, overcoming the fixed-resolution constraints of the underlying video model. Extensive experiments demonstrate that Animus3D successfully animates static 3D assets from diverse text prompts, generating significantly more substantial and detailed motion than state-of-the-art baselines while maintaining high visual integrity. Code will be released at https://qiisun.github.io/animus3d_page.