Gaussian See, Gaussian Do: Semantic 3D Motion Transfer from Multiview Video

AI summary

Overview

Research area: Computer vision and graphics — specifically 4D content generation, dynamic 3D reconstruction, and motion transfer applied to 3D Gaussian Splatting (3DGS) assets.

Technical level: Advanced. The paper assumes familiarity with latent video diffusion models, condition/embedding inversion, score-matching losses, 3D Gaussian Splatting, Linear Blend Skinning, and neural deformation fields.

Scope: The paper introduces a pipeline and the first benchmark for transferring motion from a multiview source video onto a static 3DGS target object of a different category, without rigging or structural correspondence.

What This Paper Is About

Animating arbitrary 3D objects normally requires a predefined skeleton (rigging), and existing motion-transfer work is largely restricted to human characters or objects with shared kinematic structure. This paper asks whether motion can instead be lifted from multiview video of a source object and applied to a static, unrigged 3D Gaussian Splatting target that may belong to a completely different category — for example, transferring a horse's rearing motion onto a vehicle that lifts its front wheels, or a bird's wing flap onto an elephant cartoon's ears.

The goal is a rig-free, cross-category transfer that preserves the target's identity, appearance and geometry while reproducing the semantics of the source motion.

Key Contributions

1. A new problem setup for semantic 3D motion transfer. Motion from a multiview video of a dynamic source is adapted to a static 3DGS target across vastly different categories, with no predefined skeletal correspondences and no required structural similarity between source and target.

2. A view-aware motion embedding strategy with an anchor-based interpolator. Rather than optimizing a separate motion embedding per source view (which grows linearly in runtime and storage and does not generalize to unseen angles), the method optimizes a fixed set of anchor embeddings and uses spherical linear interpolation (slerp) between the two anchors nearest to a requested viewing angle. This balances global coherence with view-dependent detail and accelerates convergence.

3. A robust 4D reconstruction pipeline. The generated supervision videos are noisy and spatially inconsistent, so the paper introduces an ARAP Rotation mechanism (replacing the MLP-predicted rotation with the rotation that minimizes the same rigid-motion objective, applied during training rather than post-processing) and swaps pixel-wise losses for the LPIPS perceptual loss. Both consolidate imperfect supervision into temporally stable dynamic 3DGS reconstructions.

4. The first benchmark for semantic 3D motion transfer, spanning diverse cross-category scenarios with curated source–target pairs of structurally different 3D objects and tailored metrics for motion fidelity and structural consistency.

Main Findings

• Quantitative superiority over adapted baselines. On the Mini-Mixamo benchmark, the method reaches Motion Fidelity 0.74 versus 0.65 for SC4D Motion Transfer and 0.61 for DreamGaussians4D (DG4D); ground truth is 0.87. On the Cross-Category benchmark it reaches 0.66 versus 0.56 (SC4D) and 0.54 (DG4D).

• Better identity preservation. CLIP-I on Mini-Mixamo: 0.950 for the method, versus 0.931 (DG4D) and 0.888 (SC4D). On Cross-Category: 0.948 versus 0.908 (DG4D) and 0.872 (SC4D).

• CLIP similarity to ground truth (Mini-Mixamo only). 0.963 for the method, 0.945 for DG4D, 0.905 for SC4D, and 1.0 for ground truth. The paper does not report a Cross-Category CLIP Score.

• Anchor interpolation generalizes to unseen views. Computed only on viewing angles unseen during training, the method achieves MSE 0.0028 ± 0.0016 and LPIPS 0.0403 ± 0.0170, beating a "Simple" per-view embedding scheme (MSE 0.0111 ± 0.0034, LPIPS 0.1058 ± 0.0179) and a "Global" single embedding (MSE 0.0069 ± 0.0049, LPIPS 0.0649 ± 0.0306).

• Human evaluators prefer the results. In a side-by-side preference study, the method achieves an average appearance-quality rating of 4.66/5 and is reported as the only one that consistently preserves object identity. For motion quality, it matches SC4D — though SC4D fails to maintain appearance — and surpasses DG4D. An ablation on the right of the figure shows both ARAP rotation and LPIPS loss improve perceived motion realism and appearance fidelity.

• The number of anchors is a trade-off. Fewer anchors speed up convergence by encouraging broader view consistency, but too few — for example a single global embedding — degrade quality and cause motion hallucinations. K = 5 anchors is chosen as the balanced setting (the paper uses 5 anchor embeddings, 16 input views, 3K inversion steps and 5K optimization iterations for 4D reconstruction).

• SDS did not help here. The authors experimented with adding a plain Score Distillation Sampling loss and with Iterative Dataset Update, and report both underperformed compared to their approach.

• Failure modes of the baselines. SC4D captures the motion to some extent but its text-based appearance transfer introduces a large visual discrepancy from the original object; DG4D, despite using the target's original 3DGS, struggles to maintain motion coherence over time.

• Real-world transfer works. Because the video diffusion prior is trained on real-world footage, the method animates 3D assets reconstructed from in-the-wild imagery, not just synthetic assets.

Methodology in Plain English

The method runs in two main stages.

Stage 1 — Extracting motion from the source. The input is a set of N videos of F frames each, filmed from known static camera positions, showing the source object performing a motion. The pipeline builds on the 2D finding that image-to-video diffusion models implicitly separate appearance (controlled by the input image) from motion (controlled by an extra embedding). Following prior work, the authors invert that embedding — optimizing a motion code so that the frozen diffusion model reproduces the input video from noise using the denoising score-matching loss.

The twist is that a code optimized for one viewing angle does not work well at another angle. Instead of training one code per view, they optimize a small fixed number of anchor codes spread evenly across source viewing angles. When a code for a specific angle is needed, they use spherical linear interpolation between the two nearest anchors. Each training iteration samples one view-video, interpolates the code, and backpropagates — so multiple anchors get updated at once. As a side effect, anchors can be interpolated to produce plausible embeddings for views never seen during optimization.

Stage 2 — Applying motion to the target. The static target object is represented as 3D Gaussian Splats. Random cameras are sampled, target frames are rendered, and those renders are fed to the frozen diffusion model conditioned on the interpolated motion embedding to produce target supervision videos. Because embeddings can be interpolated, this works for any number of views.

Consolidation. A set of control points is initialized over the target using furthest point sampling. A time-conditioned MLP predicts, for each control point at each time, a rotation and translation; Gaussians are then deformed by Linear Blend Skinning, with a relaxed Gaussian weighting between a Gaussian and a control point based on distance and a non-learned scalar that modulates each control point's area of effect. The deformation field is optimized so renders match the supervision videos. Because those videos are noisy and spatially inconsistent, two regularizers do the heavy lifting: an ARAP Rotation term that explicitly enforces rigid rotation constraints during training (the standard ARAP loss only regularizes control point positions, leaving predicted rotations unconstrained and producing artifacts), and the LPIPS perceptual loss in place of pixel-wise MSE and mask losses, so the model is not forced to match spatially inconsistent pixels.

Evaluation setup. The benchmark combines curated source videos and static 3D target objects from a subset of Mixamo and from publicly available web assets. The Mixamo-derived portion (Mini-Mixamo) contains paired motions — the same motion performed by both a source and a target character — enabling reference-based evaluation; the paper describes two source figures performing ten distinct motions and ten target figures, each with a static canonical pose and a ground-truth motion execution. All target assets were reconstructed with 3DGS. Metrics are Motion Fidelity (borrowed from 2D video motion work), CLIP-I for appearance preservation, and CLIP similarity to ground truth.

Why This Matters

This work removes two entrenched constraints in 3D animation at once: the need for a rig, and the need for source and target to belong to the same category with a shared kinematic mapping. It also shows that the implicit motion/appearance disentanglement discovered in 2D video diffusion models can be lifted into 3D, and that a single motion code can be reused across multiple target shapes. Because the benchmark it introduces is the first of its kind for this task, it gives the field a shared evaluation protocol where previously none existed — and the paired Mini-Mixamo portion enables ground-truth-based measurement rather than purely subjective assessment.

Real-world applications:

• Gaming and film animation: animating unrigged props, creatures or set pieces by demonstrating desired motion in a reference video.
• Virtual and augmented reality content: populating immersive scenes with moving objects that were captured or generated as static assets.
• Robotics and autonomous-system simulation: generating plausible motion for objects and agents whose behaviour must be simulated without hand-authored rigs.
• Viewpoint densification / novel-view synthesis: the anchor interpolation mechanism produces motion embeddings for unseen camera angles, which the authors note could be useful for view densification.
• Asset reuse pipelines: animation from in-the-wild imagery, which the paper demonstrates by animating 3DGS assets reconstructed from real-world scenes.

Industry relevance: The paper is authored across Technion, NVIDIA, University of Toronto and the Vector Institute, and the method's dependence on a video diffusion prior plus 3DGS places it squarely in the current generative-3D toolchain. The code and benchmark are released under the BigCode OpenRAIL-M v0.1 license, which explicitly prohibits impersonation and deepfakes without consent (clause h) — a deliberate acknowledgement of the misuse surface of realistic motion transfer.

Future Directions

• Reducing runtime. The authors identify runtime as an ongoing challenge: condition inversion is computationally expensive, and while the anchor-based mechanism accelerates convergence, they state further speed improvements are necessary.
• A better 3D semantic motion metric. The paper notes the lack of a robust 3D semantic motion metric and calls for better evaluation tools — the current Motion Fidelity measure was originally proposed for 2D video.
• Folding novel-view synthesis into the pipeline. The anchor mechanism already shows emergent ability to synthesize motion at unseen views for both the source (and, per Appendix D, the target). The authors want to incorporate this capability into the pipeline to further boost performance, and suggest it could serve viewpoint densification.
• Broadening the benchmark. The provided content is truncated mid-discussion of the web-crawled inter- and cross-category dataset, so the full extent of that evaluation set is not reported here.

Target Audience

Researchers and practitioners in generative 3D, 4D content creation, dynamic scene reconstruction, and neural animation — particularly those working with 3D Gaussian Splatting or diffusion-based motion priors. It is also relevant to graphics engineers in gaming, VFX, VR/AR and simulation who need to animate unrigged assets, and to anyone building or evaluating benchmarks for cross-category motion transfer. Readers without background in diffusion models, embedding inversion, or Gaussian Splatting will need to consult the cited preliminaries first.

Authors’ abstract

We present Gaussian See, Gaussian Do, a novel approach for semantic 3D motion transfer from multiview video. Our method enables rig-free, cross-category motion transfer between objects with semantically meaningful correspondence. Building on implicit motion transfer techniques, we extract motion embeddings from source videos via condition inversion, apply them to rendered frames of static target shapes, and use the resulting videos to supervise dynamic 3D Gaussian Splatting reconstruction. Our approach introduces an anchor-based view-aware motion embedding mechanism, ensuring cross-view consistency and accelerating convergence, along with a robust 4D reconstruction pipeline that consolidates noisy supervision videos. We establish the first benchmark for semantic 3D motion transfer and demonstrate superior motion fidelity and structural consistency compared to adapted baselines. Code and data for this paper available at https://gsgd-motiontransfer.github.io/