BLARM: Animating 3D Objects from Video via Blending Latent Rigid Motion Primitives

AI summary

Overview

• Research area: Computer vision and computer graphics, specifically video-driven 4D mesh animation (animating a static 3D mesh so that its motion follows a monocular video).
• Technical level: Intermediate. The paper combines transformer architectures with classical skinning-based animation, so readers benefit from familiarity with 3D meshes, linear blend skinning, and attention mechanisms.
• Scope: The paper introduces BLARM, a feed-forward method that animates a static mesh from video by predicting a compact set of time-varying latent rigid motions blended by time-invariant vertex-to-component skinning weights, and evaluates it on ActionBench, Motion80, and Consistent4D against three feed-forward mesh animation baselines.

What This Paper Is About

Animating 3D meshes usually requires a rig (skeletons, cages, or hand-authored skinning weights), while newer learning-based methods either infer rigs from geometry alone or directly regress per-vertex motion, which is high-dimensional and hard to interpret. BLARM addresses this gap by learning an intermediate representation: a small set of rigid motion components whose transformations vary over time according to the input video, plus skinning weights that stay fixed on the mesh and decide how each vertex blends those components. The goal is to produce a temporally coherent animated mesh from a monocular video and a static canonical mesh, without any rig or skinning annotations.

Key Contributions

1. A compact latent deformation representation in which learned time-varying motion components are blended by time-invariant vertex-to-component skinning weights, giving a low-dimensional animation space without skeletons, cages, or rig annotations.
2. A feed-forward transformer architecture that conditions geometry-derived deformation latents on video features using factorized attention, capturing interactions among motion components within each frame and temporal consistency across frames.
3. Training losses that favor accurate trajectory reconstruction, sparse per-vertex skinning (entropy regularization), and motion-consistent vertex-to-component assignments (a motion-aware contrastive loss), yielding coherent mesh animations.
4. Empirical results showing better geometric accuracy and temporal consistency than feed-forward vertex-space animation baselines across three benchmarks.

Main Findings

• Best geometric accuracy on ActionBench: BLARM records CD-3D 1.71, CD-4D 3.07, and CD-Motion 7.15, compared with Mesh4D (2.57, 5.01, 9.24), Motion3-to-4 (2.49, 5.34, 10.31), and ActionMesh (3.30, 5.61, 11.12) on 128 16-frame sequences.
• Best appearance metrics on ActionBench: BLARM achieves LPIPS 0.05, CLIP 0.96, FVD 426.72, and DreamSim 0.05, versus Mesh4D (0.06, 0.95, 787.38, 0.08), Motion3-to-4 (0.07, 0.92, 1270.80, 0.12), and ActionMesh (0.08, 0.93, 1126.90, 0.11).
• Best results on Motion80: On 80 variable-length sequences, BLARM reports CD-3D 1.93, CD-4D 3.46, CD-Motion 8.72, LPIPS 0.06, CLIP 0.97, FVD 482.50, and DreamSim 0.05, ahead of Motion3-to-4 (2.25, 5.89, 13.00, 0.07, 0.95, 735.57, 0.06), ActionMesh (2.44, 5.48, 12.78, 0.08, 0.94, 791.80, 0.08), and Mesh4D (5.40, 11.13, 24.26, 0.11, 0.93, 1517.45, 0.11).
• Temporal stability matters: The paper highlights the FVD gain as particularly important because FVD is sensitive to temporal artifacts in rendered videos, indicating that the root-relative rigid components plus time-invariant skinning weights improve visual stability, not just geometric alignment.
• Learned structure looks part-aware: Figure 3 visualizations of predicted skinning weights, with each latent component colored distinctly, align with coherent moving regions such as bending shoe parts and feather groups, even though no explicit rig supervision is used.
• Qualitative robustness: In Figure 2 comparisons on ActionBench, baselines often produce plausible shapes in individual frames but exhibit temporal jitter, part distortion, or inconsistent deformation, while BLARM better preserves the canonical mesh structure.
• Efficiency of the design: Full attention over all T·J spatiotemporal tokens scales as O((TJ)²), whereas the paper's factorized spatial–temporal attention scales as O(TJ²) + O(JT²). Run time is reported in Table 6 in the Appendix (values not included in the provided content).
• Loss weights used: The final objective combines reconstruction, entropy, and contrastive terms with λ_rec = 3, λ_ent = 0.001, and λ_con = 0.3.
• Training scale: Training uses animated object sequences from the Objaverse-1.0 split curated by Liang et al. (2024), after removing objects and sequences appearing in evaluation sets, resulting in approximately 10k shapes rendered at 512 × 512.

Methodology in Plain English

The system takes two inputs: a video of a moving object and one static "canonical" mesh of that object (either ground truth or reconstructed with Trellis2, Xiang et al. 2025). Instead of predicting a new position for every vertex, it predicts a small fixed number J of motion components per frame. The first component is reserved as a "root" for global object motion; the rest describe local motion relative to the root.

To build these components, the method samples surface points and normals from the mesh and passes them through the frozen shape encoder of TripoSG (Li et al. 2025) to get dense geometric features. A set of learnable queries then attends to those features, DETR-style, producing a compact set of geometry-aware deformation tokens — one per motion component. These tokens are broadcast across all frames and given a learnable per-frame embedding.

A stack of motion encoding blocks then refines each token using three operations: cross-attention to per-frame DINOv3 features (Siméoni et al. 2025) so each component sees the video evidence for its frame, spatial self-attention across the components within a frame so they can coordinate, and temporal self-attention along the sequence for each component so its motion stays consistent over time.

Each refined latent is decoded by two small MLP heads into a rotation (a 6D representation converted to a rotation matrix via Gram–Schmidt, Zhou et al. 2019) and a translation, giving a rigid transformation in SE(3). Separately, every vertex gets a skinning weight vector over the non-root components. These weights are predicted from a geometric descriptor (positional encoding of coordinates and normals through an MLP) concatenated with a frozen PartField (Liu et al. 2025c) semantic feature — no discrete part labels or segmentations are used. Vertex descriptors attend to the time-averaged motion latents, and a softmax makes the weights non-negative and normalized.

The final animation comes from linear blend skinning: the non-root transformations are blended with the vertex weights in the root-local frame, then the global root transformation is applied. Because the transformations change per frame while the weights stay attached to the mesh, motion structure and spatial structure are separated. Training combines a focal-style reconstruction loss that emphasizes high-motion-error points, an entropy penalty that pushes each vertex toward few components, and an InfoNCE-style contrastive loss on the skinning weights that uses ground-truth trajectory descriptors (translational direction, rotational direction, motion magnitude, and a static-point indicator) to pull motion-similar vertices together and push motion-dissimilar ones apart.

Why This Matters

• Research impact: The paper proposes a middle ground between geometry-only rig inference and high-dimensional vertex-space regression. It shows that a compact, video-conditioned latent motion basis can be learned without any rig, skeleton, or part supervision, and it provides a training recipe (reconstruction plus entropy plus contrastive losses) for obtaining interpretable, part-aware structure.
• Real-world applications:
  • Animating user-created 3D assets from reference video for games and interactive platforms, matching the industrial affiliations of the author list.
  • Populating virtual worlds and simulations with dynamic characters and props derived from existing static meshes.
  • Film and visual effects pre-visualization, where animators need quick motion previews before committing to full rigging.
  • AR/VR content creation, where an object captured or downloaded once can be driven by a short phone video.
• Industry relevance: Meshes are compact, editable, and broadly supported by rendering, simulation, and animation tools. Because BLARM outputs explicit transformations and skinning weights rather than only deformed vertices, its output stays compatible with standard graphics pipelines and remains editable — a practical advantage over Gaussian-based dynamic representations, which the paper notes are less directly compatible with mesh-based animation and simulation.

Future Directions

• Improving skinning weight quality in hard cases: The authors state that learned weights can be suboptimal, causing nearby or visually similar regions to receive incorrect vertex-to-component assignments and less coherent animation.
• Handling topology constraints: The method assumes the canonical mesh topology can reasonably support the target motion; when it cannot, the predicted deformation may produce artifacts. Robustness to unsuitable topology is left open.
• Evaluating the representation design space: The paper notes that the central contribution is the latent motion components and time-invariant assignments, and that other differentiable skinning formulations such as dual quaternion skinning (Kavan et al. 2007; Kavan et al. 2008) could replace linear blend skinning in Eq. (1).
• Exploring the number and role of components: The method uses a fixed-capacity set of J learned rigid components (with the first reserved as root); how best to size and structure this set for diverse articulated and deformable motions is not resolved here. The truncated content also points to ablations in Appendix A and additional consistent4D results in Appendix B.3 that are referenced but not reproduced above.

Target Audience

This paper is most useful to computer vision and graphics researchers working on 4D generation, video-driven animation, and rigging, as well as technical practitioners in games, AR/VR, and digital content creation who need to animate existing 3D assets from video without hand-authored rigs. Readers already familiar with skinning, transformer attention, and mesh representations will get the most from the architectural and loss-design details; those new to the area can follow the high-level idea that motion is factored into "how the object moves" and "which parts move."

Authors’ abstract

We introduce BLARM, a feed-forward method for video-driven 3D mesh animation. Given a monocular video and a static object mesh, BLARM predicts a temporally coherent animated mesh whose motion follows the video. Rather than relying on explicit rigs or directly regressing high-dimensional vertex motion, we represent animation using a compact set of learned, time-varying rigid motion components and time-invariant vertex-to-component skinning weights. This yields a low-dimensional deformation space without requiring skeletons, cages, skinning weights, or rig annotations. Our architecture conditions geometry-derived deformation latents on video features through factorized spatial-temporal attention, then decodes rigid transformations blended by predicted skinning weights. Trained with trajectory reconstruction, entropy regularization, and motion-aware contrastive learning, BLARM produces accurate and temporally stable animations while recovering compact, interpretable motion structure from monocular video.