Kirin: Animal Motion Generation from In-the-Wild Video

AI summary

Overview

Research area: Computer vision and computer graphics — specifically 3D animal pose/motion reconstruction from video, diffusion-based motion generation, and automatic 3D character animation.

Technical level: Advanced. The work assumes familiarity with parametric quadruped body models (SMAL), optimization-based 4D reconstruction, denoising diffusion models, transformer decoders, and linear blend skinning for mesh animation.

Scope (one sentence): The paper builds AiM3D, a large-scale dataset of reconstructed 3D quadruped motion paired with video and text, and Kirin, a text- and image-conditioned diffusion model that generates animal motion and drives automatically rigged 3D meshes.

What This Paper Is About

Animal motion is far less studied than human motion because most species cannot be brought into motion-capture studios, so existing animal motion data is small, hand-authored, or lacks 3D structure. This paper asks whether realistic, generalizable animal motion models can instead be learned directly from the abundance of in-the-wild animal videos on the Internet. The authors reconstruct 3D motion from such videos, train a generative model on the result, and connect that model to an automatic rigging pipeline so a single image plus a text prompt yields an animated 3D mesh.

Key Contributions

1. AiM3D dataset. The authors enhance state-of-the-art 3D animal reconstruction for video and build what they describe as the first large-scale animal motion dataset with aligned text, video, and motion data — 29,979 reconstructed motion sequences and 179,874 captions spanning 23 quadruped categories (videos drawn from the AiM dataset).
2. First text-and-image conditioned animal motion generation model. An adaptation of MDM that fuses a frozen DistilBERT text encoder and a frozen DINOv3 image encoder into the denoising backbone.
3. Strong empirical results. The model achieves state-of-the-art performance on the AiM3D test set and on an external out-of-distribution test set (AnimalML3D), demonstrating the value of both the dataset and the visual conditioning.
4. A fully automatic image-to-animation system. Combined with an off-the-shelf image-to-3D model (Rodin) and an auto-rigging routine, Kirin turns a 2D image plus text into a ready-to-render animated 3D mesh sequence, which the authors report outperforms existing 4D animation methods.

Main Findings

• Dataset human validation: Of 100 randomly selected test-split samples, 86 reconstructions were judged satisfactory (physically plausible, smooth, no unnatural poses or jitter). The average caption quality score was 4.62 on a 0–5 protocol. No sample scored 0 or 1; 3 samples contained some captions with an incorrect action

Authors’ abstract

Understanding animal motion is fundamental to modeling animal behavior and biomechanics, yet progress in this area lags far behind human motion research due to the scarcity of high-quality motion data. While human motion can be captured in controlled environments, it is impractical for most animal species, resulting in small, domain-limited datasets that restrict downstream applications such as animation. To address this challenge, we introduce Kirin, a framework that reconstructs motion from video, learns motion priors at scale, and generates realistic motion that can be directly applied to animated assets. Using large collections of in-the-wild animal videos, we reconstruct 3D motion sequences and pair them with captions to create AiM3D, the first large-scale dataset offering aligned video-text-motion tuples for quadruped animals. Building on this dataset, we develop a visual-guided motion generation model that conditions on both text and image to guide the generation of realistic motion across diverse animal species. Finally, by leveraging an off-the-shelf image-to-3D model, we automatically rig and animate 3D meshes using generated motion, producing ready-to-render animated animals. Together, our dataset and framework establish a new foundation for large-scale, text and image conditioned animal motion generation and animation. Project page: https://kirin-ani.github.io/.