MoCapAnything: Unified 3D Motion Capture for Arbitrary Skeletons from Monocular Videos

AI summary

Overview

Research area: Computer Vision — monocular 3D motion capture, character animation, and category-agnostic pose estimation.

Technical level: Advanced. The paper assumes familiarity with transformer architectures, graph attention networks, inverse kinematics, and SO(3) rotation representations.

Scope: The paper defines and solves Category-Agnostic Motion Capture (CAMoCap), a task that takes a single monocular video plus an arbitrary rigged 3D asset and outputs a rotation-based animation (e.g., BVH) that directly drives that asset.

What This Paper Is About

Almost all motion capture pipelines are built for one species or one template — human systems regress SMPL-family parameters, animal systems build on SMAL and a handful of quadrupeds. That means creators cannot easily animate a robot, a mech, a toy, a mascot, or an unusual creature without building a new parametric model from scratch. This paper reframes the problem as prompt-based motion capture: given a monocular video and any rigged 3D asset (skeleton, mesh, and optional rendered images) as a prompt, reconstruct the joint rotations that animate that specific asset. The authors call this setting Category-Agnostic Motion Capture (CAMoCap) and note that it covers both motion capture (when the reference matches the video subject) and retargeting (when it does not).

Key Contributions

1. Formalizes CAMoCap, a new task definition for prompt-based 3D motion capture from a monocular video and an arbitrary rigged 3D asset, covering both same-skeleton motion capture and cross-skeleton retargeting.

2. Releases a reorganized Truebones Zoo benchmark of 1,038 motion clips (104,715 frames), each providing a standardized skeleton–mesh–rendered-video triad. 60 clips are held out for testing and 978 are used for training. The test set is stratified into Seen (species with abundant training data), Rare (limited training data), and Unseen (never seen during training) splits.

3. Proposes MoCapAnything, described as the first framework for CAMoCap. It factorizes motion recovery into (i) 3D joint trajectory prediction and (ii) per-joint rotation recovery via constraint-aware IK, and introduces a coarse 4D deforming mesh as an auxiliary modality to bridge dense RGB tokens and the point-cloud-like joint space.

4. Demonstrates strong in-domain accuracy, in-the-wild generalization, and non-trivial cross-species retargeting across heterogeneous rigs, backed by comparisons against adapted baselines and ablations.

Main Findings

• Large margin over adapted baselines. On the Truebones Zoo-test set (MPJPE / MPJVE in cm, lower is better), the method reports 1.06 / 0.44 on Seen, 1.28 / 0.37 on Rare, and 1.76 / 0.36 on Unseen. The strongest baseline, GLoT, reports 3.98 / 1.37 (Seen), 3.58 / 0.84 (Rare), and 7.42 / 2.18 (Unseen). ViTPose, HRNet, and VIBE are also compared, with HRNet at 9.77 / 1.41 (Seen) and 23.53 / 1.84 (Unseen).

• Every removed component hurts, especially on generalization splits. Removing the reference image-set encoder and its cross-attention ("w/o image") gives 1.34 / 0.68 (Seen), 1.56 / 0.44 (Rare), and 2.85 / 0.60 (Unseen). Removing mesh features ("w/o mesh") gives 1.88 / 0.63, 2.25 / 0.39, and 3.16 / 0.44. Disabling graph multi-head attention ("w/o GMHA") gives 1.08 / 0.55, 1.49 / 0.38, and 1.82 / 0.37. The authors conclude that the mesh and graph-attention branches are crucial for transfer to new species.

• A balanced depth configuration wins. Increasing encoder/decoder depth (1/12, 2/12, 4/16) generally helps Rare and Unseen splits, but the chosen 4-encoder / 12-decoder configuration gives the best overall balance — 1.06 / 0.44, 1.28 / 0.37, 1.76 / 0.36 — while a 4/16 variant only slightly improves the Seen split at higher complexity with limited Rare/Unseen gains.

• Lower Chamfer Distance than GenZoo. Because GenZoo does not produce joint-aligned skeletons compatible with MPJPE/MPJVE, the authors compare with CD-Skeleton: 0.2354 (quadruped), 0.2821 (non-quadruped), 0.2549 (all) for MoCapAnything versus 0.4466, 0.4740, and 0.4580 for GenZoo.

• Reasonable rotation accuracy after IK. The supplementary material reports an average geodesic rotation error of approximately 17 degrees after IK fitting.

• Broad in-the-wild robustness. The trained model is applied to internet animal videos including birds (chickens, eagles, seagulls), quadrupeds (tigers, leopards, elephants, cats, dogs), and animals such as crabs, fish, and snakes, with qualitative success in both mocap and retargeting.

• Unconstrained cross-species retargeting emerges without explicit training. Examples include bird videos driving quadrupeds to flap or animating pterosaurs, fish swimming transferred to crocodiles or snakes, dog running animating bipedal birds, and crocodile tail-whipping retargeted to leopards or parrots.

• Metrics and normalization. MPJPE measures mean Euclidean distance between predicted and ground-truth joint positions; MPJVE measures average per-joint velocity difference for temporal consistency. Samples are normalized into a [-1, 1]³ cube for training and rescaled to a unified 1 m³ cube for evaluation, with all metrics in centimeters. Ground-truth meshes are used to compute joint positions in evaluation to avoid interference from predicted mesh quality; visualizations use predicted meshes.

• Evaluation scale. Training and testing also use 1,000 random samples from Objaverse in addition to Truebones Zoo. No quantitative results are reported on Objaverse.

Methodology in Plain English

The authors deliberately avoid regressing joint rotations directly from video. They argue this is brittle because angles depend on the asset's local frames and rest pose, monocular depth and camera motion entangle local rotations, and per-frame angle regression produces poor temporal continuity. Instead, they split the problem in two: predict where the joints go in 3D, then solve for rotations afterward.

Stage one — building a reference prompt. A Reference Prompt Encoder takes the target asset's skeleton, mesh, and rendered image set and turns them into one query vector per joint. Each joint starts from a positional encoding of its rest coordinate plus an optional embedding of its name label. Stacked fusion blocks then refine these queries three ways: graph multi-head attention that biases messages along the kinematic tree using the AnyTop design, cross-attention to sampled mesh surface points with normals (learning implicit skinning-like joint-to-geometry relations), and cross-attention to frozen DINOv2 image features that disambiguate symmetric or visually similar parts. A binary mask zeroes out padded joints so variable joint counts are handled.

Stage two — reading the video. A Video Feature Extractor builds two streams. The visual stream encodes each frame with a frozen DINOv2 image encoder to get dense appearance tokens. The geometry stream runs a pretrained image-to-3D reconstructor to get a coarse deforming surface sequence, then randomly downsamples each frame to 1024 points and embeds position, normal, and time. These 4D mesh tokens act as the bridge between dense RGB features and the point-cloud-like joint space, stabilizing depth and self-occlusion reasoning.

Stage three — fusing and predicting trajectories. A Unified Motion Decoder tiles the per-joint queries across time and refines them through four stages per layer: graph-based self-attention within each frame, cross-attention to a sliding window of video tokens, cross-attention to a matching window of 4D mesh point tokens, and a windowed temporal self-attention per joint that mixes past and future states to reduce jitter. A lightweight MLP head outputs per-frame joint positions. Training uses only a masked L1 position regression loss — no rotation-space or explicit temporal losses — since rotations are recovered afterward.

Stage four — IK fitting. A two-stage inverse kinematics procedure converts trajectories to asset-specific rotations. A closed-form geometric initialization aligns rest-pose bone directions with the observed joint positions along each kinematic chain, respecting hierarchy. A small differentiable IK optimization then minimizes the discrepancy between forward-kinematics-reconstructed joints and predicted 3D positions, regularized toward the geometric initialization and warm-started from the previous frame for temporal stability and twist suppression.

Training setup. 12 decoder layers and a 4-layer prompt encoder, trained on 8 GPUs with 64 GB memory each for 60 epochs using Adam, a learning rate of 1×10⁻⁴, and a batch size of 1 per GPU, requiring approximately 36 hours total. Training uses paired supervision for motion capture only (not retargeting), with a same-species reference asset providing image, unordered mesh, and skeleton as the prompt.

Why This Matters

The paper shifts motion capture from a per-species modeling problem to a prompt-based, asset-agnostic one. If motion can be recovered as 3D trajectories and then projected into whatever rig the user supplies, the same pipeline can drive a heterogeneous library of characters without a bespoke parametric model for each. This is a structural change for how research and production pipelines think about mocap: the output is animation-ready rotation data in the asset's own rig convention rather than a fixed body model's parameters.

Real-world applications:

• Games and crowd scenes — animating large heterogeneous asset libraries where each character may have a different skeleton, without retraining per asset.
• VTuber and virtual production — driving avatars whose topology frequently changes, including non-human mascots and creatures.
• Non-biological rigs — retargeting human or animal motion to robots, mechs, toys, and articulated props, which the paper explicitly names as a practical requirement.
• IP-specific characters — spinning up mascots and creatures without building a new parametric model per species.

Industry relevance: The work comes from Huawei (Huawei International Pte. Ltd. and Huawei Central Media Technology Institute), and the authors frame the problem in terms of practical creator pipelines — content creation, virtual production, and scalable asset animation. The explicit support for BVH-style rotation output and the release of a standardized benchmark point toward production-oriented use rather than purely academic evaluation.

Future Directions

• Physics- and contact-aware IK. The authors note the method operates primarily in camera space without explicit physics or contact reasoning, and propose end-to-end, contact- and physics-aware IK as a next step.
• World-grounded trajectory recovery. Recovering motion in world coordinates rather than camera space remains open.
• Reducing dependence on 4D reconstruction. The pipeline's reliance on a pretrained image-to-3D reconstructor is named as a limitation; video-only geometry priors are proposed as an alternative.
• Broader prompt modalities. Text-only or multimodal prompts beyond rendered images, and extensions to multi-character interaction, are listed as future work.

An additional open question the paper surfaces is retargeting evaluation: the authors state that no directly comparable baselines exist for unconstrained cross-species retargeting, so they rely on extensive qualitative analysis and ablation rather than a quantitative retargeting benchmark.

Target Audience

This paper is most valuable to researchers working on 3D motion capture, category-agnostic pose estimation, and character animation, particularly those interested in moving beyond SMPL- and SMAL-style template models. Technical artists and animation pipeline engineers who need to drive heterogeneous rigs from video will find the task framing and the Truebones Zoo benchmark directly useful. Practitioners in games, virtual production, and VTuber tooling will benefit from the prompt-based formulation, though the method's dependence on a rig with known joint structure and a pretrained image-to-3D reconstructor means it is not a drop-in solution for unrigged assets.

Authors’ abstract

Motion capture now underpins content creation far beyond digital humans, yet most existing pipelines remain species- or template-specific. We formalize this gap as Category-Agnostic Motion Capture (CAMoCap): given a monocular video and an arbitrary rigged 3D asset as a prompt, the goal is to reconstruct a rotation-based animation such as BVH that directly drives the specific asset. We present MoCapAnything, a reference-guided, factorized framework that first predicts 3D joint trajectories and then recovers asset-specific rotations via constraint-aware inverse kinematics. The system contains three learnable modules and a lightweight IK stage: (1) a Reference Prompt Encoder that extracts per-joint queries from the asset's skeleton, mesh, and rendered images; (2) a Video Feature Extractor that computes dense visual descriptors and reconstructs a coarse 4D deforming mesh to bridge the gap between video and joint space; and (3) a Unified Motion Decoder that fuses these cues to produce temporally coherent trajectories. We also curate Truebones Zoo with 1038 motion clips, each providing a standardized skeleton-mesh-render triad. Experiments on both in-domain benchmarks and in-the-wild videos show that MoCapAnything delivers high-quality skeletal animations and exhibits meaningful cross-species retargeting across heterogeneous rigs, enabling scalable, prompt-driven 3D motion capture for arbitrary assets. Project page: https://animotionlab.github.io/MoCapAnything/