Joints, Camera, Action! Co-Generating 3D Motion and Video for Arbitrary Rigs

AI summary

Overview

Research area: Computer vision and generative AI — joint generation of video and 3D skeletal motion (character/rig animation), with camera registration.

Technical level: Advanced.

Scope: The paper introduces JCA ("Joints, Camera, Action!"), a model that co-generates a video, a 3D motion trajectory for an arbitrary user-supplied rig, and a camera that places that motion inside the generated scene.

What This Paper Is About

Video generators can now show convincing motion — a monkey jumping over a crossbar — but they return only pixels, not replayable 3D motion for a specific rigged asset. Lifting motion from the video afterward can introduce errors that drift from the video, and existing joint video–motion models either handle only human skeletons or triangulate 3D motion from multi-view 2D pose maps. JCA instead co-generates video and 3D joint motion directly for rigs of any topology, plus a camera that registers the motion to the depicted scene, so the animation can be replayed on the asset in its 3D scene.

Key Contributions

1. JCA, described by the authors as the first model to directly co-generate video and 3D joint motion for arbitrary rigs, with motion that responds to the surrounding scene, image-keyframe control over the 3D motion, and the ability to generate either modality from the other in the same model.
2. A parameter-efficient architecture that treats a rig as a graph and connects a separately pretrained motion denoiser to a pretrained video backbone through zero-initialized bidirectional cross-attention "bridges," plus bridge guidance at inference that amplifies video–motion agreement.
3. Learned scene registration: an auxiliary joint-localization head predicts where each generated 3D joint appears in the image, and those 2D–3D correspondences are used to fit a camera that places the motion in the scene.
4. A data pipeline that uses a generative video editor to place labeled animated 3D assets into scenes suited to their actions, yielding the "Objaverse in Video" (OiV) dataset of 118K videos from 43K assets, which combined with 60K human and 27K robot clips gives approximately 200K video–motion pairs.

Main Findings

• Strongest prompt adherence in text-driven rig animation: scored 58.4 (0–100, judged by GPT-6 Astra) on 100 asset–prompt pairs, versus 43.4 for ActionMesh, 41.1 for Puppeteer and 20.6 for AnimateAnyMesh. The 100 pairs come from 20 assets sampled from the Puppeteer test set with five prompts per asset generated by GPT-5.6 Sol.
• Lowest mesh distortion: 0.20% median edge-length change, versus 0.36 for Puppeteer, 1.96 for AnimateAnyMesh and 7.07 for ActionMesh.
• Highest dynamic degree: .83, versus .80 (Puppeteer), .50 (ActionMesh) and .11 (AnimateAnyMesh), while motion smoothness is .983 — slightly below AnimateAnyMesh's .993, which the authors note has very subtle motion.
• Human comparison depends on fine-tuning: the general JCA model trails EchoMotion on synchronization (PA-MPJPE 114.8 vs 97.7; SA-MPJPE 151.9 vs 114.9; SA-MPJVE 598.2 vs 431.3), but JCA SMPL-FT — fine-tuned for 10K steps on the human subset — beats it on all three (72.3, 91.0, 272.6), a 26% reduction in PA-MPJPE and 37% in SA-MPJVE. Both variants score better on anatomy (.938 and .934 vs .730) and aesthetic quality (.573 and .578 vs .533).
• EchoMotion's higher dynamics may be camera motion: its estimated non-static camera rate is 74.0%, versus 45.6% (JCA) and 44.8% (JCA SMPL-FT), and its dynamic degree is .828 versus .600 and .488.
• Bridge guidance matters: removing it worsens synchronization from 72.27/90.98/272.58 to 84.78/103.18/293.06 (PA-MPJPE, SA-MPJPE in mm; SA-MPJVE in mm/s), and fully disconnecting the branches degrades it further to 140.71/173.38/456.07.
• Joint training beats motion-only pretraining: prompt adherence rises from 51.6 to 58.4 and dynamic degree from .72 to .83, with low mesh distortion in both cases (0.14% and 0.20%).
• Small parameter overhead: the motion denoiser, rig encoder, localization head and bridges add only 178M parameters, compared with roughly 7B that an EchoMotion-style hybrid design applied to half of the video backbone's 40 blocks would add.
• Camera alignment visibly matters: Figure 5 shows that before alignment the penguin's raw 3D motion sits misaligned with the scene and appears beneath the snow surface; after localization and camera fitting it lands on the visible penguin.

Methodology in Plain English

The authors keep two models rather than building one giant network. A motion model is trained from scratch on motion data alone: it treats a skeleton as a graph (joints as nodes, bones as edges), describes each joint with a learned descriptor built from its name, rest-pose geometry and optional reference-image position, and denoises motion with both global attention across joints and time and graph attention along the bones. Each joint's motion is stored redundantly as a 6D rotation plus 3D position plus their frame-to-frame changes, 18 channels in total.

A pretrained video generator (Wan2.2 I2V-A14B) supplies the visual prior. The two are linked by cross-attention "bridges" placed between each motion block and one video block spread through the backbone, exchanging features in both directions. Because the bridge output projections start at zero, training begins with both pretrained models behaving exactly as they did alone.

At inference, bridge guidance runs the model twice per step — once with bridges off and once on — and extrapolates beyond the connected prediction to strengthen cross-modal agreement; combined with text guidance this needs four forward passes per denoising step.

To place motion in the scene, a localization head lets each joint attend to the video patches of its corresponding frame and returns an image position as a weighted average of patch centers, supervised against 2D annotations. Pairing those 2D predictions with the generated 3D joint positions gives correspondences for fitting one static perspective camera per clip (rotation, translation, focal length, principal point fixed at image center).

For training data, human video supplies rich scenes but one skeleton, while Objaverse assets supply many skeletons but plain renders. The authors render each animated asset simply, use a vision-language model to describe its action and propose a fitting environment, and let a generative video editor (Bernini) insert that environment. They then check pre- and post-edit masks and a vision-language model to reject edits that invalidate the motion, regenerate failures, and require a static camera. Holding one modality clean while denoising the other turns the joint model into video-to-motion or motion-to-video.

Why This Matters

The paper reframes motion generation as something that produces replayable 3D structure, not just pixels. That matters for anyone who wants to drive an actual asset or robot with generated motion, and it shows that a small cross-attention bridge can combine two strong pretrained models far more cheaply than growing a billion-parameter motion branch out of a video backbone.

Real-world applications implied by the demonstrated results:

• Animation and VFX: text- or keyframe-driven animation of rigged characters, with the rig itself shaping generation rather than being fitted afterward.
• Robotics: the paper demonstrates a real Unitree G1 robot rig placed in a 3D scene, jointly generating video and motion from text.
• Synthetic training data: generating paired video and exact 3D motion, useful for perception and motion-learning datasets.
• Previsualization and games: placing an animated asset into a scene and replaying its motion in 3D, or using image keyframes to steer a trajectory.

Industry relevance is direct: the work comes from NVIDIA with Technion and Bar-Ilan, the data pipeline turns existing 3D asset libraries into supervision, and the approach extends a publicly named video backbone rather than requiring a model trained from scratch.

Future Directions

• Moving cameras. Camera fitting currently assumes a static camera at inference; extending it would support more general scenes and would also clarify how much of a baseline's apparent dynamics come from camera motion.
• Broader rig diversity. Training rigs skew toward bipeds — most animated Objaverse characters are bipedal with a limited action range, and the extra motion-only data is humanoid — so transferring the video prior to other topologies remains open.
• Physically grounded motion. The motion branch never sees 3D scene geometry, so nothing enforces contact or prevents penetration; conditioning on geometry or adding contact terms during training or guidance is proposed.
• Objects and multi-rig scenes. Objects the subject interacts with move in the video but are not recovered in 3D, and each clip animates a single rig — lifting both would move the method from animating an asset to animating a scene.

Target Audience

Researchers and practitioners in generative video, 3D character animation and motion synthesis who want generated motion to be usable on real rigs; graphics and robotics engineers who need an animation to be replayed in a 3D scene; and readers interested in architecture-level techniques for coupling separately pretrained generative models efficiently. Comfort with diffusion/flow-matching and transformer terminology helps, though the core ideas are explained through concrete examples.

Authors’ abstract

Video generation models can now depict remarkably realistic motion. Yet it is unclear how to use this motion beyond pixels: to animate a specific 3D asset and replay the animation in its 3D scene. For instance, a video model can show a monkey jumping over a crossbar, but it cannot return that jump as replayable 3D motion placed in the 3D scene. Here we present a model that directly co-generates the video and 3D motion for any supplied rig, along with a camera that registers this motion to the scene. Prior video-motion models either handle only human skeletons, often adding billions of parameters, or triangulate 3D motion from generated 2D pose maps. We instead keep 3D motion in a separate model that supports arbitrary skeletons, and connect it to a pretrained video model through a small set of cross-attention layers. This separation lets the motion model first learn how skeletons move from motion-only data, which is easier to obtain and augment than paired video. Additionally, our model learns to locate each joint in the video frames, which lets us recover the camera and place the 3D motion in the scene. To train across many skeletons, we use a generative video editor to place animated 3D assets in scenes suited to their actions, curating the edits so their known motion remains valid. Together with human and robot video, this yields nearly 200K training pairs. Our evaluations demonstrate that our model animates diverse rigs from text more faithfully than video-first pipelines, and its motion responds to the surrounding scene. It also turns video into rig motion, rig motion into video, and enables keyframe-based control of the 3D motion. Videos and additional results are available on our project page: https://research.nvidia.com/labs/par/projects/jca/.