RASA: Disentangled Spatial-Motional Priors for Cross-Identity Character Animation

AI summary

Overview

Research area: Computer vision — diffusion-based human video generation, specifically pose-driven character animation.

Technical level: Advanced. The paper assumes familiarity with latent diffusion models, Diffusion Transformers (DiTs), flow matching, SMPL body models, and attention mechanisms.

Scope: A single framework (RASA) that decouples 2D spatial alignment from 3D motion control in order to animate a reference character with a driving video's motion across differing identities, plus a companion benchmark (CIM-Bench) for evaluating that setting.

What This Paper Is About

Cross-identity character animation means taking a reference image of one character and driving it with the motion of a different person or character from a video. The core problem is that the two subjects rarely line up: they differ in scale, global position, and body proportions, a condition the authors call pose–reference misalignment. The goal of RASA is to neutralize those geometric mismatches at the start of generation while still transferring accurate, physically plausible motion.

Key Contributions

1. RASA, a hierarchical framework for cross-identity character animation that systematically disentangles spatial priors (where and how big the character is) from motional priors (how the body articulates).
2. A dual-injection mechanism. The Spatial Prior Calibrator (SPC) performs continuous 2D spatial calibration at each denoising step, while the Inherent Motional Guider (IMG) injects identity-agnostic 3D semantic guidance into the intermediate layers of the DiT backbone.
3. CIM-Bench, a standardized benchmark for cross-identity animation, built via a high-fidelity motion retargeting pipeline and manual curation, containing 656 high-quality character videos paired with pose sequences.
4. A shape-agnostic motion representation that discards subject-specific SMPL shape parameters (β) and builds the motion vector solely from articulation parameters (θ), keeping the motion signal independent of the character's geometry.

Main Findings

• Best image-level scores on TikTok. Under the self-driven protocol on the TikTok dataset, RASA reaches PSNR 22.03, SSIM 0.830, LPIPS 0.189, and FID 18.93, the best across all image metrics in Table 1.
• Competitive despite a smaller backbone. MTVCrafter achieves lower FVD (140.60 vs. RASA's 190.82) and Champ reports 160.82, but the authors note MTVCrafter is built on the large-scale CogVideoX-5B backbone and Champ requires five distinct motion representations as conditioning signals. RASA uses a 1.3B backbone with only 2D poses and motion parameters as conditions, and still surpasses larger models such as One-to-All-14B (PSNR 18.07, SSIM 0.812) and Human-DiT (PSNR 20.50, SSIM 0.815) on multiple metrics.
• Clear wins on CIM-Bench. On the more challenging cross-identity benchmark, RASA leads all compared methods: PSNR 15.93, SSIM 0.472, LPIPS 0.372, FID 60.96, FVD 667.96, FID-VID 17.63, versus Animate-X (14.05, 0.426, 0.474, 105.14, 1312.60, 75.47), MTVCrafter (14.19, 0.431, 0.457, 85.21, 1723.89, 85.21), Wan-Animate (11.29, 0.372, 0.585, 178.58, 1920.88, 137.05), and One-to-All-1.3B (13.75, 0.383, 0.471, 96.70, 1324.74, 76.79).
• Stronger identity preservation and efficiency. RASA records Sim-Arc 0.72 and Face-FID 90.63, and runs at 61 s latency with 11,951 MiB VRAM, compared with 109 s / 14,263 MiB for Animate-X, 170 s / 20,680 MiB for MTVCrafter, 327 s / 42,291 MiB for Wan-Animate, and 91 s / 21,880 MiB for One-to-All-1.3B.
• SPC alone helps substantially. Adding frame-wise cross-attention to the synthetic misalignment strategy improves TikTok PSNR from 18.90 to 20.86 and reduces FID from 23.04 to 22.01, and resolves spatial dislocations and proportion errors in the qualitative results.
• IMG alone is insufficient but highly complementary. Using IMG without spatial conditioning produces the worst image fidelity (PSNR 15.47 on TikTok). Combined with SPC, it drives temporal metrics sharply: FVD falls from 289.16 to 190.82 and FID-VID from 6.56 to 3.32 on TikTok.
• Attention direction matters. Swapping the SPC query/key roles so the static reference latent acts as the query suppresses dynamic motion cues; keeping the temporally evolving driving latent as the query preserves motion transfer.
• Low inference overhead. SPC and IMG add only 19.78M parameters (under 1.52% of the 1.3B backbone) and 90.29G and 0.17G FLOPs respectively, at a cost of roughly 2 seconds of latency (59 s to 61 s), measured on a single NVIDIA H100 for a 41-frame video at 832×480.

Methodology in Plain English

The system is built on Wan2.1 (1.3B), an open-source text-to-video diffusion backbone, and adds two conditioning paths.

The reference image is encoded into a latent and concatenated with the noisy video latents along the temporal dimension, with a large positional-encoding offset so the model can attend to the reference's appearance without confusing it with the moving frames.

For geometry, the authors deliberately train under mismatch. They take driving poses and apply random spatial perturbations — stochastic scaling, translation, and proportional skeletal dropout — producing a distorted pose sequence. The Spatial Prior Calibrator then learns to map this distorted pose back into a reference-consistent structure. It encodes both the reference pose and the perturbed driving pose with a shared lightweight encoder, then runs frame-wise spatial cross-attention where the driving latent supplies the queries and the reference latent supplies keys and values, with 2D Rotary Positional Encoding to preserve skeletal topology. The aligned features are added element-wise to the noisy latent at the start of every denoising step, so the correction is applied continuously rather than once.

For motion, 2D keypoints alone are view-dependent and ambiguous about depth. The Inherent Motional Guider instead uses ScoreHMR to estimate 3D human mesh parameters from the driving video, throws away the subject-specific shape parameters, and keeps only articulation parameters. These are packed into a 263-dimensional motion vector covering root-relative joint rotations, velocities, and foot contact signals. A pretrained motion encoder projects this into a latent, which is injected into the intermediate DiT blocks via 14 learnable motion projection layers — specifically blocks 2 through 15. The idea is that the early spatial calibration fixes coarse 2D layout while the intermediate injection supplies depth-aware, anatomically grounded refinement.

Training used TikTok, Champ, and UBC datasets plus roughly 5,000 human-centric videos collected from the internet, resized to 832×480 with poses extracted by DWPose. Clips of 41 consecutive frames were sampled with the first frame as the reference. Training ran on 8 NVIDIA A100 GPUs with AdamW at a learning rate of 1×10⁻⁵ for approximately two days.

For evaluation, the authors built CIM-Bench by generating stylized character identities with a text-to-image diffusion model, pairing them with real human portraits collected online, and using a commercial video synthesis backbone to make videos of different characters performing the same motion. Because pose estimation is more reliable on real human video, poses were extracted there and applied to the stylized character videos, creating authentic structural discrepancies. Manual filtering removed samples where DWPose produced unclear or temporally inconsistent keypoints.

Why This Matters

Existing animation pipelines largely assume the driver and the reference character are geometrically compatible. When they are not, prior fixes — explicit pose retargeting, pose augmentation, or late-stage feature fusion — either lose information or entangle structural calibration with appearance rendering, producing artifacts and unstable motion. RASA shows that separating these concerns yields better results even on a small 1.3B backbone, and CIM-Bench gives the field a realistic testbed instead of the synthetic-perturbation approach used by benchmarks like MisAlign100, which applies random linear transformations to poses extracted from the same subject and video as the reference.

Real-world applications include:

• Digital human synthesis — generating animated avatars that follow a performer's motion.
• Virtual production — driving a pre-designed character with an actor's movement for film and broadcast.
• Interactive media — real-time or near-real-time animated characters in games and social applications.
• Stylized character animation — transferring motion from real performers onto illustrated or synthetic identities, a case explicitly covered by CIM-Bench.

Industry relevance comes from the efficiency profile: 61 s latency and 11,951 MiB VRAM for a 41-frame 832×480 clip on a single H100, with under 1.52% added parameters to a 1.3B backbone, is far cheaper than the 327 s and 42,291 MiB required by Wan-Animate. That matters for any deployment where inference cost constrains product design.

Future Directions

• Topology-aware animation. RASA assumes a 1:1 body-part correspondence, so it fails on non-humanoid characters with fundamentally different topologies such as multi-headed, multi-armed, or limb-less figures, where reliable spatial correspondences cannot be established.
• Explicit face and hand modeling. The framework focuses on full-body animation, and hand and facial motions rely solely on pose representations. Because SMPL does not model facial expressions or hand articulations, their quality is constrained by the estimated 2D poses. The authors propose SMPL-X and MANO as next steps.
• Data scaling for robustness. The authors state that collecting more diverse, high-quality data is a priority for improving generalization.
• Closing the backbone gap. RASA trails MTVCrafter on TikTok FVD (190.82 vs. 140.60), a gap the authors attribute to MTVCrafter's CogVideoX-5B scale, leaving open the question of how much further the disentangled design could gain on a larger backbone.

Target Audience

Researchers and engineers working on diffusion-based video generation, pose-driven character animation, and digital human synthesis. The paper is most useful to readers already comfortable with Diffusion Transformers, flow matching, and SMPL-based motion representations who want to understand how structural misalignment can be handled at the latent level rather than through preprocessing. Practitioners building avatar or virtual production systems will find the efficiency numbers and the CIM-Bench protocol directly relevant.

Authors’ abstract

Cross-identity character animation aims to drive a target identity from a reference image to follow the motion of a source character from a driving video. The core challenge lies in the inherent entanglement of two capabilities: cross-identity spatial mapping (aligning position, scale, and skeletal proportions) and motion control (refining joint articulation, volumetric consistency, and view coherence). We introduce Reference-Aware Structural Alignment (RASA), a framework that disentangles spatial mapping from motion control by injecting structured priors into a Diffusion Transformer (DiT). Our approach has two stages. First, a Spatial Prior Calibrator (SPC) fuses reference identity with driving pose to generate a spatially grounded initial noise latent, ensuring correct positioning, scaling, and alignment with the driving skeleton. Second, an Inherent Motional Guider (IMG) encodes shape-agnostic SMPL articulation parameters into a semantic motion vector beyond appearance-biased 2D keypoints. Injected into intermediate DiT layers, this vector complements the base pose condition for anatomically consistent articulation and view-aware volumetric refinement. We curate CIM-Bench, a high-quality benchmark with rigorous curation, for evaluation. Extensive experiments show RASA significantly outperforms state-of-the-art methods in motion fidelity and visual quality. Our work establishes a new paradigm showing disentangled spatial and motional priors are key to robust character animation. Project page: https://hidream.ai.github.io/RASA/