HairWeaver: Few-Shot Photorealistic Hair Motion Synthesis with Sim-to-Real Guided Video Diffusion

AI summary

Overview

Research area: Computer vision and generative AI, specifically image-to-video diffusion models for human animation, with a focus on secondary dynamics (hair).

Technical level: Advanced. The paper assumes familiarity with latent diffusion, diffusion transformers (DiTs), LoRA adapters, VAEs, computer-graphics rendering (Blender Cycles), and physics-based hair simulation.

Scope: The paper presents a diffusion-based pipeline that animates a single human image with controllable, physically plausible hair motion, trained on a small synthetic dataset generated by a CG simulator and transferred to photorealistic domains.

Paper details: Authored by Di Chang, Ji Hou, Aljaz Bozic, Assaf Neuberger, Felix Juefei-Xu, Olivier Maury, Gene Wei-Chin Lin, Tuur Stuyck, Doug Roble, Mohammad Soleymani, and Stephane Grabli, with affiliations at Meta and the University of Southern California. The arXiv content header gives the title as "HairWeaver: Photorealistic Hair Motion Synthesis with Sim-to-Real Physics Transfer Guided Video Diffusion" (arXiv:2602.11117v2 [cs.CV]); the submitted metadata adds "Few-Shot" and "Guided Video Diffusion" to the title. The introduction states the model is trained on "only a few samples (around 1k videos)."

What This Paper Is About

Existing human image animation methods control body pose well, but they treat hair as a static texture attached to the head, producing stiff, fluid-like, or unnatural hair motion. HairWeaver's goal is to give users fine-grained control over hair dynamics while still generating photorealistic, identity-preserving video of a person from a single reference image. The authors achieve this by simulating physically accurate hair motion in a computer-graphics simulator, then transferring that motion control into a large pre-trained photorealistic video diffusion model without carrying over the synthetic look.

Key Contributions

1. A diffusion-based pipeline, HairWeaver, designed to synthesize expressive and dynamic hair motion for sim-to-real physics-transfer human video animation, guided by a synthetic motion signal from a simulator used as training data.

2. A Motion-Context-LoRA, described as an efficient and lightweight module that injects hair (and body) motion control as additional attention context, preserving the generative power of the frozen video diffusion backbone.

3. A two-stage training strategy using a temporary Style-Alignment-LoRA that first adapts the diffusion transformer (DiT) to the synthetic CG domain, then is frozen while the Motion-Context-LoRA learns the mapping from motion conditions to video, and is discarded entirely at inference to keep the output photorealistic.

4. A synthetic dataset generated with physics-based simulation and rendering, providing paired ground-truth motion conditions (body normal maps with 68 facial landmarks, and UVW hair maps) alongside rendered video, which the authors argue is the missing ingredient in prior approaches.

Main Findings

• State-of-the-art results on the CG hair test set: On the self-collected hair motion test set, HairWeaver achieves Hair SSIM 0.9794, PSNR 37.6347, LPIPS 0.0233, FID 50.5938, cd-FVD 434.1582, and Full Body SSIM 0.8948, PSNR 27.7903, LPIPS 0.1127, FID 43.2786, cd-FVD 407.1929. The next-best Full Body SSIM among the compared baselines is UniAnimate-DiT at 0.8724, and the next-best Hair PSNR is UniAnimate-DiT at 35.4174.

• State-of-the-art results on the photorealistic NeRSemble test set: HairWeaver reports Hair SSIM 0.9670, PSNR 34.34, LPIPS 0.0477, FID 17.79, cd-FVD 286.25, and Portrait SSIM 0.8291, PSNR 26.47, LPIPS 0.1763, FID 19.43, cd-FVD 212.61. For comparison, Wan-2.2-Animate-14B reports Hair PSNR 28.75 and Portrait PSNR 23.62, and Wan-2.2-14B reports Hair FID 113.11.

• Efficiency advantage over animation baselines: Average inference time per video sample for HairWeaver is 62 seconds, compared with 870 seconds for UniAnimate-DiT, 476 seconds for Wan-2.2-14B, 314 seconds for Wan-VACE with hair and body conditions, and 312 seconds for Wan-2.2-Animate-14B. The two LTX-Video variants are somewhat faster than HairWeaver, at 56 seconds and 58 seconds respectively, though they score lower on the reported quality metrics.

• Both conditioning signals matter: The ablation shows that with pose conditioning alone (DiT + Pose Encoder) Full Body SSIM is 0.5623 and Hair SSIM is 0.9158. Adding in-context LoRA (DiT + Pose-IC-LoRA) raises Hair SSIM to 0.9700, and adding the full HairWeaver design reaches Hair SSIM 0.9794. The authors also report that Wan-VACE finetuned with the same UVW map (Wan-VACE+UVW Map) reaches Hair SSIM 0.9724, which they use to argue the gains come from the architecture and sim-to-real strategy rather than richer conditioning alone.

• UVW maps give finer control than alpha maps: HairWeaver with alpha-map hair conditioning and DensePose body conditioning still reaches Hair SSIM 0.9693, PSNR 36.7297, LPIPS 0.0236, FID 49.0856, and cd-FVD 436.9389 — better than all baselines on the CG test set but slightly below the full UVW-based HairWeaver (Hair SSIM 0.9794, PSNR 37.6347).

• Style-Alignment-LoRA is needed for photorealistic references: Removing the Style-Alignment-LoRA gives Hair SSIM 0.9693, PSNR 36.7183, Hair FID 49.0808, and Full Body FID 38.4230. The authors state that the numerical difference on the CG test set is hard to observe because that test set is itself synthetic, so they demonstrate the module's effect qualitatively in Figure 5, where the version without it cannot preserve a photorealistic reference's appearance.

• User study preference: In a forced-choice study, HairWeaver was selected by 49.9% of responses, versus 16.4% for Wan-2.2-Animate-14B, 9.5% for UniAnimate-DiT, 9.0% for LTX-Video-ICLora, 8.2% for Wan-2.2-14B, and 6.9% for LTX-Video-0.9.8-13B. The table caption states 30 participants rated videos from six methods across eight identities, while the body text states 50 users were asked; the paper reports both numbers without reconciling them.

• The method generalizes to coarser real-world signals: Because UVW maps and body normal maps are unavailable for NeRSemble, the authors fine-tuned a Motion-Context-LoRA using alpha maps as the hair condition and DensePose maps as the body condition, and still obtained the best metrics among the compared methods on that test set.

Methodology in Plain English

The approach has two halves: making data, and teaching a video model to use it.

Making the data. Real footage of people does not come with accurate per-pixel labels for how each strand of hair moves. So the authors turned to computer graphics. They ran physics-based hair simulation on digital humans and rendered the results with Blender Cycles' path tracer, producing 1,500 videos of 100 frames each at 30 FPS — 83 minutes of footage total — at a resolution of 896 (height) × 512 (width). Each training sample is a quadruplet: a static reference image (typically the first frame), the rendered ground-truth video, a body pose condition (camera-space normal renders with hair hidden, plus 68 facial landmarks), and a hair motion condition (a UVW buffer where U and V are scalp texture coordinates at strand roots and W is the normalized arc-length along the hair curve). The UVW representation is dense and per-pixel, which the authors argue is far richer than sparse keypoints or optical flow.

Teaching the model. The base model is LTX-Video-0.9.8, whose DiT backbone weights stay frozen throughout. Training proceeds in two stages. In stage one, a Style-Alignment-LoRA is trained for 10,000 steps in a standard image-to-video manner, using the reference image and a text prompt captioned by Qwen-2.5-VL, so that the model adapts to the synthetic CG look. In stage two, that LoRA is frozen and a Motion-Context-LoRA plus a Pose Encoder are trained for another 10,000 steps on the full quadruplet. Body pose enters by element-wise addition to the noisy latent tokens; hair condition tokens are concatenated along the sequence length so the transformer's self-attention can attend to them directly. The Pose Encoder and the VAE encoder for hair both inherit weights from the pretrained LTX-Video VAE encoder; the LoRA layers are zero-initialized. Training uses the AdamW optimizer at a learning rate of 2e-4 on 97-frame clips, across 8 NVIDIA H200 GPUs with batch size 8.

The trick at inference. The Style-Alignment-LoRA — the part that learned the synthetic look — is thrown away completely. Only the original photorealistic backbone and the trained Motion-Context-LoRA remain. So the model keeps the fine-grained motion control learned from simulation while rendering with the foundation model's original photorealism.

Practical use. For the end-to-end application, the authors render a CG animation with the same Blender Cycles pipeline to get the shaded video plus normal and UVW maps, then stylize the first shaded frame into a photorealistic reference using a Flux-based image-to-image pipeline. They add only a small amount of noise (noising timestep 0.35), run 100 denoising steps with Euler integration, and use Qwen-2.5 VLM to generate a detailed prompt from the source image.

Why This Matters

Impact on research. The paper reframes hair animation as a data problem rather than an architecture problem: because accurate hair labels do not exist for real video, the authors generate perfectly aligned paired data in a simulator and then solve the domain gap with a disposable LoRA. This "train a temporary domain adapter, then delete it" pattern is a general recipe that could be applied to other secondary dynamics and other sim-to-real settings. The use of dense UVW hair representations as a diffusion conditioning signal is also distinctive; the closest prior work discussed, ControlHair, relies on approximate extracted hair direction maps rather than simulation-accurate data.

Real-world applications.

• Virtual reality and next-generation gaming, where avatars need hair that responds believably to head and body movement without hand-authored animation.
• Film and visual effects, where the authors note that achieving photorealistic CG hair currently requires top-tier studios and substantial cost, and where controllable hair dynamics are needed for storytelling.
• Human-centric synthetic data generation, explicitly called out by the authors as an immediate use case: a CG animation can be stylized into a photorealistic identity and used to train downstream models.
• Digital avatars with hairstyle editing or virtual try-on, which the authors list as downstream tasks enabled by the generated data.

Industry relevance. The paper is a collaboration between Meta and the University of Southern California, and the authors acknowledge the Meta Metasim team for the digital human asset library and pipeline. The results point at a production-relevant tradeoff: HairWeaver runs at 62 seconds per sample versus 870 seconds for UniAnimate-DiT and 312–476 seconds for the Wan-based methods, while the authors report better quality metrics and a nearly 50% user preference share. The ability to drive the system from CG simulation conditions also fits existing animation and VFX pipelines, since UVW maps and body normal maps fall straight out of the simulator.

Future Directions

• Handling large scale and pose deviations. The authors report that when the driving video involves significant zooming and large scale changes, appearance and identity preservation can be compromised, and they identify more diverse training data and more advanced backbone models as the remedy.

• Improving difficult details such as hands. Consistently accurate and realistic hands remain a failure mode, which the authors attribute to the scope of the training data and the generation ability of the LTX-Video backbone.

• Reducing dependence on simulation-exact conditions. The alpha-map and DensePose variant on NeRSemble shows the framework tolerates coarser conditions. How

Authors’ abstract

We present HairWeaver, a diffusion-based pipeline that animates a single human image with realistic and expressive hair dynamics. While existing methods successfully control body pose, they lack specific control over hair, and as a result, fail to capture the intricate hair motions, resulting in stiff and unrealistic animations. HairWeaver overcomes this limitation using two specialized modules: a Motion-Context-LoRA to integrate motion conditions and a Style-Alignment-LoRA to preserve the subject's photoreal appearance across different data domains. These lightweight components are designed to guide a video diffusion backbone while maintaining its core generative capabilities. By training on a specialized dataset of dynamic human motion generated from a CG simulator, HairWeaver affords fine control over hair motion and ultimately learns to produce highly realistic hair that responds naturally to movement. Comprehensive evaluations demonstrate that our approach sets a new state of the art, producing lifelike human hair animations with dynamic details.