HY-Motion 1.0: Scaling Flow Matching Models for Text-To-Motion Generation

AI summary

Overview

Research area: Computer vision / generative AI for 3D human motion synthesis (text-to-motion generation).

Technical level: Advanced. The paper assumes familiarity with Diffusion Transformers (DiT), flow matching, reinforcement learning from human feedback (DPO, GRPO), and motion representations such as SMPL-H.

Scope: HY-Motion 1.0 is a family of large-scale 3D human motion generation models from Tencent Hunyuan 3D Digital Human Team, presented as the first successful scaling of a DiT-based flow matching model to over one billion parameters in the motion generation domain, trained through a three-stage pretraining, fine-tuning, and reinforcement learning pipeline on 3,000+ hours of motion data.

What This Paper Is About

Generating realistic 3D human motion from natural language text remains difficult: existing models are small and struggle with complex instructions, while LLM-based approaches that extend vocabulary with discrete motion tokenizers introduce quantization that degrades motion smoothness. The authors also identify a data bottleneck — the lack of large-scale, meticulously cleaned and annotated motion datasets — and note that, unlike image and video generation, scaling of motion diffusion models is largely underexplored. The goal of HY-Motion 1.0 is to close these gaps by scaling a DiT-based flow matching architecture to the billion-parameter scale and training it through a full-stage curriculum spanning pretraining, high-quality fine-tuning, and reinforcement learning from both human feedback and reward models.

Key Contributions

1. Scaling law for text-to-motion: The authors state they are the first to successfully scale a DiT-based flow matching model to over one billion parameters for text-to-motion generation, demonstrating significant improvements in instruction-following capability.

2. Comprehensive full-stage training paradigm: A three-stage framework — large-scale pretraining (3,000+ hours), high-quality fine-tuning (400 hours of curated data), and reinforcement learning.

3. Meticulous data curation pipeline: Automated processing plus extensive manual refinement for motion cleaning and captioning, producing a dataset and a three-level taxonomy covering over 200 fine-grained motion classes across six top-level categories.

4. Open-source release: The HY-Motion 1.0 models (with DiT-1B as the primary model and DiT-0.46B as a lightweight "Lite" version) are released to the open-source community.

Main Findings

• Instruction-following performance: On a test set of over 2,000 text prompts rated by human annotators on a scale of 1 to 5, HY-Motion 1.0 achieves an average score of 3.24 versus MoMask 2.31, GoToZero 2.19, DART 2.31, and LoM 2.17.

• SSAE scores: On the Structured Semantic Alignment Evaluation metric, HY-Motion 1.0 reaches 78.6%, compared with MoMask 58.0%, GoToZero 52.7%, DART 42.7%, and LoM 48.9%.

• Motion quality: Average human rating of 3.43 for HY-Motion 1.0, versus MoMask 2.79, GoToZero 2.86, DART 3.11, and LoM 3.06.

• Scaling behavior differs by capability: Instruction-following capability consistently improves with larger models (DiT-0.05B average 3.10, DiT-0.46B 3.20, DiT-1B 3.34), whereas motion quality saturates beyond the 0.46B parameter size (DiT-0.05B 2.91, DiT-0.46B 3.26, DiT-1B 3.34).

• Data volume matters for instruction following: DiT-0.46B (pretrained on the 3,000-hour dataset) scores 3.20 on instruction following, while DiT-0.46B-400h (trained only on the 400-hour high-quality dataset) scores 3.05, underscoring the role of the larger dataset. Notably, DiT-0.46B-400h scores slightly higher on motion quality (3.31) than DiT-0.46B (3.26).

• Pretraining output characteristics: Because much of the 3,000-hour dataset contains noisy in-the-wild video extractions, the pretrained model produces semantically correct but artifact-prone motion with high-frequency jitter, foot sliding, and minor anatomical inconsistencies.

• Fine-tuning effects: Decaying the learning rate to 0.1× the pretraining value suppresses noise patterns inherited from pretraining, reduces high-frequency jitter and foot sliding, and improves sensitivity to fine-grained instructions such as distinguishing "waving the left hand" from "waving the right hand," without significant degradation in motion diversity.

• The duality of data scale and quality: Scaling data volume primarily drives instruction following and semantic understanding, while improving data quality is the decisive factor for motion fidelity and physical realism.

Methodology in Plain English

Data. The team assembled motion data from three sources: 12 million in-the-wild video clips from HunyuanVideo, about 500 hours of motion capture data and 3D animation assets. Videos went through shot boundary detection and human detection, then GVHMR reconstructed 3D human tracks as SMPL-X parameters. All motion was retargeted to a unified SMPL-H skeleton, filtered for duplicates, abnormal poses, velocity outliers, anomalous displacements, static motion, and artifacts like foot sliding, then canonicalized (resampled to 30 fps, sequences longer than 12 seconds split, normalized to Y-up with start at the origin). The result was over 3,000 hours of motion data, including 400 hours of high-quality 3D motion data. Captions were generated by feeding videos (real or rendered from SMPL-H) with motion-specific prompts into a vision-language model (for example, Gemini-2.5-Pro), then manually verified for the high-quality rendered motions, and finally standardized and paraphrased with an LLM. A three-level taxonomy organizes the data into six top-level categories and over 200 leaf-level motion classes.

Representation. Each frame is a 201-dimensional vector: global root translation (3), global body orientation (6), local joint rotations (21 × 6), and local joint positions (22 × 3), using a continuous 6D rotation representation. Explicit temporal derivatives (velocities) and foot contact labels were removed because the authors observed faster training convergence without them.

Model. The HY-Motion DiT takes a text prompt and expected duration and outputs a motion clip. It uses a hybrid Transformer with dual-stream blocks (separate QKV projections and MLPs for motion latents and text tokens, interacting through joint attention) followed by single-stream blocks where tokens are concatenated and processed with parallel spatial and channel attention. Text conditioning is hierarchical: Qwen3-8B supplies token-wise embeddings passed through a Bidirectional Token Refiner (converting causal LLM features into bidirectional ones), while CLIP-L supplies a global embedding injected through AdaLN. Attention uses an asymmetric mask (motion tokens attend to text, but text tokens are masked from motion latents to prevent diffusion noise from corrupting text embeddings) and a narrow band mask limiting temporal attention to a sliding window of 121 frames at 30 fps for linear complexity. Full Rotary Positional Embeddings are applied across a concatenated text-plus-motion sequence.

Training. Flow matching with the optimal transport path — a linear interpolation between noise and clean motion — is trained by minimizing mean squared error between predicted and ground-truth velocity; inference integrates the resulting ODE, for example with an Euler solver. Stage one pretrains on the full 3,000-hour dataset at constant learning rate. Stage two fine-tunes on the 400-hour high-quality set at 0.1× that learning rate. Stage three applies reinforcement learning in two parts: DPO on a curated subset of 9,228 pairs selected from a pool of 40,000 annotated pairs where human judges picked a winner and loser, followed by Flow-GRPO, which normalizes advantages within a group of sampled outputs and optimizes a composite reward combining a semantic reward from a custom-trained Text-Motion Retrieval model and a physical reward that penalizes foot sliding and root drift. Separately, a Qwen3-30B-A3B model is fine-tuned (supervised fine-tuning then GRPO with Qwen3-235B-A22B-Instruct-2507 as reward judge) to predict motion duration and rewrite casual user prompts into structured ones.

Why This Matters

Impact on research: The paper provides evidence that the scaling behavior of motion generation models differs by capability — data volume drives semantic understanding while data quality drives fidelity. It also demonstrates that a coarse-to-fine, multi-stage recipe (pretraining, fine-tuning, RL) can reconcile motion diversity with precision, offering a data-centric template for the field.

Real-world applications:

• Animation and game production, where artists generate character motion from text descriptions rather than hand-authoring it.
• VR/AR and interactive entertainment, where users direct digital humans through natural language.
• Digital human and virtual assistant systems combining generated motion with speech and appearance.
• Robotics, which the authors list among fields driven by 3D content creation.

Industry relevance: The authors explicitly frame the work as accelerating the transition of 3D human motion generation toward commercial maturity, and note that previous models being small-limited their ability to understand instructions and produce complex motion. The data pipeline itself — automated cleaning, captioning, plus manual refinement — is presented as a practical asset for industrial-scale production.

Future Directions

• Complex and detailed instructions: The model still struggles with highly detailed or nuanced instructions, partly because producing complete, accurate textual descriptions for intricate motions is difficult for both VLM-based captioning and manual refinement.

• Human-object interaction: The current dataset focuses on body kinematics without explicit object geometry, so the model may struggle to generate physically accurate interactions with external objects, such as precise contact points when holding a tool.

• Extending the scaling study: The paper reports saturation of motion quality beyond the 0.46B parameter size but continued gains in instruction following up to 1B, leaving open where further scale would help and which components (backbone, text encoder, data) drive the gains.

• Prompt understanding module: The LLM for duration prediction and prompt rewrite is fine-tuned to generalize to unseen motion descriptions, but the paper does not report limits of that generalization, suggesting an area for further evaluation.

Target Audience

Researchers and engineers working on generative models for 3D human motion, diffusion and flow matching architectures, and multimodal text conditioning will benefit most, as will practitioners in animation, games, and digital human production evaluating whether text-to-motion models are ready for production use. Readers seeking benchmark comparisons against standard motion-generation evaluation suites (such as FID or R-Precision on HumanML3D) should note that this paper evaluates with human raters on a 1-to-5 scale plus the SSAE video-VLM metric, and does not report those conventional metrics.

Authors’ abstract

We present HY-Motion 1.0, a series of state-of-the-art, large-scale, motion generation models capable of generating 3D human motions from textual descriptions. HY-Motion 1.0 represents the first successful attempt to scale up Diffusion Transformer (DiT)-based flow matching models to the billion-parameter scale within the motion generation domain, delivering instruction-following capabilities that significantly outperform current open-source benchmarks. Uniquely, we introduce a comprehensive, full-stage training paradigm -- including large-scale pretraining on over 3,000 hours of motion data, high-quality fine-tuning on 400 hours of curated data, and reinforcement learning from both human feedback and reward models -- to ensure precise alignment with the text instruction and high motion quality. This framework is supported by our meticulous data processing pipeline, which performs rigorous motion cleaning and captioning. Consequently, our model achieves the most extensive coverage, spanning over 200 motion categories across 6 major classes. We release HY-Motion 1.0 to the open-source community to foster future research and accelerate the transition of 3D human motion generation models towards commercial maturity.