OmniMotion-X: Versatile Multimodal Whole-Body Motion Generation

AI summary

Overview

Research area: Computer Vision and generative modeling — specifically multimodal whole-body human motion generation using diffusion transformers.

Technical level: Advanced. The paper assumes familiarity with diffusion models, transformers, autoregressive generation, and motion representations such as SMPL-X.

Scope: The paper introduces OmniMotion-X, a unified autoregressive diffusion transformer that generates whole-body human motion from text, music, speech, global spatial-temporal signals, and reference motion, together with OmniMoCap-X, a 286.2-hour unified motion capture dataset assembled from 28 public MoCap sources.

What This Paper Is About

Existing motion generation methods are mostly task-specific: separate models are built for text-to-motion, music-to-dance, and speech-to-gesture, and they are trained on small, format-inconsistent datasets. Unifying these tasks in one model is hard because different conditions impose different kinds of constraints — text is semantic, trajectories and joints are physical, and audio is rhythmic — and simply mixing them during training causes conflicts.

The goal of this work is to build one model that handles all of these conditions, plus their combinations, while generating high-quality whole-body motion (including hands and face), and to build the large, high-quality, consistently annotated dataset needed to train it.

Key Contributions

1. OmniMotion-X, a multimodal autoregressive diffusion transformer for versatile whole-body motion generation. It introduces reference motion (user-designed or previously generated motion) as a novel conditioning signal, which the authors report improves consistency of content, style, and temporal dynamics and enables clip-by-clip autoregressive generation.

2. A progressive weak-to-strong mixed conditions training strategy that first establishes motion-semantic alignment with text, then progressively adds finer-grained conditions (reference motion, global motion, speech, music) to reduce optimization conflicts between multi-granularity constraints.

3. OmniMoCap-X, described as the largest unified multimodal motion dataset to date: 28 publicly available MoCap sources across 10 distinct tasks, standardized to SMPL-X at 30 fps, totaling 64.3 million frames and 286.2 hours. Captions are generated by rendering motions into videos and annotating them with GPT-4o, producing structured, hierarchical text that combines low-level actions with high-level semantics.

4. State-of-the-art results on Text-to-Motion, Global Spatiotemporal Controllable Generation, Music-to-Dance, and Speech-to-Gesture, evaluated on more challenging test sets comprising 280 samples uniformly sampled from OmniMoCap-X.

Main Findings

• Text-to-Motion: On the OmniMoCap-X test set, OmniMotion-X achieves Top-1 R-Precision 0.303, FID 5.040, Multimodal Distance 4.678, Diversity 8.650, and MultiModality 1.696. With reference motion (Ours + RM), results improve to Top-1 R-Precision 0.346, FID 3.199, Multimodal Distance 4.106, Diversity 8.009, and MultiModality 1.143. For comparison, MoMask* trained on OmniMoCap-X reaches Top-1 R-Precision 0.267 and FID 17.428, and MotionCraft* reaches 0.236 and 47.428.

• Global spatial-temporal control: OmniMotion-X records FID 4.224, R-Precision 0.682, Multimodal Distance 4.377, Ctrl L2 0.424, Skating 0.004, Fail@20 0.516, and Fail@50 0.330, compared with OmniControl at FID 63.725, R-Precision 0.392, Ctrl L2 0.820, Skating 0.089, Fail@20 0.844, and Fail@50 0.794. Ground truth is listed at FID 0.013, R-Precision 0.821, Multimodal Distance 2.493.

• Speech-to-Gesture: Against MotionCraft, OmniMotion-X improves FID (Whole-Body) from 3.422 to 2.641, Face MSE from 0.182 to 0.045, and Diversity from 1.003 to 1.664. FID (Hands) is higher for OmniMotion-X (9.095 vs. 5.370).

• Music-to-Dance: OmniMotion-X reports FID (Whole Body) 16.209 versus MotionCraft's 9.875, FID (Hands) 5.827 versus 7.099, and Diversity 4.716 versus 3.798. The authors attribute the higher whole-body FID to small test-set distributions for S2G and M2D relative to the diverse tasks in OmniMoCap-X.

• Ablation on training strategy: Removing the weak-to-strong progressive strategy degrades T2M performance (FID 9.574 vs. 5.040, R-Precision 0.232 vs. 0.571, Diversity 3.118 vs. 8.650) and GSTC performance (FID 10.247 vs. 4.224, R-Precision 0.491 vs. 0.686, Diversity 2.438 vs. 6.292). The authors state progressive training also lowers computational load in early stages and converges faster than joint end-to-end training on all conditions.

• Model scale and speed: The model has 27.22M parameters and requires 2.14 seconds per sample; scaling tenfold to 355.13M parameters increases this only to 3.00s, which the authors interpret as evidence that the bottleneck is the long multimodal context rather than parameter count.

Methodology in Plain English

The model is a Transformer Encoder with 8 layers and 8 attention heads, a hidden dimension of 1536 (128 × 12, where 128 is the embedding size for each of 12 body parts), and a feedforward dimension of 3072. It is trained on a single H800 GPU.

Each condition — text, global motion, speech, music, and reference motion — is passed through its own encoder (T5-XXL for text, a wav encoder for speech, Librosa for music, and body-wise encoding for motion), projected into a shared space, and concatenated as prefix tokens alongside the noisy motion tokens. The diffusion model predicts the clean motion directly rather than predicting noise, following prior work. Instead of training on all conditions at once, training proceeds in stages: text-only for 460K steps, then adding reference motion for another 460K steps, then global spatiotemporal control for 230K steps, and finally full audio conditions for 920K steps, with batch sizes of 48, 48, 48, and 16 respectively. Optimization uses AdamW with an initial learning rate of 1×10⁻⁴ (reset when new conditions are added) and a cosine schedule decaying to 1×10⁻⁵ within the first 460K steps. The default length of motion reference and prediction is 150.

For the dataset, the team curated 28 open-source MoCap datasets covering T2M, M2D, S2G, HOI, HSI, and HHI, ranked their capture quality (Marker with manual correction, Vicon Marker, IMU, Multi-View RGB, Single-View RGB), and converted everything from BVH, FBX, and SMPL(-H) into a unified SMPL-X representation with normalized translation scale, normalized initial root orientation, and resampling to 30 FPS. The motion representation uses a whole-body format with 127 joints from SMPL-X and 53 joints covering body, hands, and jaw, plus Flame-format facial features in R¹⁰⁰. Captions were created by rendering motion to video and feeding it to GPT-4o together with existing text annotations.

Why This Matters

This paper argues that scale alone is not enough for unified motion generation — data quality, consistent annotation, and training strategy matter. It provides one of the first frameworks to handle text, music, speech, and spatial-temporal control in a single model, and shows that reference motion is a simple but effective conditioning signal that prior multimodal work overlooked.

Real-world applications identified in the paper:

• Animation and character motion production
• Gaming, including interactive character control
• Virtual reality experiences with speech- and music-driven avatars
• Embodied intelligence, where agents need controllable whole-body motion

Industry relevance: The released dataset and code (at the project's GitHub repository) lower the barrier for teams that cannot collect their own MoCap data. The clip-by-clip autoregressive formulation with reference motion supports "interactive generation of realistic, coherent, and controllable long-duration motions," which maps directly onto content pipelines where animators want to guide a model with an existing motion clip rather than start from scratch.

Future Directions

• Interaction-aware conditioning. The authors state the model lacks explicit conditioning for physical interactions with scenes, objects, or other people, and struggles to generate plausible interactive motions in complex environments. They propose conditioning on scene geometry and enforcing explicit physical constraints.

• Inference efficiency. At 2.14 seconds per sample, the model is slower than comparable text-conditioned diffusion models. Since a tenfold parameter increase only raises this to 3.00s, the authors identify the aggregated multimodal context length, not model size, as the bottleneck, and call for more efficient architectures for long-sequence multimodal interaction.

• Wider task coverage. Quantitative results in the paper cover T2M, GSTC, M2D, and S2G; the dataset additionally supports HOI, HSI, and HHI, but this paper does not report generation metrics for those interaction tasks, leaving them as open evaluations.

• Further caption and data refinement. The paper relies on rendered-video annotations from GPT-4o; the published content indicates quality was checked through iterative prompt optimization and comparison of multiple VLMs, without reporting a quantitative caption-quality benchmark.

Target Audience

Researchers and engineers working on generative human motion, character animation, and multimodal generative models, particularly those already familiar with diffusion transformers and motion representations like SMPL-X. It is also relevant to dataset builders interested in cross-dataset standardization and VLM-based annotation, and to applied teams in animation, gaming, and VR who need controllable whole-body motion from mixed inputs. Readers looking for quantitative results on human-object, human-scene, or human-human interaction generation will not find them here, since those tasks are covered in the dataset but not evaluated in the reported tables.

Authors’ abstract

This paper introduces OmniMotion-X, a versatile multimodal framework for whole-body human motion generation, leveraging an autoregressive diffusion transformer in a unified sequence-to-sequence manner. OmniMotion-X efficiently supports diverse multimodal tasks, including text-to-motion, music-to-dance, speech-to-gesture, and global spatial-temporal control scenarios (e.g., motion prediction, in-betweening, completion, and joint/trajectory-guided synthesis), as well as flexible combinations of these tasks. Specifically, we propose the use of reference motion as a novel conditioning signal, substantially enhancing the consistency of generated content, style, and temporal dynamics crucial for realistic animations. To handle multimodal conflicts, we introduce a progressive weak-to-strong mixed-condition training strategy. To enable high-quality multimodal training, we construct OmniMoCap-X, the largest unified multimodal motion dataset to date, integrating 28 publicly available MoCap sources across 10 distinct tasks, standardized to the SMPL-X format at 30 fps. To ensure detailed and consistent annotations, we render sequences into videos and use GPT-4o to automatically generate structured and hierarchical captions, capturing both low-level actions and high-level semantics. Extensive experimental evaluations confirm that OmniMotion-X significantly surpasses existing methods, demonstrating state-of-the-art performance across multiple multimodal tasks and enabling the interactive generation of realistic, coherent, and controllable long-duration motions.