The Quest for Generalizable Motion Generation: Data, Model, and Evaluation

AI summary

Overview

Research area: 3D human motion generation (text-to-motion synthesis) in computer vision, with cross-pollination from video generation, 3D human modeling, and generative modeling.

Technical level: Intermediate. Familiarity with diffusion/flow-matching models, transformer architectures, and 3D human body representations (SMPL-X) helps, but the core ideas are accessible.

Scope: The paper presents a unified data-model-benchmark framework (ViMoGen-228K, ViMoGen, and MBench) aimed at making text-to-motion models generalize to rare, out-of-distribution, and long-tail text prompts rather than only standard everyday actions.

What This Paper Is About

Text-to-motion models perform well on common actions like walking or sitting, but they break down on unusual descriptions such as "a knight is jousting" or "body surfing." The reason is largely data scarcity: motion capture datasets are tiny compared to text, image, or video corpora, so their semantic coverage is narrow. The authors tackle this by systematically transferring knowledge from video generation models—which already understand a huge range of human behavior—into motion generation, while also fixing the data and evaluation gaps that hold the field back.

Key Contributions

1. ViMoGen-228K dataset. A 228,000-clip motion corpus drawn from three complementary sources: 30 unified optical motion capture datasets (172K high-fidelity text-motion pairs), aggressively filtered in-the-wild web videos (~1% of 10M candidates retained), and synthetically generated videos produced by a state-of-the-art text-to-video model prompting for rare actions. Includes both text-motion pairs and text-video-motion triplets.

2. ViMoGen model. A flow-matching diffusion transformer with a dual-branch, gated cross-attention design: a Text-to-Motion (T2M) branch that leans on precise mocap priors, and a Motion-to-Motion (M2M) branch that injects semantic knowledge from video-derived motion tokens. A VLM-based adaptive gate selects which branch to use per prompt.

3. ViMoGen-light. A distilled variant that inherits the generalization benefits of the dual-branch teacher without needing video generation at inference. It is trained on 14,000 synthetic prompts constructed from LLM-expanded action verbs.

4. MBench. A hierarchical benchmark with nine evaluation dimensions spanning motion generalization, motion-condition consistency, and motion quality, using open-world vocabulary and VLM-based assessment validated against human preference annotations.

Main Findings

• Generalization gap is real and measurable. All seven baseline models (MDM, T2M-GPT, MotionLCM, MoMask, MotionDiffuse, FineMoGen, MotionCraft) score below 0.55 on motion generalizability; ViMoGen reaches 0.68 and ViMoGen-light 0.55.

• Adaptive gating beats either branch alone. T2M-only scores 0.54 on generalization, M2M-only 0.59, but the adaptive gate reaches 0.68. The mechanism falls back to T2M when the video prior would be unreliable (e.g., dynamic actions like falling), and uses M2M when the video prior is semantically rich.

• Synthetic video data punches above its weight. Adding only 14K synthetic clips raised the generalization score from 0.50 to 0.55—a larger gain than adding 83K additional mocap clips (0.48–0.50). Semantic diversity, not raw volume, drives generalization.

• T5-XXL text encoder outperforms CLIP and MLLMs. T5-XXL balanced generalization (0.44) and motion quality best, suggesting that earlier work's reliance on CLIP features caps generalization.

• Training on verbose "video-style" captions and testing on concise "motion-style" prompts works best. Rich descriptions act as data augmentation, improving robustness (0.48 generalizability vs. 0.40 for the reverse).

• Two branches mutually benefit. Sharing ~66% of DiT parameters, the M2M branch injects diverse motion priors that improve T2M plausibility, while T2M enforces semantic alignment that helps M2M—outperforming a simple "select best of two SOTA models" baseline.

• Trade-off with motion quality. ViMoGen produces less jitter than baselines but lower dynamic degree, because incorporating diverse video-derived motions shifts the data distribution toward stable, non-locomotion actions.

Methodology in Plain English

The authors start by fixing the data problem. Rather than relying on a handful of studio-recorded mocap datasets, they merge 30 of them into a common SMPL-X format, add text captions generated by Gemini 2.0 from rendered depth videos, and then supplement with two additional streams: motion extracted from real internet videos (heavily filtered for quality) and motion extracted from generated videos. The generated videos are the clever part—by prompting a video model with rare action descriptions, they can synthesize training data for behaviors that essentially do not exist in any mocap dataset.

For the model, they use a diffusion transformer that predicts a velocity field via flow matching. Instead of one conditioning pathway, they build two: one branch attends to text embeddings (reliable, mocap-informed), the other attends to motion tokens extracted from a generated video (semantically rich but noisier). Both branches share most parameters; a vision-language model decides at inference time which branch to activate based on whether the video actually matches the text. During training, they simulate video-derived motion tokens by adding controlled noise to real motions—this avoids needing to run video generation during training and skips the modality gap that hurt earlier attempts to use intermediate video features directly.

For evaluation, they built MBench, which scores outputs across nine axes—three measuring whether the motion generalizes to novel prompts, three measuring whether it matches the text, and three measuring physical plausibility and quality. Rare-action prompts are constructed by expanding dictionary entries into richer phrases ("trample over flowerbeds with heavy, lumbering steps") so the benchmark tests deep language-motion understanding, not memorization.

Why This Matters

Impact on research. The paper reframes motion generation's bottleneck as a data-and-evaluation problem rather than just a modeling one, and shows that adjacent generative fields (video) contain transferable priors that can be exploited cheaply at distillation time. The finding that 14K synthetic clips outperform 83K mocap clips is a concrete, actionable signal for how the field should allocate data-collection effort going forward. MBench also provides the first fine-grained, human-validated evaluation harness for generalization, replacing coarse FID-style scores.

Real-world applications:

• Animation and game production, where characters need to perform rare or specified actions from natural language rather than pre-authored clips.
• Robotics and embodied AI, where diverse motion priors can inform motion planning and imitation learning.
• Virtual assistants and AR/VR avatars that respond to free-form user instructions.
• Rehabilitation and sports analysis, where generating reference motions for uncommon movements could support therapy or coaching.

Industry relevance. The paper targets a well-known pain point for studios and platform companies: existing mocap libraries are expensive to expand and cover a narrow expressive range. A model that generalizes to long-tail prompts without requiring new mocap sessions—and a distilled variant that runs without a video-generation model in the loop—directly addresses cost and latency constraints in production pipelines.

Future Directions

• Multi-person motion. The current architecture handles only single-person generation; extending it to interacting individuals remains open and important for social and combat scenes.

• Contact-aware visual mocap. The authors acknowledge that foot sliding and inaccurate global trajectories in video-derived motion are the primary source of remaining quality trade-offs; better extraction algorithms would directly improve the M2M branch's usefulness.

• Dynamic extrapolation. For high-dynamic actions like gymnastic twists, the model still depends on the video prior for initialization and cannot fully correct distorted dynamics. Strategies that force the model to extrapolate high-quality dynamics from imperfect video initializations are needed.

• Scaling and semantic coverage. Whether the synthetic-data insight (small, semantically diverse datasets beating large homogeneous ones) scales further, and whether richer intermediate video features could eventually close the video-motion modality gap, are open questions.

Target Audience

Researchers and practitioners working on 3D human motion synthesis, text-driven content generation, and generative modeling more broadly. Also relevant to animation and game industry engineers evaluating whether motion generation is production-ready, and to researchers studying cross-modal knowledge transfer between video and motion domains. Readers with some familiarity with diffusion models and 3D human body representations will get the most from the methodological details; the dataset and benchmark contributions are accessible to a wider audience.

Authors’ abstract

Despite recent advances in 3D human motion generation (MoGen) on standard benchmarks, existing text-to-motion models still face a fundamental bottleneck in their generalization capability. In contrast, adjacent generative fields, most notably video generation (ViGen), have demonstrated remarkable generalization in modeling human behaviors, highlighting transferable insights that MoGen can leverage. Motivated by this observation, we present a comprehensive framework that systematically transfers knowledge from ViGen to MoGen across three key pillars: data, modeling, and evaluation. First, we introduce ViMoGen-228K, a large-scale dataset comprising 228,000 high-quality motion samples that integrates high-fidelity optical MoCap data with semantically annotated motions from web videos and synthesized samples generated by state-of-the-art ViGen models. The dataset includes both text-motion pairs and text-video-motion triplets, substantially expanding semantic diversity. Second, we propose ViMoGen, a flow-matching-based diffusion transformer that unifies priors from MoCap data and ViGen models through gated multimodal conditioning. To enhance efficiency, we further develop ViMoGen-light, a distilled variant that eliminates video generation dependencies while preserving strong generalization. Finally, we present MBench, a hierarchical benchmark designed for fine-grained evaluation across motion quality, prompt fidelity, and generalization ability. Extensive experiments show that our framework significantly outperforms existing approaches in both automatic and human evaluations. The code, data, and benchmark will be made publicly available. Homepage: https://motrixlab.github.io/2026_iclr_vimogen.