MoLingo: Motion-Language Alignment for Text-to-Human Motion Generation

AI summary

Overview

Research area: Computer Vision / generative modelling of 3D human motion, specifically text-to-motion (T2M) generation using diffusion and flow-based methods in a learned continuous latent space.

Technical level: Advanced (assumes familiarity with autoencoders, diffusion/flow matching, transformers, and cross-attention).

Scope: The paper (arXiv:2512.13840v3 [cs.CV], by Yannan He, Garvita Tiwari, Xiaohan Zhang, Pankaj Bora, Tolga Birdal, Jan Eric Lenssen and Gerard Pons-Moll, with affiliations at the University of Tübingen, Tübingen AI Center, Max Planck Institute for Informatics, Imperial College London and Zuse School ELIZA) studies two questions — how to make a motion latent space that is friendly to diffusion, and how to inject text conditioning — and builds a model, MoLingo, that sets state-of-the-art results on standard text-to-motion metrics and in a user study.

What This Paper Is About

Text-to-motion models generate 3D human motion from a written description. Recent systems compress a motion sequence into a compact continuous latent space and then denoise inside that space, but it is unclear what makes such a latent space well suited to diffusion and how the text prompt should best be fed in. MoLingo answers both questions: it builds a semantically aligned motion encoder trained with frame-level text labels, and it replaces the usual single-token text conditioning with multi-token cross-attention, then generates motion with masked auto-regressive rectified flow.

Key Contributions

1. MoLingo, a masked auto-regressive rectified-flow model that produces natural, text-faithful human motion in a continuous latent space and sets a new state of the art on FID, R-Precision and a user study.
2. A semantically aligned autoencoder (SAE), trained on frame-level text labels so that motion latents with similar text meaning sit close together in the latent space; the paper also provides a plain VAE variant and shows alignment helps the model follow text instructions.
3. A comparison of text-conditioning schemes, showing that multiple text tokens with cross-attention to the motion latents gives significantly better motion realism and text–motion alignment than the common single-token approach.
4. A broad evaluation across multiple evaluators (MARDM-67, MS-272 and, for the first time applied to a large set of prior work, TMR-263), plus a user study and a transfer experiment to a physics-based RL tracking controller. Code and models are promised for release.

Main Findings

• New state of the art on MARDM-67: MoLingo (VAE) reaches FID 0.049 ± .003, R-Precision Top-1 0.528 ± .002, Top-2 0.721, Top-3 0.815, and CLIP-Score 0.672 ± .001. MoLingo (SAE) reaches FID 0.066 ± .003 but higher alignment: R-Precision Top-1 0.544 ± .002, Top-2 0.739, Top-3 0.832, and CLIP-Score 0.686 ± .001. The paper describes this as reducing FID from 0.053 (DisCoRD) to 0.049 for the VAE variant and improving R-Precision Top-1 from 0.522 (ACMDM-XL-PS2) to 0.542 for the SAE variant. The real-motion reference row reports FID 0.000, R-Precision Top-1 0.503, Top-2 0.696, Top-3 0.795, and CLIP-Score 0.639.
• Multi-token cross-attention beats single-token conditioning: in the ablation on MARDM-67 (4× downsampling, 16-d latent), AdaLN with CLIP text and a plain AE gives FID 0.114 and R-Precision Top-1 0.500 (the MARDM baseline); switching to a T5 text encoder with AdaLN improves this to FID 0.077 and Top-1 0.508; multi-token cross-attention with T5 yields FID 0.049 with the VAE, 0.051 with the AE, and 0.066 with the SAE.
• The text encoder matters: using T5 instead of CLIP improves performance even without multi-token cross-attention.
• Semantic alignment helps text adherence: SAE achieves the best text–motion alignment scores (R-Precision Top-1 0.544, CLIP-Score 0.686) among the autoencoder variants while keeping FID comparable to state-of-the-art models.
• Cosine similarity beats InfoNCE, and KL helps: in the SAE design ablation, cosine similarity with KL divergence and a small semantic weight (λ_sem = 0.001) is the best configuration (FID 0.066, R-Precision Top-1 0.544, Top-2 0.739, Top-3 0.832, CLIP 0.686). InfoNCE with λ_sem = 0.001 gives FID 0.129, Top-1 0.523, Top-2 0.721, Top-3 0.813, CLIP 0.671. The authors argue InfoNCE is too rigid because the available frame-level labels are repetitive and human motion is continuous and ambiguous.
• Latent size trade-off: moving from 4× to 2× temporal compression (more latents) consistently improves R-Precision, while increasing the latent dimension (tested from 16 to 128) has a negative effect on overall performance. The 4× setting with 16-d latents is chosen as best.
• User study preference: participants preferred MoLingo over DisCoRD in 83.75% of cases, over MoMask in 77.70%, and over MotionStreamer in 84.70%. The study used 20 randomly selected sequence pairs per baseline, 15 users per comparison, and randomized method order.
• Text adapter depth: the supplementary ablation on adapter depth (0, 3, 6 and 9 layers) reports that a 6-layer adapter gives the best overall trade-off, improving FID and R-Precision over the others.
• Physical plausibility: when combined with an RL tracking controller, MoLingo produces consistently realistic foot–ground interaction across the whole sequence, whereas MotionStreamer frequently shows balance artifacts; the improvement also reduces foot sliding and penetration.

Methodology in Plain English

The system has two halves. First, an autoencoder compresses a motion sequence into a shorter sequence of continuous latent vectors using stacked 1D temporal convolutions (a causal architecture), so N motion frames become l = N/h latents of dimension d; a decoder turns them back into motion. Three variants are compared: a variational autoencoder (VAE), a vanilla autoencoder (AE), and the proposed semantically aligned autoencoder (SAE).

The SAE is the distinctive part. Using the BABEL dataset, which supplies frame-level text labels, the authors take the text labels temporally aligned with each motion latent, encode them with a frozen text encoder, average the embeddings and project them linearly to get a "class token" for that latent. A cosine-similarity loss pulls each motion latent towards its class token. Because BABEL labels are highly repetitive, consecutive identical class tokens would cause overly strong alignment, so the authors compute the cosine similarity between consecutive class tokens and discard tokens (and their latents) whose similarity exceeds a threshold. The SAE is trained with a reconstruction loss (on motion features, joint positions and joint velocities) plus the semantic loss and a KL divergence term.

Second, a generative model denoises in that latent space. The text prompt is encoded with a frozen T5-Large encoder and passed through a text adapter of transformer encoder blocks. During training, a random subset of the motion latents is replaced with a learnable mask token, and a decoder-only transformer with self-attention, cross-attention and MLP layers produces a conditioning vector for each position. The text tokens act as keys and values in cross-attention, while motion latents act as queries. An MLP then predicts the rectified-flow velocity, trained to map between Gaussian noise and clean latents via a linear interpolation over time. Training uses bidirectional attention so masked latents can see all unmasked latents. At inference, all latents start as the learned mask token and are progressively replaced with clean ones, position by position, until the full latent sequence is decoded into motion.

Implementation-wise, models are trained in PyTorch on HumanML3D (29,024 motions and 87,834 text descriptions, unified to 20 FPS and up to 10 seconds long, with a 272-dimensional 30 FPS version used for comparison with MotionStreamer). The autoencoders are trained for 5000 epochs with batch size 256 and learning rate 5×10⁻⁵; the auto-regressive flow model is trained for about 800 epochs with batch size 256, a linear warm-up over the first 100 epochs and a constant learning rate of 8×10⁻⁴ thereafter, using EMA and classifier-free guidance (10% of prompts replaced with a null prompt, CFG scale 5.5 at inference). The best configuration trains in roughly 10 hours on four Nvidia H100 GPUs. Evaluation uses FID for distributional realism, R-Precision and CLIP-Score for text–motion alignment, and MModality for diversity, reported as means over 20 generation runs with 95% confidence intervals, across the MARDM-67, MS-272 and TMR-263 evaluators.

Why This Matters

Impact on research. The paper separates two design axes that prior text-to-motion work often conflated: what makes a latent space "diffusable" and how text should be injected. Its findings — that soft semantic alignment of latents with frame-level text labels improves instruction following, and that a single text token is not expressive enough compared with multi-token cross-attention — are directly transferable to other conditional generative settings that use latent diffusion or flow. The evaluation across MARDM-67, MS-272 and TMR-263 also pushes the field towards multi-protocol reporting, since earlier work showed the classic 263-dimensional Guo representation is redundant and relies on inverse-kinematics-derived rotations that introduce errors.

Real-world applications.

• Computer animation and character motion authoring, where animators describe a movement in text rather than keyframing it.
• AR/VR entertainment, generating lifelike avatar motion for interactive experiences.
• Instruction-following agents, since motion generation from text is described as crucial for building agents that can follow human instructions.
• Robotics and physics simulation, transferring generated motion to a character or robot through a pretrained RL tracking controller with physical constraints, where the paper reports improved foot–ground contact and less foot sliding and penetration than MotionStreamer.

Industry relevance. Studios, game developers, AR/VR platform builders and robotics teams all need scalable ways to produce realistic, controllable human motion. The reported training cost (about 10 hours on four Nvidia H100 GPUs for the best configuration) and the promise to release code and models lower the barrier to adopting the approach, and the robot-transfer demonstration shows the outputs are usable downstream rather than only visual.

Future Directions

• Hands and full-body detail. The stated limitation is that the method focuses on main body dynamics and does not generate full-body motion with detailed hand movements, which the authors call an essential aspect of human motion and an exciting future direction.
• Latent space design. The latent-size ablation leaves open what the optimal number of latents and latent dimensionality is, and why larger latent dimensions hurt performance; the 4× / 16-d choice is empirical rather than derived.
• Semantic supervision beyond BABEL. The SAE depends on BABEL's frame-level labels, which the paper itself notes lack diversity; better or richer frame-level annotation, or label sources beyond the BABEL–HumanML3D intersection, could extend the approach.
• Evaluation and robot transfer at scale. The paper introduces large-scale TMR-263 reporting and a qualitative and quantitative comparison against MotionStreamer in a physics-simulation setting; standardizing these protocols and testing broader controller transfer remain open.

Target Audience

Researchers and graduate students working on text-to-motion, human motion synthesis, latent diffusion and flow matching, and generative modelling more broadly; engineers building animation, AR/VR or embodied-agent systems who need to understand the current state of the art in latent-space motion generation and the trade-offs between VAE and semantically aligned latents; and robotics or graphics practitioners interested in feeding generated motion into physics-based tracking controllers.

Authors’ abstract

We introduce MoLingo, a text-to-motion (T2M) model that generates realistic, lifelike human motion by denoising in a continuous latent space. Recent works perform latent space diffusion, either on the whole latent at once or auto-regressively over multiple latents. In this paper, we study how to make diffusion on continuous motion latents work best. We focus on two questions: (1) how to build a semantically aligned latent space so diffusion becomes more effective, and (2) how to best inject text conditioning so the motion follows the description closely. We propose a semantic-aligned motion encoder trained with frame-level text labels so that latents with similar text meaning stay close, which makes the latent space more diffusion-friendly. We also compare single-token conditioning with a multi-token cross-attention scheme and find that cross-attention gives better motion realism and text-motion alignment. With semantically aligned latents, auto-regressive generation, and cross-attention text conditioning, our model sets a new state of the art in human motion generation on standard metrics and in a user study. We will release our code and models for further research and downstream usage.