IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation

AI summary

Overview

Research area: computer vision / text-driven 3D human motion generation, motion-aware large language models (MoLLMs), and multimodal interleaved reasoning. Technical level: Advanced. Scope: the paper proposes a paradigm and model that interleave motion generation with motion assessment and refinement to improve alignment between generated motion and goal text, trained through a three-stage scheme with an automated annotation pipeline.

What This Paper Is About

Motion-aware large language models can both understand motion (for example, caption a motion) and generate motion (for example, produce motion from text), but they are trained and run as separate tasks, so understanding never feeds back into generation. This paper introduces a paradigm where those tasks are linked through an iterative text-motion dialogue: the model generates a motion, assesses how well it matches the goal text, produces a refinement instruction, refines the motion, and repeats until the motion is judged satisfactory. The authors build IRG-MotionLLM, described as the first model that natively interleaves motion generation, assessment, and refinement for this purpose.

Key Contributions

1. IRMoGen paradigm: Interleaved Reasoning for Motion Generation, which couples motion generation with motion assessment and motion refinement through iterative text-motion dialogue, using assessment and refinement as the bridge tasks that let motion understanding knowledge flow into motion generation.
2. IRG-MotionLLM: a motion-aware LLM that natively interleaves generation, assessment, and refinement, built on a pre-trained Motion-aware LLM base model with a Motion VQVAE plus LLM architecture (K motion tokens plus boundary tokens <Motion> and </Motion> added to the LLM vocabulary).
3. A three-stage training scheme: Stage-1 IRMoGen Initialization over eight atomic sub-tasks (four basic tasks and four improving tasks), Stage-2 IRMoGen-CoT Learning with a reasoning chain-of-thought template, and Stage-3 IRMoGen Reinforcing using GRPO with format, text-motion alignment, and motion-motion alignment rewards.
4. An automated data engine that synthesizes interleaved reasoning annotations from existing text-motion datasets, using a pre-trained motion encoder plus frontier LLMs to assign goal analysis, ranked negative text-motion pairs at different alignment levels, and evaluation and refinement instructions.

Main Findings

• Assessment and refinement tasks are the bridge: after Stage-1 training, IRG-MotionLLM improves on text-to-motion generation and also outperforms existing methods on the motion-to-text caption task. On the caption task, the Stage-1 model reaches Bleu-1 62.05, Bleu-4 22.53, Rouge 47.51, CIDEr 52.4, and BERT 42.68, versus 58.73, 20.53, 46.12, 47.27, and 40.99 for the model trained with the M2T task only.
• Task gap is real: on HumanML3D generation, adding other basic tasks to the base model gives only slight Top-1 benefit, while combining all tasks gives consistent improvement across metrics (Stage-1: Top-1 0.504, Top-3 0.785, FID 0.141, MM-Dist 3.018, Diversity 9.814, against the MotionLLM base model at 0.496, 0.782, 0.198, 3.029, 9.864).
• Interleaving helps final motions across every stage: comparing the first generated motion with the final generated motion after assessment and refinement, Top-1 rises from 0.504 to 0.522 for Stage-1, from 0.497 to 0.526 for Stage-2, and from 0.503 to 0.535 for Stage-3.
• Stage-2 explicit CoT training adds alignment: the Stage-2 model reaches Top-1 0.526, Top-3 0.810, FID 0.111, MM-Dist 2.885, Diversity 9.819, and removing assessment plus refinement (w/o Asse.+Ref.) or removing the "Ignore Incorrect" strategy hurts results, with w/o I.I. dropping to Top-1 0.448 and FID 0.592.
• Stage-3 GRPO unlocks multi-round reasoning: after Stage-2, more than 70% of reasoning processes on test samples contain only one-round generation without refinement (a separate observation states that for more than 75% of cases the model ends after initial analysis and one generation); after RL tuning, the model extends its reasoning traces, and Stage-3 reaches Top-1 0.535, Top-3 0.820, FID 0.242, MM-Dist 2.785, Diversity 9.900. GRPO can introduce FID fluctuation due to the lack of dense supervision on generated motion tokens, but remains competitive with motion-aware LLMs supporting both text and motion outputs.
• Cross-benchmark performance: on HumanML3D the Stage-3 model reports Top-1 0.535 ± .002, Top-2 0.725 ± .002, Top-3 0.820 ± .002, FID 0.242 ± .006, MM-Dist 2.785 ± .006, Diversity 9.900 ± .094, compared with Motion-R1 at 0.515 ± .003, 0.719 ± .002, 0.818 ± .002, 0.201 ± .004, 2.854 ± .010, and 10.026 ± .075. On KIT-ML the Stage-3 model reports Top-1 0.445 ± .005, Top-2 0.681 ± .003, Top-3 0.781 ± .004, FID 0.432 ± .013, MM-Dist 2.740 ± .017, Diversity 11.115 ± .086.
• Scaling RL data and duration helps further: with Stage-3 data scaled to a similar scale as UniMo and extended GRPO duration (2,600 steps versus 900), the model reaches Top-1 0.564, Top-2 0.754, Top-3 0.841, FID 0.208, MM-Dist 2.628, Diversity 9.883, against UniMo at 14,000 steps reporting 0.539, 0.738, 0.831, 0.177, 2.768, and 10.042.
• Cross-evaluator performance: under the newer MARDM evaluator on HumanML3D, the Stage-3 model reports 0.509, 0.701, 0.802 on R-Precision Top-1/Top-2/Top-3, FID 0.259, and MM-Dist 3.166, versus the MotionLLM base at 0.470, 0.664, 0.771, 0.240, and 3.377.
• Emergent cross-model and cross-task synergy: beyond performing T2M on its own, the model can act as a transferable proxy to enhance an existing motion generator in an RLAIF manner, a property the authors state has not been explored in previous MoLLM work, and the method also benefits adaptation to the motion editing task.

Methodology in Plain English

The system keeps the architecture of earlier motion-aware LLMs: a motion VQVAE turns a motion sequence into discrete motion tokens, and an LLM predicts text and motion tokens together with a standard next-token objective; motion spans are marked with boundary tokens so generated motions can be extracted.

Training happens in three phases. In Stage-1, the base model is fine-tuned on eight atomic tasks. Four are basic text-motion tasks: motion captioning, direct motion generation, prompt analysis, and analysis-guided motion generation. Four are improving tasks: text-motion alignment evaluation, motion refinement instructing, direct motion refinement, and instruction-guided motion refinement. At this stage the interleaved reasoning can only be driven manually by chaining prompts.

Stage-2 defines an IRMoGen-CoT template: the model plans and analyzes the goal, then loops through generating a motion, assessing alignment, and refining, with pre-defined plan-making sentences linking steps, until the assessment says no refinement is needed. A trajectory with n generations contains intermediate motions m1 to m(n-1) and a final motion mn. Intermediate motions are deliberately misaligned but progressively closer to the goal, while the final motion is the ground-truth motion. Because training on incorrect intermediate motions would damage the learned text-motion alignment, the authors use an "Ignore Incorrect" strategy that masks losses and gradients on those tokens.

Stage-3 applies GRPO: for each instruction and text input, G trajectories are sampled and each receives a scalar reward. Rewards are a format reward (1 if the response follows the IRMoGen-CoT template, otherwise 0), a text-motion alignment reward (negative squared distance between the paired text encoder and motion encoder embeddings of the goal text and the final motion), and a motion-motion alignment reward (negative squared distance between the ground-truth motion embedding and the final motion embedding, set to 0 when no motion-text pair exists). Rewards are pre-normed across the group to the same scale and summed before the standard GRPO objective.

To get training data, an automated engine uses GPT-4o to write detailed goal analysis for each text prompt, uses a pre-trained motion encoder to rank all motions in the dataset by embedding distance to an anchor motion, and samples negative motions from the top p fraction of highest-distance instances, removing that fraction each time so negatives span a range of alignment levels. LLMs then produce alignment evaluations and refinement instructions from the anchor and negative text descriptions, and humans check and filter failure cases such as empty responses or refinements to already correct motions.

Evaluation uses HumanML3D and KIT-ML, reporting R-Precision, FID, MM-Dist, and Diversity with the official evaluators plus a newer evaluator from MARDM, averaging over 20 evaluation runs, and caption metrics Bleu, Rouge-L, CIDEr, and BertScore. Stage-3 training samples approximately 9k data per dataset, drawing on the original HumanML3D and InstructMotion data, and on KIT-M

Authors’ abstract

Recent advances in motion-aware large language models have shown remarkable promise for jointly learning motion understanding and generation knowledge. However, these models typically treat understanding and generation separately, limiting the mutual benefits that could arise from interactive feedback between tasks. In this work, we reveal that motion assessment and refinement tasks can act as crucial bridges to enable knowledge flow from motion understanding to generation. Specifically, we propose Interleaved Reasoning for Motion Generation (IRMoGen), a novel paradigm that tightly couples motion generation with assessment and refinement through iterative text-motion dialogue. To realize this, we introduce IRG-MotionLLM, the first model that seamlessly interleaves motion generation, assessment, and refinement to improve the alignment between generated motion and goal text. IRG-MotionLLM is developed progressively with a novel three-stage training scheme, initializing and subsequently enhancing native IRMoGen capabilities. To facilitate this development, we construct an automated data engine to synthesize interleaved reasoning annotations from existing text-motion datasets. Extensive experiments demonstrate the properties brought by IRMoGen training, and the advanced cross-benchmark and cross-evaluator performance of IRG-MotionLLM. Code and models are available at https://github.com/HumanMLLM/IRG-MotionLLM.