Language-Guided Transformer Tokenizer for Human Motion Generation

AI summary

Overview

• Research area: Computer Vision / multimodal human motion generation, specifically discrete motion tokenization for text-to-motion synthesis.
• Technical level: Advanced. The paper assumes familiarity with VQ-VAE, Residual VQ-VAE, transformer attention, autoregressive and masked generative models, and evaluation metrics such as FID and R-Precision.
• Scope: The paper proposes Language-Guided Tokenization (LG-Tok), a transformer-based motion tokenizer that uses natural language as auxiliary guidance during tokenization, and evaluates it on three text-to-motion benchmarks (HumanML3D, KIT-ML, Motion-X).

What This Paper Is About

Existing motion tokenizers face a trade-off: increasing the number of tokens improves motion reconstruction quality (rFID) but makes it harder for generative models to learn, hurting generation quality (gFID). The authors ask whether tokens can stay numerous enough for high-fidelity reconstruction while becoming easier for generative models to model. Their answer is to align natural language with motion during tokenization itself, producing compact, high-level semantic representations, and to replace the convolutional tokenizer architecture with a transformer.

Key Contributions

1. Language-Guided Tokenization (LG-Tok): A tokenization scheme that aligns natural language descriptions with motion during the tokenization phase, so latent tokens absorb linguistic semantics instead of redundantly encoding high-level meaning.
2. A transformer-based tokenizer and detokenizer: Both components use attention rather than 1D convolutions, allowing global language guidance and stronger global contextual awareness. The tokenizer concatenates learnable latent tokens with linearly projected motion and text embeddings; the detokenizer reconstructs motion from learnable mask tokens interacting with dequantized embeddings and text via separate cross-attention layers.
3. A language-drop scheme: Text conditions are randomly removed during training with probability p = 0.1, preventing shortcut learning over text embeddings and enabling language-free guidance decoding in the motion space (a Classifier-Free Guidance analog applied to motion rather than logits or noise).
4. Three model variants at different token scales: LG-Tok-mini (25 tokens; 104 total), LG-Tok-mid (36 tokens; 160 total), and LG-Tok (49 tokens; 236 total), all using N = 10 quantizer scales.

Main Findings

• The reconstruction–generation trade-off is confirmed empirically: In Table 1, increasing tokens from 104 to 160 to 236 improves reconstruction (rFID 0.143 → 0.110 → 0.090) but not generation (gFID 0.230 → 0.205 → 0.257).
• Compact semantic representations simplify generative learning: On Motion-X, the authors report model perplexity dropping from 146.5 to 103.1 and FID improving from 0.257 to 0.088.
• Best HumanML3D results: LG-Tok achieves a Top-1 R-Precision of 0.542 and FID of 0.057, outperforming MARDM (0.500 and 0.114) and the prior best MoSa (0.518 and 0.064). LG-Tok also reaches MultiModal Dist 2.997, Top-2 0.736, Top-3 0.830, and CLIP-score 0.669.
• Efficiency at half the tokens: LG-Tok-mini (104 tokens) reaches Top-1 0.521 and FID 0.085 on HumanML3D with competitive MultiModality (1.728) and CLIP-score (0.655), versus LG-Tok's 236 tokens.
• Strong Motion-X results: LG-Tok-mid achieves the best Top-1 R-Precision of 0.591, with LG-Tok-mini at 0.588 and LG-Tok at 0.582. LG-Tok reports the best FID on Motion-X at 0.071 for LG-Tok-mini (LG-Tok-mid 0.076, LG-Tok 0.088).
• Tokenizer comparison (Table 3, HumanML3D): LG-Tok reconstruction FID 0.022, MPJPE 39.0; without text guidance, reconstruction FID 0.025 and MPJPE 39.0 — the language guidance improves reconstruction. MoMask (LG-Tok) improves on MoMask-reprod. (0.029 FID, 30.9 MPJPE) with 0.019 FID and 26.4 MPJPE.
• Generalization across generative backbones: Applying LG-Tok to MoMask yields consistent improvements, indicating the tokenizer is not tied to the MoSa backbone.
• Qualitative gains: Compared against StableMoFusion, MARDM, and MoSa on HumanML3D, LG-Tok better handles spatial cues ("in the middle"), realistic postures ("dodges quickly"), and direction ("then turns to the left").
• KIT-ML results are stated to be in Appendix C, which is not included in the provided content, so those numbers are not reported here.

Methodology in Plain English

The pipeline is two-stage: first tokenize motion into discrete codes, then use a generative model over those codes.

1. Encode motion and text together. A frozen LLaMA-3.2-1B text encoder produces text embeddings. These are concatenated with a set of learnable latent tokens and a linearly projected motion sequence, and fed into a transformer tokenizer that uses self-attention only. The tokenizer outputs only the positions corresponding to the learnable latent tokens.
2. Quantize into multi-scale tokens. The latent output is quantized into discrete tokens across multiple downsampled scales (following the MoSa multi-scale residual quantization scheme), producing N = 10 scales.
3. Drop language sometimes. With probability p = 0.1, the text input is removed during training, so the tokenizer cannot simply copy text semantics.
4. Decode back to motion. A transformer detokenizer takes learnable mask tokens and lets them cross-attend to dequantized embeddings and to text embeddings through two separate cross-attention layers, then regresses to motion space with a linear layer. Training uses smooth L1 reconstruction loss with no text reconstruction.
5. Generate. A scale-wise autoregressive generative model (MoSa by default; also tested with MoMask) samples tokens conditioned on text. At decoding, since the detokenizer works with and without text, the authors run conditional and unconditional passes and extrapolate in motion space with guidance scale g.

Implementation specifics: AdamW, batch size 128, 200 epochs, learning rate 2×10⁻⁴ reduced to 2×10⁻⁵ at the 180th epoch, gradient clipping 0.01, no velocity loss. Tokenizer and detokenizer each use 9 layers, 4 heads, 256 latent dimensions, SwiGLU dimension 1024, RMSNorm, SwiGLU activation, RoPE with base 100, and UNet-like long skip connections. Mixed-precision training and PyTorch 2.2.0 with flash attention; training fits on a single RTX 4090. Max text length 77. Guidance scale g is 2.0 on HumanML3D, 1.0 on Motion-X, and 2.0 on KIT-ML. CLIP-ViT-B/32 is used for the generative model and retrieval evaluation, matching baselines; LLaMA features are exclusive to LG-Tok's tokenizer and detokenizer.

Datasets and setup: HumanML3D (14,616 motions, 44,970 texts, three captions per motion), KIT-ML (3,911 motions, 6,278 texts), and Motion-X filtered to motion-text pairs under 200 frames, giving 37,751 motion sequences and 61,637 text captions. Splits are 80% train / 5% validation / 15% test. All datasets use the Meng 64–67 dimension representation; finger and facial information is ignored. Metrics are FID, R-Precision, Multimodal distance, Multimodality, and CLIP-score, each evaluated 20 times with 95% confidence intervals.

Why This Matters

• Research impact: The paper moves language conditioning earlier in the pipeline — from generation into tokenization — and argues that this both improves reconstruction and eases generative modeling. It also extends Classifier-Free Guidance from logit/noise space into motion space, and challenges the dominant convolutional tokenizer design in favor of a fully attention-based one, including a design that fully decouples token count from frame count (196 frames encoded via N learnable queries, versus M2DM's 1:1 frame-to-latent alignment on 64-frame crops).
• Real-world applications (as named in the paper):
  • Game animation
  • Virtual reality
  • Video motion editing
• Industry relevance: Training completes on a single RTX 4090, and LG-Tok-mini holds competitive quality with 104 tokens instead of 236 — half the tokens, which matters for inference cost and for downstream generative modeling. The tokenizer also transfers to a different generative backbone (MoMask), suggesting it can be dropped into existing pipelines.

Future Directions

• KIT-ML and appendix results: The paper defers KIT-ML evaluation and extended applications, inference-time analysis, and limitations to appendices; these are not present in the provided content, so their findings remain unverified here.
• Tuning token budget per application: With three tested scales (25, 36, 49 latent tokens) showing different reconstruction–generation trade-offs, the principled choice of token count and quantizer scales remains open.
• Language guidance beyond motion: The tokenizer architecture and language-drop idea are motivated by image tokenization work (VQ-GAN, TikTok, TxtTok); whether the same transformer-plus-language-drop design generalizes to other modalities is untested.
• Interaction between motion-space guidance and other guidance methods: The authors present motion-space guidance as complementary to logit-space guidance, but the combined effect is not quantified beyond the reported guidance scales.

Target Audience

Researchers and practitioners working on text-driven human motion generation, discrete tokenization, VQ-VAE-based representation learning, and multimodal alignment. It is most useful to readers already familiar with generative transformers, masked or autoregressive token modeling, and motion generation benchmarks such as HumanML3D, KIT-ML, and Motion-X; readers seeking an introductory treatment of motion synthesis will find the quantizer notation and multi-scale formulation demanding.

Authors’ abstract

In this paper, we focus on motion discrete tokenization, which converts raw motion into compact discrete tokens--a process proven crucial for efficient motion generation. In this paradigm, increasing the number of tokens is a common approach to improving motion reconstruction quality, but more tokens make it more difficult for generative models to learn. To maintain high reconstruction quality while reducing generation complexity, we propose leveraging language to achieve efficient motion tokenization, which we term Language-Guided Tokenization (LG-Tok). LG-Tok aligns natural language with motion at the tokenization stage, yielding compact, high-level semantic representations. This approach not only strengthens both tokenization and detokenization but also simplifies the learning of generative models. Furthermore, existing tokenizers predominantly adopt convolutional architectures, whose local receptive fields struggle to support global language guidance. To this end, we propose a Transformer-based Tokenizer that leverages attention mechanisms to enable effective alignment between language and motion. Additionally, we design a language-drop scheme, in which language conditions are randomly removed during training, enabling the detokenizer to support language-free guidance during generation. On the HumanML3D and Motion-X generation benchmarks, LG-Tok achieves Top-1 scores of 0.542 and 0.582, outperforming state-of-the-art methods (MARDM: 0.500 and 0.528), and with FID scores of 0.057 and 0.088, respectively, versus 0.114 and 0.147. LG-Tok-mini uses only half the tokens while maintaining competitive performance (Top-1: 0.521/0.588, FID: 0.085/0.071), validating the efficiency of our semantic representations. Code and checkpoints are available at https://eanson023.github.io/LG-Tok/