Do Motion Tokenizers for Co-Speech Gesture Generation Encode Gesture Semantics?

AI summary

Overview

• Research area: Computer vision / co-speech gesture generation, specifically the interpretability of discrete motion tokenizers (VQ-VAE codebooks) used to represent human body motion.
• Technical level: Intermediate. The probing methodology and codebook architecture are explained clearly, but readers will benefit from familiarity with vector quantization, linear probing, and gesture-annotation terminology.
• Scope in one sentence: The paper probes a reconstruction-trained gesture codebook with 19 descriptors spanning raw motion to communicative function, and asks which of those properties are actually recoverable from the tokens.

What This Paper Is About

Co-speech gesture systems increasingly convert continuous motion into sequences of discrete tokens drawn from a learned codebook, but those codebooks are trained only to reconstruct motion accurately. The authors ask whether a codebook optimized purely for reconstruction nonetheless preserves the properties that make a gesture meaningful — handshape, motion category, communicative intent — or whether those properties are lost at the tokenization stage. To answer this, they probe one codebook with 19 gesture descriptors and check whether the results hold on a second, architecturally different tokenizer.

Key Contributions

1. A systematic probing framework across 19 descriptors. The authors organize descriptors into six groups spanning raw motion to abstract communicative function, applied to 778 semantic gesture clips from TED.
2. Two-level analysis of the codebook. They combine linear and shallow nonlinear probes over pooled token embeddings with an information-theoretic analysis of discrete code-activation patterns, distinguishing "information is accessible in the averaged vector" from "information lives in which codes are used."
3. A link between reconstruction fidelity and descriptor accessibility. They correlate probe macro-F1 with mean per-joint position error (MPJPE) across 27 model configurations to test whether better reconstruction implies better semantic accessibility.
4. A demonstration that the pattern generalizes across architectures. The same accessibility gradient is reproduced on EMAGE, a second tokenizer, indicating the finding is not specific to one design.

Main Findings

• Geometry and handedness are strongly decodable. Spatial geometry reaches a group-level mean macro-F1 of 0.77 (linear) and handedness 0.73, with geometry significant for all 9 descriptors and handedness for 1/1. At the descriptor level, right depth reaches 0.912 and left depth 0.898 macro-F1.
• Higher-level descriptors are only weakly decodable. Group-level linear macro-F1 drops to 0.30 for annotated spatial form, 0.16 for handshape, and 0.11 for motion pattern and function. Motion pattern and function is essentially unchanged by a shallow MLP probe (0.113 to 0.112), so weak linear accessibility is unlikely to reflect nonlinear encoding alone.
• Motion category leaves a hidden code-level trace. Although motion category is weakly decodable from pooled embeddings, its group-level mutual information is the highest reported (3.62), driven by motion category itself (MI = 3.8, q = 0.006). The information is carried by which codes are activated, not by the averaged embedding geometry.
• Communicative intent and handshape do not show reliable code-level separation. Communicative intent yields q = 0.06 and handshape q = 0.70 in the code-activation analysis, showing no reliable overall separation.
• Reconstruction quality tracks motion-grounded properties. Negative correlations between probe macro-F1 and MPJPE are strongest for handedness (the text reports r = −0.88; Table 1 lists −0.89 for the handedness group) and spatial geometry (r = −0.69), and weaker for handshape (r = −0.67) and motion pattern and function (r = −0.45).
• The gradient is stable across tokenizers. Repeating the analysis on EMAGE shows the same ordering of descriptor groups — spatial geometry strongest, handshape weak — though absolute scores differ slightly.
• Qualitative cases corroborate the probing results. The paper shows reconstruction-only VQ-VAEs preserve upper-body geometry but average out fine-grained hand poses, which the authors attribute to the scale difference between hand and upper-body joints in the reconstruction loss.

Methodology in Plain English

The authors take an already-trained motion codebook and ask what a simple classifier can read out of it.

1. Pick a dataset and describe it. They use 778 semantic gesture clips from TED, drawn from the Semantix dataset and building on gesture regions identified by Ram et al. (2025). Every clip gets 19 descriptors: some computed automatically from the 3D skeleton (joint positions, elbow bend, hand reach), and some manually annotated (handshape, motion category, communicative intent, hand orientation, spatial form). Descriptors are grouped into six families, and classes with fewer than five clips are dropped.
2. Represent each gesture two ways. Either as the mean-pooled codebook embedding, or as a normalized histogram of which discrete codes fired during the clip.
3. Train simple probes. For each descriptor, they fit multinomial logistic regression with balanced class weights and report 5-fold cross-validated macro-F1 against majority-class and random-assignment baselines. A shallow MLP with one 128-unit ReLU layer and L2 regularization checks whether weak linear results are actually hiding nonlinear structure.
4. Analyze the codes directly. They compare class-average code-usage profiles using mean pairwise Jensen–Shannon divergence, with label-permutation tests and Benjamini–Hochberg correction across descriptors. They also sum the mutual information between each active code's presence and the descriptor label — an additive measure that avoids estimating a full joint table over all 2048 codes.
5. Test whether reconstruction quality explains accessibility. They sweep a 3×3×3 grid of 27 models (latent dimension L in {64, 128, 256}, codebook size K in {1024, 2048, 4096}, temporal downsampling d in {4, 8, 16}), then correlate macro-F1 with MPJPE.
6. Repeat on a second tokenizer. The same pipeline runs on EMAGE.

The primary analysis uses the best-reconstructing configuration: L = 256, K = 2048, d = 4. The underlying codebook is a two-stream VQ-VAE over a 38-joint upper-body skeleton, split into an 8-joint body stream and a 30-joint hand stream with separate encoders and codebooks but a single shared decoder. It is trained on a combined BEAT and TED gesture corpus (native frame rates 30 FPS and 24–30 FPS, balanced by duration) with Adam, learning rate 1×10⁻³, batch size 8, up to 25 epochs with early stopping, minimizing position reconstruction loss plus a VQ commitment loss weighted by β = 1.0.

Why This Matters

Impact on research. The paper argues that reconstruction accuracy alone is not a sufficient measure of what a motion tokenizer has learned. A codebook can reconstruct well while failing to preserve communicatively important distinctions, which means benchmarks that report only reconstruction error may give a misleading picture of tokenizer quality. The two-level probing protocol — pooled-embedding probes plus code-activation analysis — offers a template for evaluating codebooks on interpretable properties rather than one aggregate error number.

Real-world applications:

• Sign language and gestural interfaces. Systems that rely on fine hand configuration (a cupped or pinched hand) may lose exactly the distinctions those configurations carry if motion is passed through a reconstruction-only tokenizer.
• Virtual agents and avatars. If communicative intent is weakly encoded, generated gestures may look smooth and plausible while failing to convey the intended emphasis, negation, or enumeration.
• Human-robot interaction. Robots that interpret gestures as commands depend on handshape and motion category being recoverable from whatever motion representation they use internally.
• Automated gesture analysis and annotation tools. Researchers building tools that categorize gestures (iconic, metaphoric, and so on) need representations that preserve those categories.

Industry relevance. Any product pipeline that compresses motion into discrete tokens — avatar animation, motion capture post-processing, gesture-driven user interfaces — inherits whatever the tokenizer discards. The paper's finding that reconstruction quality mainly improves motion-grounded properties suggests that teams optimizing only for reconstruction loss may be optimizing the wrong objective for applications where gesture meaning matters.

Future Directions

1. Test other domains, motion representations, and tokenizer objectives. The authors note their analysis covers only 778 TED gestures and two skeleton-based, reconstruction-trained tokenizers, so the accessibility gradient may not generalize.
2. Ablate the training corpus. The codebook was trained on BEAT and TED jointly, and the contribution of each dataset to descriptor accessibility remains untested.
3. Move beyond pooled representations to sequence-aware probes. The authors stress that low probe performance indicates limited recoverability under the selected pooled representations and probe families, not that a property is entirely absent from the representation.
4. Test whether descriptor accessibility predicts the semantic quality of generated gestures. This is posed as an open question, along with using larger and more diverse datasets and addressing the sparsity and subjectivity of the manual handshape and communicative intent annotations.

Target Audience

Researchers working on co-speech gesture generation, discrete motion tokenization, and representational probing; practitioners building avatar, motion-capture, or gesture-interface systems who need to understand what their motion compression pipeline discards; and anyone interested in how objective-driven training shapes what a learned representation actually captures. Readers without a background in gesture studies will still follow the central argument, but the descriptor taxonomy and probing setup assume some familiarity with representation-learning evaluation practice.

Authors’ abstract

Discrete motion tokenizers encode motion as atomic units and are widely used for co-speech gesture generation. It remains unclear which motion properties, especially those relevant to gesture semantics, are recoverable from these codebooks. We probe a reconstruction-trained codebook using 19 co-speech gesture descriptors spanning from raw motion to abstract communicative function. Results show that geometry and handedness are readily decodable from token embeddings, while motion category is only weakly decoded despite showing systematic differences in discrete code usage. This gap between reconstruction quality and descriptor decodability suggests that reconstruction objectives alone do not guarantee that gesture semantics are captured, and that evaluating codebooks on such properties can guide the design of more semantic motion tokenizers.