Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue

AI summary

Overview

Research area: Speech and audio processing (cs.SD), at the intersection of spoken dialogue models, co-speech full-body motion generation, and multimodal joint training.

Technical level: Advanced. The paper assumes familiarity with discrete speech tokenizers, VQ-VAE motion codebooks, autoregressive LLM decoders, and standard co-speech motion metrics (FGD, BC, LSE-C/LSE-D).

Scope: The paper introduces Motion-Omni, an end-to-end framework and reference model (Motion-Omni-Q7) that generates a spoken dialogue response together with facial expression, hand, upper-body and lower-body motion directly from the hidden states that produce the speech, plus a pseudo-labeling data pipeline, a new evaluation set (SwDA-500), and an evaluation protocol for open-ended spoken dialogue with full-body motion.

What This Paper Is About

Today, an avatar that both talks and gestures usually runs two separate models in sequence: a spoken dialogue model produces the speech, then a co-speech motion model is run over the finished audio. That cascade costs a second full inference pass and means motion supervision can never update the speech or dialogue parameters. Motion-Omni asks whether full-body motion can instead be a native output of a spoken dialogue model — generated from the same internal states that produce the speech — without losing motion quality or spoken-dialogue ability.

Key Contributions

1. An end-to-end spoken motion framework. Motion is generated from the Speech Generator's hidden states rather than from a rendered waveform, and motion supervision jointly updates the LLM, Speech Generator and Motion Generator. Ablations show this co-adaptation is necessary for speech-motion alignment, and the resulting model matches the same-audio teacher cascade on motion quality while removing its separate audio-to-motion stage and responding 5.4 times faster.

2. A scalable, model-agnostic supervision pipeline. A replaceable motion teacher (LOM in this instantiation) pseudo-labels consistent-voice speech responses, and a dual-metric quality score drives a curriculum. Applied to InstructS2S-200K, this yields 422,856 paired speech-motion samples (1,402 hours).

3. SwDA-500 and an evaluation protocol. A 500-prompt dialogue-text evaluation set derived from the Switchboard Dialog Act Corpus, plus what the authors describe as the first publicly released evaluation protocol for stochastic open-ended full-body spoken dialogue, matching audio across motion systems and unifying rendering, automatic metrics, human evaluation, and latency measurement.

4. Four-component architecture with a four-stage training curriculum. Speech Projector, LLM Backbone, Speech Generator and Part-Aware Motion Generator, bridged across a rate mismatch between 12.5 Hz speech units and 30 Hz motion.

Main Findings

• Speech intelligibility: On Seed-TTS-Eval (English split, 1,088 samples), Motion-Omni-Q7 reaches 2.62% word error rate on test-en — the lowest among the omni-modal LLMs compared (Qwen2.5-Omni 2.72, Ex-Omni 2.67). Several dedicated TTS systems report lower WER on this speech-only metric, including Qwen3-TTS-12Hz-1.7B (1.24) and CosyVoice 3-1.5B RL (1.45); the Human Reference row is 2.14 and is not ranked.

• Dialogue ability retained: On VoiceBench, a nine-subset benchmark, Motion-Omni-Q7 reaches an Overall of 47.63, above LLaMA-Omni (41.12), Ex-Omni (43.57), Mini-Omni2 (33.49) and Moshi (29.51).

• Naturalness proxy: UTMOSv2 on SwDA-500 generated audio gives Motion-Omni-Q7 3.77 — above GLM-TTS, VoxCPM1.5, F5-TTS and CosyVoice, and below Qwen3-TTS and CosyVoice 3.

• Motion quality: On SwDA-500, Motion-Omni-Q7 places first or second on seven of eight metrics. Among systems that do not run LOM at motion-inference time it obtains the highest beat correlation (7.59), the highest diversity (13.67), and the best score on every facial and lip-sync metric (LSE-C 7.011, LSE-D 7.023, MSE 6.723, LVD 6.427). Its teacher-reference FGD is 3.03.

• Matching the teacher cascade: The model matches the same-audio teacher cascade to within 2% on reference-free motion metrics (the conclusion states "within about 1%"). The rows that surpass it are the LOM teacher-reference cascades, which invoke at motion-inference time the same model that supplied its training targets.

• Human evaluation: Four trained young-adult male annotators (mean age approximately 26) rated 25 paired clips per arm on rhythm, semantic alignment and body naturalness. Against MO-audio + EMAGE the pooled margin is +25 (45 wins, 10 ties, 20 losses). Against MO-audio + LOM — the teacher itself — the outcome is 25/27/23 (+2), with ties the single most frequent verdict. The authors state that with four annotators and 25 pairs per arm, the smaller margins are not read as evidence of a reliable preference.

• Latency: Motion-Omni-Q7 completes a full response in 4.32 s (RTF = 0.78, faster than real time). The matched MO-audio + LOM cascade takes 23.35 s (RTF 4.39), 5.4 times slower. Qwen2.5-Omni + LOM takes 99.08 s (RTF 18.34). MO-audio + EMAGE matches the response time (4.63 s, RTF 0.84) but gives up 0.27 BC; MO-audio + LOM exceeds Motion-Omni-Q7's BC by 0.08 at 5.4 times the response time.

• Curriculum effect on motion: Stage-wise teacher-reference FGD decreases from 0.3974 at Stage 3a to 0.3040.

• Joint training is required: In an initial pilot that kept the Speech Generator frozen, the motion loss plateaued above the level reached by joint training and the rendered motions were clearly misaligned with the speech audio.

Methodology in Plain English

The system has four parts. A frozen Whisper-large-v3 encoder (hidden dimension 1280) turns the user's 16 kHz waveform into continuous features, and a speech projector concatenates every five frames and passes them through a two-layer MLP into the LLM's embedding space, downsampling fivefold. A Qwen2.5-7B-Instruct backbone processes those projected features plus text tokens. A Speech Generator initialised from Qwen2.5-0.5B-Instruct then emits GLM-4-Voice discrete speech units at 12.5 Hz from a vocabulary of 16,384 units (plus three control tokens), using a Token-as-Query Gated Fusion block. Finally, a Motion Generator with four parallel per-part decoders emits LOM VQ codes at 30 Hz for face, hands, upper body and lower body.

The key design choice is what the Motion Generator looks at. Instead of consuming the finished audio, it attends to the Speech Generator's last-layer hidden states as keys and values, and consumes a learned embedding of the speech tokens as the query, linearly interpolated from 12.5 Hz to 30 Hz. This is what removes a separate audio-to-motion inference stage.

Training proceeds in four stages. Stage 1 trains only the speech projector with ASR supervision while the LLM is frozen. Stage 2 trains the Speech Generator on TTS-style pairs. Stage 3 attaches the Motion Generator and jointly trains it with the Speech Generator on the TTSM corpus under a four-substage curriculum exposing the network to progressively larger quality quantiles (12.5% / 25% / 50% / 100%). Stage 4 unfreezes everything and optimises a four-task ASR/TTS/S2SM/T2T mixture. The Whisper encoder and the LOM VQ-VAE stay frozen throughout.

Supervision comes from pseudo-labeling because captured audiovisual corpora contain many voices that do not match the single target voice the Speech Generator is trained to produce. A replaceable motion teacher labels every response waveform, and each sample is scored on weighted VQ-VAE reconstruction error and a beat correlation score; after 5th–95th percentile normalisation the combined score uses α = 0.5. Per-stage training data totals 4,362,198 samples (12,144.6 hours). At inference, speech units go through a CosyVoice chunk-aware flow-matching decoder and a HiFi-GAN-style vocoder to 22.05 kHz waveforms, while motion codes are decoded by the frozen LOM VQ-VAE into SMPL-X body/hand parameters and FLAME facial-expression coefficients.

Why This Matters

Impact on research. The paper tests a specific hypothesis: that motion supervision can update the same states that generate speech and improve alignment rather than degrade dialogue ability. Its ablation showing that a frozen speech pathway leaves motion misaligned is a concrete data point for joint multimodal co-learning. It also argues that existing full-body motion benchmarks, which assume fixed supplied speech, cannot evaluate a model whose own spoken response is valid but unpredictable, and releases a protocol to fill that gap.

Real-world applications.

• Conversational avatars and virtual agents in customer service or information desks, where a single pass produces speech and body language.
• Telepresence and virtual meeting stand-ins that need matching face, hand and body movement while speaking.
• Game and interactive-media NPCs with dialogue-driven full-body performance.
• Accessible interfaces and digital humans for education or training, where gesture reinforces what is being said.

Industry relevance. The headline practical benefit is latency: eliminating the second audio-to-motion pass gives RTF = 0.78, faster than real time, and 5.4 times faster than the same speech model paired with the teacher motion model as a cascade. The pipeline is model-agnostic and its components are swappable, so stronger backbones or teachers can be substituted by re-running the recipe.

Future Directions

• Better motion coverage. Motion quality is bounded by the LOM VQ-VAE codebook and the teacher pseudo-labels, so motion outside that distribution cannot be expressed. The authors leave stronger backbones, larger co-speech motion generation teachers and continuous motion heads to future work.
• Streaming interaction. Motion-Omni-Q7 ingests the whole user utterance before emitting the first response token, making it an offline response generator. Achieving low-latency interactive behaviour as in streaming systems such as MIBURI requires a different design and evaluation protocol, and the authors note their response times are not directly comparable to MIBURI's.
• Identity, emotion and language. The current model does not condition on explicit speaker identity or emotion, and all training data is English, restricting generalisation to other languages and motion cultures.
• Evaluation scale and metrics. Existing automatic motion metrics (FGD, BC) are imperfect proxies for perceived naturalness, the four-annotator study is exploratory, and a larger non-author human study remains a limitation. The authors also report that an initial attempt to scale rubric scoring with a video-input LLM-as-judge correlated with human raters only on speech quality.

Target Audience

Researchers and engineers working on spoken dialogue systems, co-speech gesture and full-body motion generation, and multimodal LLMs. It is most useful to readers already comfortable with discrete token representations, autoregressive generation and motion metrics, and to practitioners who need to decide between a cascade and a jointly trained model for an avatar pipeline. Those focused on streaming, real-time interaction, non-English languages, or identity- and emotion-conditioned motion will find the paper's own limitations section directly relevant.

Authors’ abstract

An avatar that holds a conversation should decide what to say and to move while saying it, yet these abilities live in separate model families: spoken dialogue models produce speech without motion, and co-speech motion models produce motion only from audio handed to them. The standard remedy is a cascade that first generates the spoken response and then runs a motion model over the finished audio, which requires a second full inference pass and precludes any joint optimisation between the two. We present Motion-Omni, an end-to-end framework in which a spoken dialogue model natively outputs explicit facial expression together with hand, upper-body and lower-body motion, generated directly from the hidden states that produce the speech. Joint training is not optional here: with the speech pathway frozen, motion remains misaligned with the audio, and co-adapting the LLM, Speech Generator and Motion Generator under both objectives is what recovers alignment while retaining spoken-dialogue ability. Supervision comes from a scalable, model-agnostic pipeline that pseudo-labels consistent-voice speech responses with a replaceable motion teacher, yielding 422,856 quality-ranked pairs (1,402 hours). We further release SwDA-500 and, to our knowledge, the first public evaluation protocol for stochastic open-ended full-body spoken dialogue, matching audio across motion systems while unifying rendering, automatic metrics, human evaluation, and latency measurement. Instantiated with a Qwen2.5-7B-Instruct backbone, Motion-Omni-Q7 matches the same-audio teacher cascade to within 2% on reference-free motion metrics while responding 5.4 x faster (RTF=0.78, faster than real time), surpasses all non-teacher cascades on beat correlation and diversity, and reaches a 2.62% word error rate, the lowest among the omni-modal systems compared.