MIBURI: Towards Expressive Interactive Gesture Synthesis

AI summary

Overview

Research area: Computer Vision / graphics — co-speech gesture synthesis and Embodied Conversational Agents (ECAs).

Technical level: Advanced. The paper combines speech-language modeling, residual vector-quantized autoencoders, and autoregressive transformer design; the summary below is written to be readable without that background, but the underlying method assumes familiarity with generative motion models.

Scope: The paper presents Miburi, an online causal framework that generates full-body gestures and facial expressions synchronized with real-time spoken dialogue, built on top of the speech-text foundation model Moshi.

What This Paper Is About

Conversational agents driven by large language models can talk but have no body: they cannot produce the gestures and facial expressions that carry meaning in human face-to-face interaction. Existing gesture-generation research splits into two unsatisfying groups — rule-based or data-driven ECA systems that run in real time but produce rigid, low-diversity motion, and generative co-speech gesture models that look natural but need future speech context and long runtimes. Miburi's goal is to close that gap by generating expressive full-body gesture and facial motion causally, in real time, straight from the internal token stream of a spoken dialogue model.

Key Contributions

1. A new paradigm for online gesture generation. Instead of the conventional pipeline where an LLM's text is converted to speech, then re-tokenized to condition a gesture model, Miburi taps directly into the internal speech and text token stream of Moshi, avoiding the latency-inducing intermediate steps while reusing Moshi's semantic and acoustic context.
2. A causal architecture and tokenization strategy that preserve expressiveness. Body-part-aware gesture codecs (separate codecs for face, upper body, and lower body) encode motion into multi-level discrete tokens via Residual VQ-VAE, and a two-dimensional causal generator with a temporal transformer and a kinematic transformer predicts those tokens autoregressively without needing future context.
3. Auxiliary objectives for expressiveness. A contrastive InfoNCE loss applied to Gumbel-Softmax-approximated latents encourages diverse, speech-aligned motion, and a voice activation loss (binary listening vs. speaking classification) suppresses "phantom" gestures during listening.
4. A comprehensive design analysis. Perceptual and numerical experiments plus ablations over speech encoding, transformer design, loss terms, and number of codebook levels, comparing Miburi against recent baselines in single-speaker and 23-speaker settings.

Main Findings

• Real-time latency. The online demo system achieves a latency of 36ms per frame on an RTX3090, including model runtime and rendering on a web dashboard. In the controlled comparison (wall-clock time from the beginning of the forward pass to conversion of outputs into SMPL-X parameters, render times excluded), Miburi records 0.0349 ± 0.0017 seconds on an A100, versus GestureLSM (8 steps) at 0.1447 ± 0.0034, MambaTalk at 0.0529 ± 0.0039, and EMAGE at 0.0374 ± 0.0004 — the lowest latency of the compared methods, with only 2 frames generated per step versus 124 (GestureLSM) and 60 (EMAGE, MambaTalk).
• Scaling to multiple speakers. Miburi achieves the best results among compared methods in the 23-speaker setting, reaching FGD 0.480 and BeatAlign 0.461 (Miburi (+Face)), against GT at FGD 0.446 and BeatAlign 8.45, RAG-Gesture at 0.515 / 0.648, GestureLSM at 0.537 / 0.481, and EMAGE* at 0.850 / 0.236. The paper states an FGD of 0.480 is state-of-the-art metric performance in FGD when trained with more speakers.
• Naive causality hurts baselines. Converting real-time baselines to causal operation degrades them sharply — GestureLSM (Causal*) drops to FGD 2.792 and MambaTalk (Causal*) to 1.222 — which the authors read as evidence that current architectures depend on future speech context for quality.
• Single-speaker trade-off. On the single-speaker (scott) setting, Miburi (+Face) records the best BeatAlign listed at 0.790, but baselines that use ground-truth seed sequences achieve lower FGD (GestureLSM 0.410, EMAGE 0.552 versus Miburi 0.806). Miburi's single-speaker CFG scale is 1.5; multi-speaker is 2.3.
• Perceptual evaluation. In pairwise user studies, Miburi produces more natural and appropriate motion than non-causal baselines such as EMAGE and GestureLSM, but has not reached the quality or speech appropriateness of ground-truth data. The paper reports significance at p < 0.05 and p < 0.001 against a 50% chance level.
• Moshi's internal tokens beat wav2vec. Replacing Moshi's internal token stream with standard wav2vec encoding worsens FGD and BeatAlign — for Miburi (+Face), 0.665 / 0.363 with wav2vec versus 0.480 / 0.461 with Moshi features — and adds the extra computation cost of computing audio embeddings.
• Two transformers beat one. A single transformer handling both the temporal (T) and kinematic (K) dimensions gives FGD 1.256, BeatAlign 0.731, L1-Div 5.48 and step time 0.096 seconds, versus Miburi's 0.480 / 0.461 / 10.44 at 0.035 seconds.
• Contrastive loss helps; direct MSE hurts. Starting from L_CE + L_va (FGD 0.499), adding the contrastive loss improves FGD to 0.480, while applying a direct MSE loss on estimated latents degrades FGD to 0.577.
• More residual codebook levels improve reconstruction. MPJPE falls from 0.043 m at K=1, to 0.032 m at K=2, 0.022 m at K=4, and 0.016 m at K=8, with FGD following a similar pattern (0.55, 0.42, 0.135, 0.059 respectively).
• Larger models are not better. Training larger versions of Miburi in the multi-speaker setting showed that leaner versions are equivalent or better.
• Causality is distinct from speed. The paper emphasizes that causal models use only past inputs without latency constraints, while real-time interactive systems must additionally meet strict timing requirements; MambaTalk is based on the causal Mamba architecture yet its cross-attention speech conditioning works against low-latency output.

Methodology in Plain English

Miburi builds on Moshi, an open-source spoken dialogue system that produces text and speech tokens in parallel streams for full-duplex conversation. Rather than waiting for that system's outputs to be turned into audio and re-encoded, Miburi reads Moshi's internal token embeddings directly — they already carry semantic and prosodic information — and uses them as the conditioning signal for gesture generation.

Generating motion requires converting continuous poses into discrete units, so the authors train separate Residual VQ-VAE codecs for three body regions: upper body with hands, lower body with global translation and foot contacts, and facial expressions represented with FLAME parameters. Each codec's encoder uses causal attention and downsampling convolutions to produce multi-level discrete tokens, and its decoder reconstructs the motion. Using residual quantization rather than a single codebook captures fine detail, from large arm movements down to subtle finger gestures.

Predicting all temporal and kinematic tokens in one sequence would blow up the attention context, so generation is split across two transformers: a temporal transformer predicts the first level token at each timestep from previous timesteps plus speech, text, and identity embeddings; a kinematic transformer then fills in the remaining levels at that fixed timestep conditioned on the temporal context. Both are trained with cross-entropy and teacher forcing.

To avoid the mean-pose collapse and drift that plague autoregressive motion models, the authors add two objectives. A contrastive InfoNCE loss operates on latents obtained by passing the transformers' logits through Gumbel-Softmax and the RVQ codebooks, since discrete sampling is otherwise non-differentiable. A voice activation loss classifies each timestep as listening or speaking, which discourages gestures during listening. The total objective is L_CE + α L_con + β L_va with α = 0.1 and β = 0.01.

For speed, key and value caches retain past context, self-attention is limited to 25 tokens and cross-attention to 50, and cross-attention for lower-body tokens is masked out because the lower body relates weakly to speech. Training used the BEAT2 dataset at 25 FPS with T = 125 (a 10-second sequence); the temporal transformer has 4 layers and 2 attention heads, the kinematic transformer 2 layers and 1 attention head, and optimization used AdamW with a starting learning rate of 1e-4. Because Moshi emits 12.5 tokens per second (0.08 seconds of audio each), Miburi generates 2 frames of gesture at each step. Inference uses top-p sampling (0.8 for the temporal transformer, 0.95 for the kinematic transformer, softmax temperature 0.9) plus classifier-free guidance.

Why This Matters

The paper reframes gesture synthesis as an integral part of a spoken dialogue system rather than a separate offline animation stage. By conditioning on a dialogue model's internal tokens and generating gestures causally at the same rate as the speech, it makes gesture generation compatible with the streaming, full-duplex interaction patterns that LLM-based agents are moving toward. It also provides evidence that the expressiveness of offline diffusion methods and the low latency of ECA systems need not be mutually exclusive, while documenting the cost of simply making existing architectures causal.

Real-world applications:

• Virtual assistants and embodied avatars in customer service or kiosks, where an agent must respond while the user is still talking rather than after a turn ends.
• Telepresence and remote collaboration, including driving a virtual or robotic body with synchronized speech and gesture in real time.
• Games and virtual worlds, where non-player characters need continuous, low-latency, speech-aligned body language instead of looping animations.
• Accessibility and education, such as sign-adjacent or language-learning tutors whose visible body motion reinforces spoken content.
• Content creation and virtual production, where a speaker's streamed audio can drive an animated character live.

Industry relevance: the method's dependence on an open-source speech-text foundation model and its low per-frame latency suggest it could be integrated into existing conversational-AI stacks, though it currently models only the agent's motion.

Future Directions

• Dyadic and multi-party gesture. The stated limitation is that Miburi models only the agent's motion and ignores the user's body dynamics and full dyadic context; extending it to perceive and respond to a partner's gestures is named as the key future direction.
• Closing the gap to ground truth. The perceptual study found Miburi has not reached the quality or speech appropriateness of ground-truth motion, leaving headroom in expressiveness and alignment.
• Larger-scale and cross-dataset validation. The paper reports an evaluation on the recently released Embody3D dataset in the supplemental material; broader validation beyond BEAT2, which was reduced from 25 to 23 speakers after removing carla and itoi, would test generalization.
• Model scaling behavior. Since larger versions of Miburi performed equivalently or worse than leaner ones in the multi-speaker setting, the relationship between capacity, data, and gesture quality remains an open question.

Target Audience

Researchers and practitioners working on co-speech gesture synthesis, Embodied Conversational Agents, and real-time avatars; engineers building spoken dialogue systems who need to add body language under latency constraints; and graphics or animation researchers interested in discrete motion tokenization, residual vector quantization, and autoregressive generative models for human motion.

Authors’ abstract

Embodied Conversational Agents (ECAs) aim to emulate human face-to-face interaction through speech, gestures, and facial expressions. Current large language model (LLM)-based conversational agents lack embodiment and the expressive gestures essential for natural interaction. Existing solutions for ECAs often produce rigid, low-diversity motions, that are unsuitable for human-like interaction. Alternatively, generative methods for co-speech gesture synthesis yield natural body gestures but depend on future speech context and require long run-times. To bridge this gap, we present MIBURI, the first online, causal framework for generating expressive full-body gestures and facial expressions synchronized with real-time spoken dialogue. We employ body-part aware gesture codecs that encode hierarchical motion details into multi-level discrete tokens. These tokens are then autoregressively generated by a two-dimensional causal framework conditioned on LLM-based speech-text embeddings, modeling both temporal dynamics and part-level motion hierarchy in real time. Further, we introduce auxiliary objectives to encourage expressive and diverse gestures while preventing convergence to static poses. Comparative evaluations demonstrate that our causal and real-time approach produces natural and contextually aligned gestures against recent baselines. We urge the reader to explore demo videos on https://vcai.mpi-inf.mpg.de/projects/MIBURI/.