ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Body

AI summary

Overview

Research area: Computer Vision, specifically multimodal motion generation, 3D avatars, and conversational AI (speech–language–behavior modeling).

Technical level: Advanced. The paper assumes familiarity with transformer architectures, mixture-of-experts routing, rotary positional embeddings, autoregressive token prediction, and parametric human body/face models (SMPL-X, FLAME).

Scope: The paper introduces ViBES, a unified speech–language–behavior model that jointly plans language and 3D body movement for a conversational agent, together with the roughly 1,000-hour Converse3D audio–text–motion dataset and a new conversational-behavior benchmark.

What This Paper Is About

Most existing systems treat human body language as a translation problem: a fixed utterance (audio or text) is mapped to a motion clip, with no decision-making about when to move, what to do, or how to stay consistent across a multi-turn dialogue. ViBES instead reframes non-verbal behavior as a core component of conversational intelligence, building a single agent that reasons over interleaved speech, text, facial expression, and body-motion tokens and generates dialogue-conditioned behavior. The goal is an agentic 3D virtual body that answers questions, gestures appropriately, and can also follow explicit body-action directives such as "Could you step back and wave?"

Key Contributions

1. ViBES, a speech–language–behavior (SLB) model with a mixture-of-modality-experts (MoME) backbone. Three Transformer experts — a text–speech (TS) expert, a facial-expression expert, and a body-motion expert (upper body, lower body, hands) — are coupled through Speech–Language–Behavior Attention (SLB-Attn). Routing is hard and deterministic by modality rather than learned.

2. A lightweight, side-car expert design that preserves a pretrained backbone. The face and body experts are small Transformer blocks (40 layers, hidden size 512, FFN dimension 4096, roughly 430M parameters) that generate queries and read keys/values from the frozen text–speech stream. The backbone (GLM-4 has 40 layers, hidden size 4096, FFN dimension 13,696, roughly 9B parameters) stays intact, so the method avoids large-scale audio–text–motion pretraining.

3. Converse3D, a roughly 1,000-hour audio–text–motion corpus. It combines in-the-wild YouTube conversation videos (interviews, podcasts, talks, casual dialogues) with existing motion datasets — HumanML3D (~24K text–motion pairs), BEAT2 (~70 hours audio–motion), Embody3D (~50 hours audio with full body), and TFHP — plus conservative synthesis to fill missing modalities. All streams are aligned to a unified 25 fps clock.

4. A new benchmark for conversational behavior on a Converse3D test set of 4,921 samples (3,128 motion-descriptive from AMASS, 1,793 conversational from BEAT2), using a Question-to-Motion (Q2M) contrastive evaluator with a CLIP-style architecture, balanced R-Precision, Matching Distance, FID, and Diversity, plus multimodal LLM judges.

Main Findings

• Conversational behavior benchmark (Table 2): ViBES reaches R1-Balanced 0.467, R1-Motion 0.308, R1-Conv 0.745, R3-Balanced 0.671, MM Dist 3.178, FID 93.9, and Diversity 10.73. For comparison, LoM (response audio input) obtains 0.323 / 0.025 / 0.621 / 0.498 / 3.435 / 373.8 / 2.49, and MoMask (question text input) obtains 0.293 / 0.159 / 0.524 / 0.528 / 3.675 / 265.5 / 7.38. Ground truth is 0.712 / 0.546 / 0.999 / 0.883 / 2.342 / 0.0 / 11.05.

• Speech metrics (Table 3): ViBES records Context Relevance 4.584 and Character Consistency 4.376, versus the SynMSI ground-truth upper bound of 4.838 and 4.893, and versus SOLAMI (LoRA and full-parameters variants, both 0.824 / 3.634) and LLM+Speech with Llama2 (3.859 / 3.157). The authors attribute this to inheriting competence from the pretrained GLM-4-Voice backbone.

• Talking-head synthesis (Table 4, TFHP test split): ViBES achieves LVE 10.87, MOD 2.29, and FDD 6.05, against DiffPoseTalk (11.01 / 2.55 / 37.68) and ARTalk (11.67 / 2.46 / 20.77). The paper notes LVE computation is inconsistent across the literature and follows the ARTalk protocol, recomputing all baselines with their released checkpoints.

• Co-speech gesture generation (Table 5, BEATv2, speaker-independent split): ViBES reports FGD 5.257, BC 8.103, and Diversity 13.028, compared with LoM (5.301 / 7.780 / 15.167) and EMAGE (5.512 / 7.724 / 13.060). FGD and BC are scaled by 10⁻¹.

• Text-to-motion: The paper states ViBES shows stronger linguistic grounding and handles rare or long-form instructions more reliably, and provides qualitative examples, but the excerpt does not report a quantitative text-to-motion results table.

• Ablation finding: Enabling face-to-body attention yields no measurable improvement. Once both streams are conditioned on the text–speech stream, the face and body streams appear largely independent, so face↔body attention is disabled by design.

• Timing and representation design: Audio tokens are produced at 12.5 fps and all motion streams are resampled to 25 fps. Face tokens are produced at 25 fps; body tokens use compositional tokenization (upper body, lower body, hands) with 4× temporal downsampling, giving 6.25 fps; the HumanML3D setting follows MotionGPT and also yields 6.25 fps tokens.

Methodology in Plain English

The researchers start from a strong pretrained speech-language model rather than training a new multimodal system from scratch. They convert every modality into discrete tokens: text through Tiktoken from GLM-4-9B, speech through the 12.5 Hz tokenizer variant from GLM-4-Voice (fine-tuned on whisper-large-v3 with a pooling layer and a vector-quantization layer), facial motion through FLAME parameters (shape in R¹⁰⁰, expression in R¹⁰⁰, 6D pose in R⁶), and body motion through SMPL-X parameters (shape in R³⁰⁰, joint poses in R⁵⁵).

These tokens are interleaved into one sequence with modality labels. Each modality group has its own feed-forward network and LayerNorm, and its own query/key/value/output projections. Text and speech tokens attend to each other normally; face and body tokens produce their own queries but read keys and values only from the text–speech stream. Because the face and body experts are side-car modules that only read from the backbone, the original model weights stay untouched.

A key technical piece is multi-modal fractional RoPE. Because different modalities run at different frame rates, the authors assign each token a scalar index on a single rotary timeline. Text and speech tokens get integer indices, and motion tokens get fractional indices computed by interpolating between surrounding text–speech anchors using wall-clock timestamps (with left and right extrapolation using the median anchor interval outside the anchor range). This keeps cross-modal timing precise within one shared positional encoding.

To train this, they build Converse3D: they process YouTube conversation videos with voice activity detection, word-level ASR, and speaker diarization, then estimate SMPL-X body and hands plus FLAME face parameters with monocular methods, resample everything to 25 fps, and filter low-confidence geometry and misaligned audio/text. They consolidate existing motion datasets by retargeting to SMPL-X/FLAME, and synthesize missing modalities conservatively — ARTalk refines lip sync into FLAME trajectories aligned to audio, LoM completes occluded hands only, and GPT-4o generates situational prompts rendered with a high-fidelity TTS when only motion and a caption exist.

For evaluation, the model must retrieve the correct motion for motion-descriptive questions (strict 1-to-1 matching) and any conversational gesture for open-ended questions (relaxed matching), in batches containing 16 motion-descriptive and 16 conversational samples. Motions are represented as 22 body joints in 6D rotation plus root velocity.

Why This Matters

Research impact: The paper argues for reframing non-verbal behavior as part of conversational intelligence rather than a downstream translation task. It offers a practical recipe for adding new modalities to a frozen pretrained backbone through cross-attention reads only, and it contributes both a large-scale aligned corpus and a benchmark that separates semantic action execution from ambiguous conversational gesturing.

Real-world applications:

• Embodied virtual assistants and NPCs in games and VR that gesture and emote while answering.
• Human–robot interaction, where robots need to communicate and react to humans in a human-like way (the paper explicitly cites robotics).
• Video avatar generation — the supplementary material shows ViBES driving avatar videos with Runway AI conditioned on its generated head motions.
• Telepresence and remote meetings where a controllable 3D body represents a participant.

Industry relevance: The design reuses off-the-shelf checkpoints and adds only lightweight per-layer experts, which lowers the compute barrier for companies that already serve speech-language models and want to add visual embodiment. The 3D representation also extends naturally into spatial and robotic settings, unlike 2D portrait animation.

Future Directions

• Scaling the data. The authors state the current dataset scale is still limited for fully training a speech-LLM backbone.
• Exploiting LLM reasoning. The framework does not yet use the full reasoning capability of modern LLMs, leaving room to integrate stronger planning into behavior generation.
• Better evaluation. Conversational behavior remains hard to evaluate; the authors propose building a preference model aligned with human preferences as a more reliable signal.
• Reducing reconstruction artifacts. Because Converse3D is built from web videos, it inherits artifacts from monocular reconstruction, which future pipelines would need to address.

Target Audience

Researchers and practitioners in computer vision, multimodal large language models, motion generation, and 3D avatar synthesis. The paper is also relevant to human–computer interaction and robotics groups working on socially competent embodied agents, and to engineers who want to extend a pretrained speech-language model with a controllable visual body without retraining the core.

Authors’ abstract

Human communication is inherently multimodal and social: words, prosody, and body language jointly carry intent. Yet most prior systems model human behavior as a translation task co-speech gesture or text-to-motion that maps a fixed utterance to motion clips-without requiring agentic decision-making about when to move, what to do, or how to adapt across multi-turn dialogue. This leads to brittle timing, weak social grounding, and fragmented stacks where speech, text, and motion are trained or inferred in isolation. We introduce ViBES (Voice in Behavioral Expression and Synchrony), a conversational 3D agent that jointly plans language and movement and executes dialogue-conditioned body actions. Concretely, ViBES is a speech-language-behavior (SLB) model with a mixture-of-modality-experts (MoME) backbone: modality-partitioned transformer experts for speech, facial expression, and body motion. The model processes interleaved multimodal token streams with hard routing by modality (parameters are split per expert), while sharing information through cross-expert attention. By leveraging strong pretrained speech-language models, the agent supports mixed-initiative interaction: users can speak, type, or issue body-action directives mid-conversation, and the system exposes controllable behavior hooks for streaming responses. We further benchmark on multi-turn conversation with automatic metrics of dialogue-motion alignment and behavior quality, and observe consistent gains over strong co-speech and text-to-motion baselines. ViBES goes beyond "speech-conditioned motion generation" toward agentic virtual bodies where language, prosody, and movement are jointly generated, enabling controllable, socially competent 3D interaction. Code and data will be made available at: ai.stanford.edu/~juze/ViBES/