UniLS: End-to-End Audio-Driven Avatars for Unified Listening and Speaking

AI summary

Overview

Research area: Computer vision and 3D facial animation, specifically audio-driven conversational avatar generation.

Technical level: Advanced. The work assumes familiarity with autoregressive transformers, vector-quantized codecs, and parametric 3D face models, though the core insight is explainable in plain terms.

Scope: This paper introduces UniLS, an end-to-end system that generates both speaking and listening facial motions for two conversational partners from their audio streams alone, solving the "stiff listener" problem that has blocked prior joint speak-listen approaches.

What This Paper Is About

Conversational avatars need to do two things: talk with accurate lip sync, and react naturally while the other person talks. The second half is surprisingly hard. When researchers train a model end-to-end to do both, the listening side collapses into an almost frozen face with no blinks, nods, or micro-expressions. The only prior system that handled both (DualTalk) worked around this by first generating the speaker's face, then using that motion to drive the listener—an extra step that breaks end-to-end processing and prevents real-time use. UniLS's goal is to generate unified speak-listen motion from audio alone, in real time, without the intermediate step.

Key Contributions

1. First end-to-end unified speak-listen framework. UniLS takes only dual-track audio (both speakers' voices) as input and directly outputs facial motion for both participants, with no dependency on pre-generated speaker motion. This removes the non-end-to-end bottleneck that prevented real-time deployment in prior work.

2. A two-stage training paradigm that fixes listening stiffness. Stage 1 trains an audio-free autoregressive generator to learn an internal motion prior (blinks, subtle head motion, micro-expressions). Stage 2 fine-tunes that generator with dual-track audio via cross-attention. This separates "spontaneous dynamics" from "audio-driven modulation" instead of forcing the model to learn everything from a weak audio-to-listening-motion mapping.

3. A new large-scale 3D conversational motion dataset built from Seamless Interaction. The authors track FLAME parameters for both speakers across 657.5 hours of dyadic conversational video, plus 546.5 hours of multi-scenario video, providing supervision for both speaking (251.5 hours, 22.6M frames) and listening (406.0 hours, 36.5M frames) behaviors.

4. State-of-the-art results with real-time throughput. UniLS achieves the best speaking accuracy, up to 44.1% improvement on listening distributional metrics, and runs at 560.6 FPS on a single RTX 5090 with fewer parameters (421.3M) than competing methods.

Main Findings

• Listening stiffness has a measurable cause. The authors use t-SNE to show that audio features cluster tightly with speaking motion but are scattered far from listening motion. The speaker's own speech strongly predicts their mouth and face; the interlocutor's speech only loosely predicts the listener's face. Naive joint training therefore finds a cheap solution: output a low-variance, near-neutral face for the listening branch.

• Removing audio from Stage 1 is what enables natural listening. An ablation trained with Stage 2 only ("w/o stage 1") shows degraded FDD, PDD, JDD, and FID on listening metrics. The audio-free generator supplies the spontaneous dynamics that the listening branch would otherwise never learn.

• Multi-scenario data strengthens the motion prior. Training Stage 1 only on conversational data instead of the broader multi-scenario mix (news, interviews, streams, casual talking) hurts both speaking and listening metrics, suggesting the prior generalizes better when learned from diverse facial behavior.

• Two cross-attention layers beat one mixed-audio layer. Replacing the separate speaker-A and speaker-B cross-attention with a single layer over a mixed audio stream causes a large drop in lip-sync accuracy (LVE 11.48 versus 5.83), because the model cannot disentangle the two voices.

• Speaking accuracy improves as well. UniLS reaches LVE 5.83 and MHD 1.89, better than DualTalk (6.35 / 1.95) and ARTalk (7.46 / 2.12), so the listening fix does not come at the cost of speaking quality.

• Listening metrics improve dramatically. On F-FID, UniLS scores 4.304 versus DualTalk's 13.143 and ARTalk's 10.779. Pose and jaw deviation metrics follow the same pattern, indicating the generated listening motion matches ground-truth dynamics rather than collapsing toward a static pose.

• Human evaluators strongly prefer it. In a 25-participant pairwise study, over 90% preferred UniLS over DualTalk for listening reaction naturalness, and 86% preferred it for lip synchronization.

• Real-time performance is viable. Despite higher GFLOPS (739.4), UniLS runs at 560.6 FPS because it avoids the sequential dependency chain of DualTalk, which cannot be timed at all due to its non-end-to-end design.

Methodology in Plain English

The authors work with a parametric 3D face model called FLAME, which represents any face as a set of numbers controlling expression, head pose, jaw pose, eye gaze, and identity shape. Rather than predicting raw vertices, they predict these compact parameters, which makes the learning problem tractable.

Their system is an autoregressive transformer: given some past motion and a style code describing the speaker, it predicts the next short chunk of motion. To make this efficient, they first train a multi-scale vector-quantized codec that compresses motion into discrete tokens—similar to how image codecs or audio codecs work—so the transformer operates over tokens rather than continuous values.

The training happens in two distinct phases.

In Stage 1, they deliberately throw away the audio. The model is given only past motion and a style embedding, and must predict what comes next. Because there is no audio signal to lean on, the only way to reduce prediction error is to learn the statistics of natural facial motion itself: how often people blink, how heads drift and settle, how expressions shift on their own. This stage is trained on unpaired video from many sources—news, interviews, streams, casual conversation—so the model sees a wide variety of human behavior.

In Stage 2, they reintroduce audio on paired conversational clips where both speakers' voices are available. They add two cross-attention layers to each transformer block: one lets the model consult speaker A's speech, the other lets it consult speaker B's speech. When generating speaker A's face, the first layer handles speaking and the second handles listening; for speaker B, the roles swap. Crucially, they do not retrain the entire network. The new attention layers are trained from scratch, while the pretrained backbone is adapted with LoRA, a lightweight fine-tuning technique that adds small trainable matrices instead of overwriting the original weights. This preserves the internal motion prior learned in Stage 1 and lets audio merely modulate it, which is exactly the structure the authors argue natural listening follows.

Why This Matters

Impact on research. This paper reframes listening behavior as a prior-plus-modulation problem rather than a direct audio-to-motion mapping. That reframing is generalizable: any task where the driving signal is weakly correlated with the target behavior—nonverbal reactions, idle animation, spontaneous gesture—could benefit from a similar two-stage decomposition. It also provides a substantial new 3D supervision dataset for dyadic conversation, which the field has been missing.

Real-world applications:

• Interactive digital humans and virtual agents. Customer service avatars, tutors, and companions that react believably while the user speaks, not just when they speak themselves.
• Telepresence and remote meetings. Real-time avatar-driven video where participants' faces are animated from audio, useful for bandwidth-constrained or privacy-preserving communication.
• Game NPCs and virtual streamers. Characters that listen with visible engagement during player dialogue, improving immersion without hand-authored animation.
• Content creation and dubbing. Generating matched conversational footage for both sides of a dialogue from an audio track alone, useful for localization and synthetic media production.

Industry relevance. The 560.6 FPS throughput on consumer hardware and the reduced parameter count make UniLS deployable in production settings where latency matters. The end-to-end single-pass design removes the two-model pipeline that prior work required, cutting engineering complexity and inference cost. Companies building conversational AI, virtual humans, or live avatar pipelines have a direct use case.

Future Directions

• Add semantic understanding. The current model reacts to acoustic cues like rhythm and emphasis but has no idea what is being said. It cannot nod to agree or shake its head to disagree. Integrating speech recognition or language models to ground listener reactions in meaning is the most obvious next step.

• Move beyond fixed-length chunks. The autoregressive chunk formulation occasionally produces subtle discontinuities at chunk boundaries. Continuous or variable-length generation would improve long-horizon smoothness for extended conversations.

• Model longer conversational context. Listening reactions depend on what was said several turns earlier, not just the current utterance. Extending the context window or adding memory mechanisms could produce more coherent interaction.

• Generalize to more than two speakers and richer settings. The current setup is strictly dyadic. Group conversations, turn-taking among three or more participants, and blending with body or gesture generation are open problems.

Target Audience

Researchers and graduate students working on 3D facial animation, speech-driven motion synthesis, and conversational avatars will find the core technical contribution most relevant. Practitioners building digital human or virtual agent products will benefit from the real-time performance analysis and the practical two-stage training recipe. The motivation section, which diagnoses the audio-motion correlation imbalance, is also readable by anyone with a general machine learning background who wants to understand why listening is harder than speaking to generate.

Authors’ abstract

Generating lifelike conversational avatars requires modeling not just isolated speakers, but the dynamic, reciprocal interaction of speaking and listening. However, modeling the listener is exceptionally challenging: direct audio-driven training fails, producing stiff, static listening motions. This failure stems from a fundamental imbalance: the speaker's motion is strongly driven by speech audio, while the listener's motion primarily follows an internal motion prior and is only loosely guided by external speech. This challenge has led most methods to focus on speak-only generation. The only prior attempt at joint generation relies on extra speaker's motion to produce the listener. This design is not end-to-end, thereby hindering the real-time applicability. To address this limitation, we present UniLS, the first end-to-end framework for generating unified speak-listen expressions, driven by only dual-track audio. Our method introduces a novel two-stage training paradigm. Stage 1 first learns the internal motion prior by training an audio-free autoregressive generator, capturing the spontaneous dynamics of natural facial motion. Stage 2 then introduces the dual-track audio, fine-tuning the generator to modulate the learned motion prior based on external speech cues. Extensive evaluations show UniLS achieves state-of-the-art speaking accuracy. More importantly, it delivers up to 44.1\% improvement in listening metrics, generating significantly more diverse and natural listening expressions. This effectively mitigates the stiffness problem and provides a practical, high-fidelity audio-driven solution for interactive digital humans. Code and demos are available at https://xg-chu.site/project_unils/.