Talking Together: Synthesizing Co-Located 3D Conversations from Audio

AI summary

Overview

Research area: Computer vision — audio-driven 3D facial animation, specifically the generation of dyadic (two-person) conversational motion for co-located digital humans.

Technical level: Advanced. The paper assumes familiarity with denoising diffusion models, 3D morphable face models (3DMMs), cross-attention, FiLM conditioning, and latent-space generative training.

Scope in one sentence: The paper introduces a dual-stream diffusion system that converts a single mixed two-speaker audio track, plus an optional text prompt describing the scene, into the complete 3D facial performance of two participants sharing a 3D space — expression, head rotation, gaze, and metric head translation — along with the dataset pipeline built to train it.

What This Paper Is About

Most audio-driven talking-head systems generate one isolated person at a time, producing what the authors describe as a "video conference call" effect: two disconnected faces that never occupy a shared physical space. Real in-person conversation depends on non-verbal cues that only exist in a shared 3D space — where each person sits, how they orient toward each other, when they make eye contact, and how a listener reacts while the other speaks.

The goal of this work is to model that spatial and interactive relationship explicitly. Given one mixed audio stream, the system synthesizes both participants simultaneously with precise lip-sync, along with relative position, orientation, and mutual gaze, and it lets a user control the participants' relative layout by typing a description such as "an intimate conversation" or "arguing across a table."

Key Contributions

1. An automated data curation pipeline that builds a large-scale dyadic conversation dataset from in-the-wild videos (over 2 million interacting pairs), together with a high-fidelity single-speaker corpus used for robust lip-sync training.

2. A dual-stream diffusion architecture with a shared U-Net backbone, using inter-speaker cross-attention, speaker role embeddings, and FiLM conditioning to disentangle a single mixed audio track and model speaker–listener interaction.

3. A mixed-data, multi-stage training strategy: pre-training on massive real conversational data for interaction grounding, then fine-tuning on high-quality data (real single-speaker video plus an augmented conversation subset) for precise lip articulation.

4. Two mechanisms for spatial and gaze realism: an LLM-based few-shot text-to-3D translation interface for controlling relative layout, and an auxiliary eye gaze loss applied selectively to a curated subset of samples with large head movements.

Main Findings

• State-of-the-art quantitative performance against baselines. In Table 2, the full model reaches an FD of 10.43 and P-FD of 18.24, compared with DualTalk at 28.41 / 38.29, SelfTalk at 43.58 / 53.98, CodeTalker at 47.23 / 70.54, FaceFormer at 52.66 / 59.84, DIM at 55.09 / 45.20, and L2L at 38.92 / 66.13. The authors report the best performance across nearly all metrics. Retrieval-based baselines (Listen-R, Listen-A, Listen-M) occasionally produce more diverse rotations and translations, but at low fidelity.

• A speaker-only model is competitive only on speaker motion. The "Ours (Single)" variant scores lower than the full model on FD (19.58 vs. 10.43) and P-FD (29.03 vs. 18.24), and the paper notes it slightly outperforms the conversation model on speaker vertex prediction while performing worse on listener and interaction modeling.

• Every component contributes, per the ablation in Table 3. The full model records FD 21.71 / P-FD 22.56 / EXP 5.97 / TRAN 1.50 / ROT 2.48. Removing cross-attention degrades to FD 30.49 / P-FD 40.87; removing the speaker embedding gives FD 35.92 / P-FD 35.93; removing the gaze loss gives FD 37.46 / P-FD 42.90 and raises expression MSE to 7.33; removing the second training stage gives FD 60.12 / P-FD 64.44; training on single-person data only gives FD 50.45 / P-FD 50.05.

• Training data alone is insufficient. The paper reports that single-person-only training performs poorly because it provides no guidance for listener behavior, while conversation-only training degrades facial expression accuracy — confirming that in-the-wild conversation data offers high-quality interactions but low-quality expressions.

• Human raters strongly prefer the full model. In a forced-choice study with nineteen participants viewing 14 groups of clips, the full model was chosen as best on lip quality (79.3%), speaker movements (73.8%), listener movements (73.0%), interaction quality (71.4%), and eye gaze (68.3%). DualTalk scored 3.9 / 6.3 / 7.2 / 5.6 / 7.9 and SelfTalk scored 0.9 / 0.9 / 1.6 / 1.6 / 2.4.

• Overlapping speech is handled. Qualitative results show that when both participants speak concurrently, each stream maintains accurate lip articulation and independent head motion while preserving conversational context.

• Text-driven spatial control works qualitatively. Text prompts compose 3D layouts while the model maintains realistic head translation and gaze behavior, though the authors state that text-to-translation is not the focus of their evaluation and provide a user study on it only in the Appendix.

• Dataset scale. Per Table 1, the conversation and synthetic datasets each have 50,000+ hours and 10k+ identities, compared with DualTalk (50h, 1000+), L2L (72h, 6), RealTalk (8h), Lm_listener (7h, 4), ViCO (1.6h, 92), BIWI (1.44h, 14), and VOCASET (0.5h, 12). The conversation dataset is the only one marked as having spatial relationship and simultaneous talking; the synthetic dataset is marked as having simultaneous talking but not interaction, multi-round conversations, or spatial relationship.

Methodology in Plain English

The problem setup. The input is a single audio waveform containing both speakers. The output, for each participant, is a sequence of 3D face parameters: an expression vector, rotations for four skeletal joints (neck, head, left and right eyes), and a global head translation in approximate metric meters. Concatenated, each frame is 78 numbers, and the model predicts 250 frames (10 seconds at 25 fps) for both people at once.

A shared two-lane generator. Rather than training two separate networks, the system runs one U-Net over both participants in parallel. Inside the decoder, cross-attention layers let each participant's features attend to the other's, so stream A can react to what stream B is doing and vice versa. This is the mechanism the authors credit for turn-taking and mutual responsiveness.

Telling the streams apart. Because the audio is mixed, the model also takes two per-frame "speaking probability" masks. These are deliberately imperfect — the authors note the noise stabilizes training and improves robustness to overlapping speech. Two learnable embeddings, one for "speaking" and one for "listening," are blended per frame using that speaker probability, giving each participant a continuous representation of their conversational role. Audio features (Wav2Vec), role embeddings, and the masks are combined into one conditioning vector that is both concatenated with the noisy input and used to modulate intermediate features through FiLM.

Two-stage training. First, the model is pre-trained on the large in-the-wild conversation corpus to learn interaction and rotation. Second, it is fine-tuned on high-resolution single-speaker data and a super-resolved conversation subset, where the reconstruction loss is applied only to the 20 expression parameters corresponding to lip and jaw articulation for the speaking participant; all other losses are zeroed out. This is the fix for the poor lip-sync inherent in in-the-wild conversation video, where people face each other and occlude their own mouths.

Data pipelines. The dyadic conversation dataset is built by filtering out video-conference-style clips (by inspecting background color distribution across the left and right halves), aggressively dropping frames with occluded, tiny, or blurry faces, separating the two audio sources, running face-specific super-resolution, and fitting a 3D face model to every frame with a temporal smoothness term; absolute scale is recovered by assuming an average inter-ocular distance. The synthetic dubbing dataset takes high-quality frontal single-person videos, cuts random speech segments, and reassembles them into alternating-speaker dialogues, which yields perfect speaker-activity masks and clean lip motion for training on overlapping audio.

Spatial control. The model is trained to predict movement relative to a center-normalized starting position, conditioned on the ground-truth first-frame translations of both heads. At inference, that conditioning becomes an interface: a few-shot prompt with text-to-translation examples is given to a large language model, which outputs 3D coordinates for a new user prompt. The supplementary material describes using Gemini 2.5 Pro via API with a system instruction

Authors’ abstract

We tackle the challenging task of generating complete 3D facial animations for two interacting, co-located participants from a mixed audio stream. While existing methods often produce disembodied "talking heads" akin to a video conference call, our work is the first to explicitly model the dynamic 3D spatial relationship -- including relative position, orientation, and mutual gaze -- that is crucial for realistic in-person dialogues. Our system synthesizes the full performance of both individuals, including precise lip-sync, and uniquely allows their relative head poses to be controlled via textual descriptions. To achieve this, we propose a dual-stream architecture where each stream is responsible for one participant's output. We employ speaker's role embeddings and inter-speaker cross-attention mechanisms designed to disentangle the mixed audio and model the interaction. Furthermore, we introduce a novel eye gaze loss to promote natural, mutual eye contact. To power our data-hungry approach, we introduce a novel pipeline to curate a large-scale conversational dataset consisting of over 2 million dyadic pairs from in-the-wild videos. Our method generates fluid, controllable, and spatially aware dyadic animations suitable for immersive applications in VR and telepresence, significantly outperforming existing baselines in perceived realism and interaction coherence.