Avatar Forcing: Real-Time Interactive Head Avatar Generation for Natural Conversation

AI summary

Authors: Taekyung Ki, Sangwon Jang, Jaehyeong Jo, Jaehong Yoon, Sung Ju Hwang (KAIST, NTU Singapore, DeepAuto.ai) arXiv: 2601.00664v2 [cs.LG], 30 May 2026 | Category: Machine Learning | License: CC BY 4.0

Overview

Research area: Generative modeling of talking-head avatars — specifically dyadic, interactive head avatar generation conditioned on multimodal user signals (audio, motion) plus avatar audio.

Technical level: Intermediate. The paper builds on flow matching, diffusion forcing, transformer attention masking, motion latent auto-encoders, and Direct Preference Optimization. Readers unfamiliar with diffusion/flow-matching notation will need to consult the Background sections (Section 3 and Appendix A), which the paper supplies.

Scope: A single framework, Avatar Forcing, that combines causal diffusion forcing in a learned motion latent space with a label-free preference optimization method to produce real-time, reactive, and expressive interactive head avatar video.

What This Paper Is About

Existing talking-head systems generate one-way, audio-synchronized speech animation rather than the back-and-forth of a real conversation, where listening behaviors (nodding, smiling, focusing) matter as much as speaking. Two obstacles block truly interactive avatars: generating motion under causal, real-time constraints, and learning expressive reactions without labeled data defining what a "good" reaction is. Avatar Forcing addresses both by generating avatar motion latents causally from live user audio, user motion, and avatar audio, then fine-tuning for expressiveness using preference pairs built by dropping user signals — no human annotation required.

Key Contributions

1. A causal diffusion-forcing framework for interactive head avatars. Avatar Forcing models the user–avatar interaction autoregressively in a learned motion latent space, processing multimodal user inputs (user audio, user motion) and avatar audio continuously, using key-value (KV) caching so past information is reused rather than recomputed. This removes the need for the full future conversation context that bidirectional approaches such as INFP require.

2. A blockwise causal attention mask with look-ahead. The authors introduce a mask allowing each block of frames to attend to a limited number of future frames (look-ahead size l) while preserving overall causality, which mitigates per-frame motion jittering observed with a strict blockwise causal mask.

3. A label-free preference optimization method (DPO) for expressiveness. Preference pairs are synthesized automatically: the preferred sample is the motion latent from the ground-truth video, and the less-preferred sample is generated by a separately trained talking avatar model conditioned solely on avatar audio. This concentrates the learning signal on active listening and reactive motion while leaving lip sync and speech-driven motion unchanged.

4. Demonstration of real-time interaction. The system runs at roughly 0.5s latency versus 3.4s for the reproduced INFP baseline, with higher reactiveness and motion richness scores.

Main Findings

• Latency: Avatar Forcing achieves approximately 500ms latency (0.5s in Table 1), described in the abstract as a 6.8× speedup compared to the baseline. The reproduced INFP* baseline reports 3.4s, and the non-interactive FLOAT baseline reports 2.4s. The paper notes INFP must wait for over 3 seconds of full conversation context, including future frames.

• Reactiveness (RealTalk dataset, Table 1): Avatar Forcing reaches rPCC-Exp 0.003 and rPCC-Pose 0.036, versus INFP* at 0.035 and 0.064, and FLOAT at 0.054 and 0.182. Ground truth is 0.000 on both. Lower rPCC is better in this table.

• Motion richness (RealTalk): Avatar Forcing achieves SID 2.442 and Var 1.734, above INFP* (2.343, 1.638) but below the ground-truth SID of 3.972.

• Visual quality and lip sync (RealTalk): Avatar Forcing records FID 24.328 and FVD 170.874 versus INFP* at 24.551 and 159.000; CSIM 0.833 versus 0.867; LSE-D 8.060 versus 8.027; LSE-C 6.723 versus 6.536. The paper describes visual quality and lip synchronization as "comparable to INFP*."

• Human preference study: 42 recruited participants evaluated 12 video sets on five perceptual metrics (Reactiveness, Motion Richness, Verbal Alignment, Non-verbal Alignment, Overall Preference). Avatar Forcing is "strongly preferred across all metrics," achieving over 80% preference in overall quality. The abstract states it is preferred over 80% against the strongest baseline.

• Talking-head capability (HDTF, Table 2, 50 randomly selected videos): Avatar Forcing achieves the best image and video quality among the compared models — FID 20.332 and FVD 149.798, ahead of FLOAT (25.110, 167.463), INFP* (27.155, 187.977), Hallo3 (32.794, 184.341), and SadTalker (64.744, 342.996). CSIM is 0.870; LSE-D 7.700; LSE-C 7.560. FLOAT records the best CSIM at 0.881 and the best LSE-C at 8.006; Hallo3 records the best LSE-D at 8.498.

• Listening-head capability (ViCo, Table 3): Avatar Forcing attains the best FD on expression at 16.64 (INFP* 17.52, DIM 23.88, L2L 33.93, RLHG 39.02) and the best Var on expression at 2.80. It reports rPCC-Exp 0.01 and rPCC-Pose 0.01. SID-Exp is 3.12 (below RLHG's 3.62 and DIM's 3.71) and SID-Pose is 3.00.

• Ablation — user motion (Table 4): Removing user motion input changes rPCC-Exp from 0.003 to 0.052 and rPCC-Pose from 0.036 to 0.175, with SID dropping from 2.442 to 2.165 and Var from 1.734 to 1.586. Qualitatively, without user motion the avatar stays static during user silence even amid strong non-verbal cues such as smiling.

• Ablation — preference optimization (Table 4): Without DPO, SID is 2.236 and Var is 1.408 versus 2.442 and 1.734 with DPO. The paper reports that models without DPO show reduced diversity in facial expressions and head movement and fail to respond to the user's smile.

• Motivation from data statistics: The authors' variance visualization of the L2-norm of 3DMM expressions on the ViCo dataset shows listening data has lower motion expressiveness than speaking data, which they identify as a cause of passive, non-expressive learned listening behavior.

Methodology in Plain English

The system works entirely in a compressed motion latent space rather than on raw pixels. A motion latent auto-encoder (retrained from Ki et al.) splits an image into an identity component (appearance) and a motion component (facial expression, head movement). Generating video then means generating a sequence of these motion latents, which is far cheaper than generating frames directly.

For prediction, the model conditions each next motion latent on its past latents and a condition triplet: user audio, user motion, and avatar audio. A Dual Motion Encoder fuses these — first aligning user motion and user audio via cross-attention to capture holistic user behavior, then integrating avatar audio via a second cross-attention layer to learn the user–avatar causal relationship. A causal diffusion forcing transformer (DFoT) then predicts the next block of latent frames. Frames are grouped into blocks, each block shares a single noise timestep, and attention is masked so a block cannot see future blocks — except for a small look-ahead window of l frames, which smooths transitions without breaking causality. Because generation is blockwise and causal, the model can cache keys and values from previous steps and generate stepwise in a rolling-window fashion, which is what makes real-time latency possible.

Training combines two losses: the standard diffusion forcing objective (regressing the target vector field) and a DPO term. The DPO preference pairs are built without any human labeling — the "winner" is the ground-truth motion latent from the actual video, and the "loser" is a motion latent produced by a talking avatar model that only saw avatar audio, and therefore lacks the reactive, interactive component. Pushing the model away from that audio-only behavior sharpens expressive, reactive motion.

Implementation details: Adam optimizer, learning rate 10⁻⁴, batch size 8, latent dimension d = 512, 8 attention heads with hidden dimension 1024, 1D RoPE, 8 DFoT transformer blocks plus a transformer head, and AdaLN scale-shift modulation. Training uses N = 50 frames over B = 5 blocks (10 frames per block) with l = 2 look-ahead frames. Audio encoding uses 12 multi-scale features from Wav2Vec2.0. Sampling uses 10 NFEs with the Euler solver and classifier-free guidance. All experiments ran on a single NVIDIA H100 GPU. Data preprocessing used PySceneDetect, Face-Alignment, and a visual-grounded speech separation model, with faces cropped and resized to 512×512, video at 25 fps, and audio resampled to 16 kHz.

Why This Matters

Impact on research. The paper reframes talking-head generation as a bidirectional dyadic problem rather than a one-way audio-to-video mapping, and shows that diffusion forcing — previously applied to interactive video generation — transfers to the motion latent space for avatars. Its key methodological claim, that preference pairs can be synthesized simply by withholding user conditioning, offers a general recipe for improving behaviors that are hard to annotate because they are one-to-many by nature.

Real-world applications:

• Virtual presenters, hosts, and educators that can substitute for human presence in content delivery, as the paper's introduction envisions.
• Interactive character chat, letting users converse with customized avatars of their favorite characters.
• Dyadic video communication, where a remote participant is represented by a reactive avatar that mirrors expressions such as smiling when the user smiles.
• Content creation and visual communication tools needing expressive, low-latency animated portraits.

Industry relevance. The 0.5s latency figure places the system in the range where live conversational deployment becomes plausible; INFP*'s 3.4s latency is explicitly characterized as unsuitable for real-time applications. Because the model runs on a single NVIDIA H100 and operates in a compact 512-dimensional latent space with 10 NFEs, the compute profile is relevant to real-time serving economics. The authors report funding from IITP, NRF, CARAI (DAPA and ADD), and the InnoCORE program.

Future Directions

The paper defers extended discussion of limitations, ethical considerations, and future work to Appendix E, which is not included in the provided content, so those specifics are not reported here. The following are natural extensions raised by the work itself:

1. Closing the gap to ground-truth expressiveness. SID for Avatar Forcing is 2.442 against a ground-truth value of 3.972, and Var is 1.734 against 1.658 for ground truth — the remaining distance to human-level rhythmic and expressive motion is an explicit open target.

2. Improving head pose reactivity. On RealTalk, rPCC-Pose is 0.036 versus 0.064 for INFP*, indicating the pose channel reacts less to user pose than the baseline, even as expression reactivity (0.003) is far better.

3. Lifting listening diversity. On ViCo, SID-Exp of 3.12 and SID-Pose of 3.00 fall below RLHG (3.62, 3.17) and DIM (3.71, 2.35), so the richness of listening-side motion remains partially unsolved.

4. Generalizing the synthetic preference-pair construction. The DPO recipe relies on a separately trained audio-only talking avatar model as the source of losing samples; whether other forms of degraded conditioning produce equally effective or better preference signals is an open question.

Target Audience

This paper is most useful to researchers and engineers in generative avatar and talking-head synthesis who already understand diffusion or flow-matching models and want a concrete design for causal, low-latency interactive motion generation. It also suits preference-optimization researchers interested in label-free construction of preference pairs, and practitioners building real-time conversational agents or virtual presenters who need to weigh latency, visual quality, and lip-sync trade-offs against reactive expressiveness. Readers looking for a beginner-level introduction to diffusion forcing should treat the paper's Section 3 and Appendix A as prerequisites rather than starting points.

Authors’ abstract

Talking head generation creates lifelike avatars from static portraits for virtual communication and content creation. However, current models do not yet convey the feeling of truly interactive communication, often generating one-way responses that lack emotional engagement. We identify two key challenges toward truly interactive avatars: generating motion in real-time under causal constraints and learning expressive, vibrant reactions without additional labeled data. To address these challenges, we propose Avatar Forcing, a new framework for interactive head avatar generation that models real-time user-avatar interactions through diffusion forcing. This design allows the avatar to process real-time multimodal inputs, including the user's audio and motion, with low latency for instant reactions to both verbal and non-verbal cues such as speech, nods, and laughter. Furthermore, we introduce a direct preference optimization method that leverages synthetic losing samples constructed by dropping user conditions, enabling label-free learning of expressive interaction. Experimental results demonstrate that our framework enables real-time interaction with low latency (approximately 500ms), achieving 6.8X speedup compared to the baseline, and produces reactive and expressive avatar motion, which is preferred over 80% against the baseline.