AUHead: Realistic Emotional Talking Head Generation via Action Units Control

AI summary

Overview

• Research area: Audio-driven emotional talking head generation (computer vision, generative diffusion models, audio-language models).
• Technical level: Advanced.
• Scope: The paper proposes AUHead, a two-stage pipeline that predicts facial Action Unit (AU) sequences from speech with a fine-tuned audio-language model and then uses those AU sequences to condition a diffusion model for emotionally expressive talking-head video synthesis.

What This Paper Is About

Most audio-driven talking head systems feed speech and a portrait directly into a generative model, which yields good lip synchronization but often emotionally flat or neutral facial expressions. This paper's goal is fine-grained, interpretable emotion control: it extracts facial Action Units from the audio as an intermediate representation and then uses those AUs to steer a diffusion model toward realistic, emotionally expressive facial animation while preserving identity and lip sync.

Key Contributions

1. A two-stage emotional talking-head framework that decouples emotion-oriented speech understanding from video synthesis, using AUs as the bridge between the two.
2. The first exploration (to the authors' knowledge) of generating facial AU sequences through large Audio-Language Models. This is achieved via spatial-temporal AU tokenization and an "emotion-then-AU" chain-of-thought mechanism that captures subtle emotional cues in speech.
3. An AU-driven controllable diffusion framework combining AU representation (1D sequences mapped to 2D facial representations), context-aware AU embedding, and AU-vision cross-attention interaction.
4. An AU disentanglement guidance strategy at inference time that separately modulates AU conditioning strength and other conditions (audio, motion priors) to trade off expression control, identity consistency, and visual quality.

Main Findings

• Chain-of-thought beats direct AU prediction: In Stage 1, the "Audio → Emotion → AU" configuration reached Recall 0.71, Precision 0.71, Accuracy 0.58, F1 0.69, MAE 0.2085, and 67.01% emotion classification accuracy. Comparable numbers for the alternatives were: Audio+Emotion → AU (0.74 Recall, 0.72 Precision, 0.61 Accuracy, 0.71 F1, MAE 0.1928); Audio → AU (0.63, 0.65, 0.50, 0.62, MAE 0.2447); Audio → AU → Emotion (0.66, 0.68, 0.53, 0.65, MAE 0.2200, 51.76% emotion accuracy).
• MAE is within human annotation variability: The reported MAE of 0.2 is described as comparable to the inter-annotator variability reported in FEAFA.
• Sparse tokenization reduces sequence length: Because statistically only 7/24 AUs are activated, the spatial-temporal tokenization cuts output sequence length by 80.95% on average, easing the ALM's context-window constraint (a 4-second video at 25 FPS would otherwise yield roughly 13K tokens).
• 2D AU representations outperform 1D sequences: On MEAD, MEMO + RoM reached PSNR 23.3585, SSIM 0.7399, FID 10.8701, and M/F-LMD 1.8602/2.1536, versus the retrained MEMO baseline at 23.1910, 0.7345, 11.1237, and 2.0684/2.2473. On CREMA, MEMO + RoM reached PSNR 24.3113, SSIM 0.7417, FID 8.3352, versus baseline 24.2808, 0.7410, 8.3881.
• A small sync trade-off with 2D AUs: Sync scores slightly decreased when using 2D AU inputs (e.g., MEAD baseline 6.9885 vs. MEMO + RoM 6.6095), which the authors attribute to increased focus on expression accuracy over temporal lip alignment.
• Best guidance scale is 3.5: Increasing the AU classifier-free guidance scale improves emotion accuracy and lowers MAE, while FID first decreases then increases; the best quality-emotion trade-off occurs at scale 3.5.
• Competitive state-of-the-art comparison: AUHead (MEMO) achieved MEAD Sync 6.6311, PSNR 23.3466, SSIM 0.7395, FID 10.9671, M/F-LMD 1.8608/2.1604, and CREMA Sync 6.2050, PSNR 24.2912, SSIM 0.7413, FID 8.2361, M/F-LMD 1.9313/2.3991. AUHead (HalloV1) achieved MEAD Sync 6.0201, PSNR 22.0132, SSIM 0.7113, FID 12.8421, M/F-LMD 2.3836/2.4595, and CREMA Sync 4.7100, PSNR 23.0818, SSIM 0.7201, FID 9.7086, M/F-LMD 2.2964/2.5337.
• Higher Sync confidence exists in some baselines: For example, Wav2lip scored MEAD Sync 8.7778 (PSNR 23.0296, SSIM 0.7395, FID 32.8043) and Sonic scored MEAD Sync 8.0988 and CREMA Sync 6.8620; AUHead's Sync confidence is lower than several of these.
• User study favors AUHead: Against HalloV2, AUHead received 64.63% preference for emotional expression, 63.63% for video quality, 71.00% for audio-lip sync, and 67.75% for overall performance. HalloV2 received 18.88%, 21.28%, 13.75%, and 16.13% respectively; the "same" option took 16.49%, 15.09%, 15.25%, and 16.12%. The study used 32 clips from the MEAD test set (4 identities, 8 emotions) rated by 25 participants in a blind setting.
• Generalization to unseen styles: Ten-second examples on unseen data covered line-art sketches, oil-painting portraits, and realistic faces, with temporally coherent expressions and identity preservation.
• Generation quality is limited by AU quality: The paper notes that generating valid dense AU vectors remains challenging due to the limited context window of Qwen-Audio-Chat and redundancy in dense AU representations.

Methodology in Plain English

The system works in two stages.

Stage 1 — reading emotion out of speech. The authors fine-tune an audio-language model (Qwen-Audio-Chat, referred to elsewhere in the paper as Audio-Qwen-Chat) using LoRA, with ground-truth AU sequences as supervision, training via next-token prediction with cross-entropy loss. Because AU vectors are dense and subtle, they tokenize them instead of regressing them: each frame's AU vector is compressed into a sparse set of index-intensity pairs, keeping only AUs above a sparsity threshold (the sparsity coefficient λ was set to 0), and the AU supervision is downsampled by a factor γ = 0.2. To make generation easier, they use a chain-of-thought-style "emotion-then-AU" scheme: the model first predicts an emotion category from the audio, then generates the AU sequence, using the emotion as high-level context. AUs are produced at 5 sequences per second with 24 dimensions each, then converted back to dense form.

Stage 2 — turning AUs into video. The generated AU sequence is linearly interpolated up to the target frame rate and mapped to a structured 2D facial representation — either keypoint-based landmarks (LMK) or a rendering of mesh (RoM). A context-aware embedding step concatenates a local temporal window of AU features (window size 5, n = 2) and encodes them with a lightweight temporal convolutional network to keep expressions smooth over time. These embeddings are injected into a pretrained diffusion model through an AU adapter made of cross-attention layers with zero-initialization, while all other components stay frozen. Only the adapter is trained, using the standard diffusion noise-prediction loss, with each condition (audio, image, AU embedding) randomly zeroed so unconditional modeling remains possible. At inference, a disentanglement guidance formulation with separate scales for AU conditioning (s^AU) and other conditions (s^H) lets the user dial expression strength against visual quality.

Experimental setup. Experiments used MEAD (10,000 clips, 8 emotion categories, identity-based splits) and CREMA (7,442 clips from 91 actors, 6 emotions, official subject-based splits), with all videos resampled to 25 FPS and 512×512, audio at 16 kHz, and mel-spectrograms computed with window and hop length of 640 samples. Stage 1 was trained on 4× NVIDIA A100 GPUs for about 24 GPU-hours at a learning rate of 1×10⁻⁴; Stage 2 used Hallo V1 and MEMO as base models, trained on 4× A100 GPUs for 12 GPU-hours at 5×10⁻⁶ (Hallo V1) and 1×10⁻⁵ (MEMO). Evaluation covered PSNR, SSIM, FID, SyncNet, Mouth Landmark Distance (M-LMD), Facial Landmark Distance (F-LMD), emotion classification accuracy, Precision, Recall, and MAE.

Why This Matters

Impact on research. The paper argues that AUs are a compact, interpretable, semantically meaningful middle ground between audio and pixels — richer than coarse emotion labels and more controllable than latent emotion codes. It is also a new use for audio-language models: not just transcribing or classifying speech, but inferring facial muscle dynamics. The authors frame interpretable intermediate spaces as a general strategy for more controllable and reliable cross-modal generation. Code is released at github.com/laura990501/AUHead_ICLR.

Real-world applications (as named in the paper):

• Virtual avatars and interactive agents
• Film production and film-level cinematic content
• Communication and education content creation
• Creative applications and accessible visual content, per the ethics statement

Industry relevance. Talking-head generation underpins digital humans, dubbing and localization, virtual assistants, and social media content pipelines. Interpretable AU control is attractive for production because it gives artists a dial for expression intensity rather than a single emotion label, and the two-stage design lets existing diffusion backbones (Hallo V1, MEMO) be upgraded without retraining from scratch. The authors also flag risks: deepfake misuse, bias in generated faces, privacy concerns, and the environmental cost of large-scale training.

Future Directions

• In-the-wild generalization: The authors explicitly aim to extend AUHead to scenarios with diverse head poses and backgrounds.
• Better dense AU generation: The paper identifies limited context window and AU redundancy as ongoing obstacles to generating valid dense AU vectors, motivating further work on tokenization and modeling.
• Resolving the lip-sync trade-off: Sync confidence drops slightly with AU conditioning, which the authors attribute to timing mismatches between predicted AUs and speech; closing this gap is an open problem.
• More thorough comparison scope: The paper compares only against methods using the same input setting, leaving open how AUHead would fare against systems built on larger multimodal or LLM-derived guidance.

Target Audience

Researchers and graduate students in computer vision, audio-visual generation, and affective computing; engineers building talking-head or digital-avatar systems who already understand diffusion models and lip-sync metrics; and practitioners in film, virtual avatar, and interactive-agent development who need controllable emotional expression rather than neutral lip-synced video. Readers without a background in diffusion models or facial Action Units will find the method sections dense, since the paper assumes familiarity with latent diffusion, LoRA fine-tuning, and AU conventions.

Authors’ abstract

Realistic talking-head video generation is critical for virtual avatars, film production, and interactive systems. Current methods struggle with nuanced emotional expressions due to the lack of fine-grained emotion control. To address this issue, we introduce a novel two-stage method (AUHead) to disentangle fine-grained emotion control, i.e. , Action Units (AUs), from audio and achieve controllable generation. In the first stage, we explore the AU generation abilities of large audio-language models (ALMs), by spatial-temporal AU tokenization and an "emotion-then-AU" chain-of-thought mechanism. It aims to disentangle AUs from raw speech, effectively capturing subtle emotional cues. In the second stage, we propose an AU-driven controllable diffusion model that synthesizes realistic talking-head videos conditioned on AU sequences. Specifically, we first map the AU sequences into the structured 2D facial representation to enhance spatial fidelity, and then model the AU-vision interaction within cross-attention modules. To achieve flexible AU-quality trade-off control, we introduce an AU disentanglement guidance strategy during inference, further refining the emotional expressiveness and identity consistency of the generated videos. Results on benchmark datasets demonstrate that our approach achieves competitive performance in emotional realism, accurate lip synchronization, and visual coherence, significantly surpassing existing techniques. Our implementation is available at https://github.com/laura990501/AUHead_ICLR