THEval. Evaluation Framework for Talking Head Video Generation

AI summary

Overview

Research area: Computer vision, specifically evaluation methodology for talking head (TH) video generation — audio-driven and video-driven portrait animation.

Technical level: Intermediate. The paper combines facial-landmark geometry, off-the-shelf perceptual image quality models, and a human-preference study. The metric definitions are given in full equations, so readers comfortable with basic signal statistics (standard deviation, median absolute deviation, correlation) can follow along.

One-sentence scope: The paper proposes THEval, an 8-metric, 3-dimension evaluation framework for talking head videos, validated against a 3,519-rating user study on 85,000 videos generated by 17 state-of-the-art models over a newly curated 5,011-clip multilingual dataset.

What This Paper Is About

Talking head generation has improved rapidly, but the metrics used to judge it have not kept pace. Current evaluations lean on a narrow set of tools — FID, FVD, SSIM, PSNR for image quality, and Syncnet's LSE-C/LSE-D for lip synchronization — which the authors argue capture little about motion quality, temporal coherence, or what human viewers actually perceive. THEval's goal is to replace this thin metric set with a decomposed, interpretable evaluation that breaks performance into quality, naturalness, and synchronization, and that correlates strongly with human judgment.

Key Contributions

1. A new 8-metric evaluation framework (THEval) spanning three dimensions — (i) quality, (ii) naturalness, (iii) synchronization — reporting a Spearman correlation of ρ = 0.870 between its composite Final Score and human ratings.

2. A new evaluation dataset of over 5,000 videos (5,011 clips, from 31 public YouTube channels, six languages, 1080p, over 18 hours total, average clip length 13 seconds), designed to test generalization on content unseen during model training.

3. An extensive benchmark of 17 state-of-the-art audio- and video-driven models, producing 85,000 videos and a detailed per-metric analysis of each method's strengths and weaknesses.

4. A user study demonstrating alignment with human preferences and showing that existing metrics (SyncNet LSE-C/LSE-D, FID, FVD, LMD-F, LMD-L) correlate poorly with human ratings.

Main Findings

• THEval's Final Score tracks human preference closely. The composite score achieves a Spearman ρ = 0.870 (p < 0.0001, 95% CI [0.648, 0.967]) against human win rates across models, with bootstrap confidence intervals from n = 10,000 resamples.

• Existing metrics fail to align with human ratings. LSE-C yields ρ = -0.164 (p = 0.530), LSE-D ρ = -0.269 (p = 0.297), FVD ρ = 0.289 (p = 0.260), FID ρ = 0.210 (p = 0.416), LMD-F ρ = 0.231 (p = 0.389), and LMD-L ρ = 0.227 (p = 0.399). The negative SyncNet correlations occur because Wav2Lip, trained with SyncNet as a discriminator, optimizes that metric directly yet was among the least selected methods in the user study.

• The strongest individual metrics are region-specific and motion-based. Mouth Visual Quality reaches ρ = 0.765, Head Motion Dynamics ρ = 0.763, Quality (aggregate) ρ = 0.713, Naturalness ρ = 0.702, Face Quality ρ = 0.699, Synchronization ρ = 0.603, Global Aesthetics ρ = 0.544, Eyebrow Dynamics ρ = 0.527, Silent Lip Stability ρ = 0.484, Lip Dynamics ρ = 0.414, and Lip-Sync ρ = 0.404.

• Video-driven models outperform audio-driven ones overall. The top Final Scores are LivePortrait (0.9345), X-Portrait (0.8999), and LIA-X (0.8806); the best audio-driven entry is Hallo2 at 0.8477.

• Audio-driven methods struggle with expressiveness and head motion, despite strong lip-sync. Hallo2 scores 0.9883 on Lip Dynamics but only 0.2395 on Head Motion; OmniAvatar scores 0.4650 on Lip Dynamics and 0.6039 on Head Motion while reaching 0.9972 on Lip-Sync; Real3dPortrait drops to 0.0895 on Head Motion and Dimitra to 0.1279.

• Mouth synthesis and face synthesis are not equally difficult. The paper's separation of mouth-centric and face-centric quality is motivated by the observed difficulty gap in synthesizing realistic mouth motion.

• Over-articulation is a recurring failure mode. FLOAT and OmniAvatar exaggerate mouth movements. OmniAvatar's behavior is attributed to the WanVideo image-to-video model it uses, and it also shows temporal drift in longer videos — identity divergence from the source, growing visual artifacts, and an emerging orange color cast.

• Video-driven models trade expressivity against artifact-free rendering. Earlier video-driven methods show tearing, blur, and instability in facial regions under large head movements, producing lower quality scores, while retaining stronger naturalness.

• SyncNet scores are unstable to encoding choices unrelated to lip-sync. Switching audio encoding from mp4a to mpga changes LSE-D and LSE-C; across the entire HDTF dataset the average absolute difference is 0.4, reaching up to 1.2 for some samples. Similar behavior appears when comparing H.264 and H.265 video encodings, with no noticeable human-visible difference.

• The composite score is deliberately unweighted. The Final Score normalizes each metric to a ground-truth reference via s = 1 − |Model_Score − GT_Score| / GT_Score and averages the eight normalized metrics equally, prioritizing transparency over maximizing correlation on the test set.

• Ratings were consistent across participants. The user study collected 3,519 ratings distributed equally among the 17 models, with each rater seeing 153 videos total; Krippendorff's α = 0.74 indicates substantial agreement.

Methodology in Plain English

The authors start from a simple observation: a talking head video can be bad in many different ways, but existing metrics collapse all of them into one or two numbers. Their response is to decompose the problem.

Building the test set. They collect 5,011 clips from 31 YouTube channels, filtering for a single speaker, a clear and primarily frontal face, and 1080p resolution. The mix spans Spanish, Italian, English, French, Japanese, and Chinese — English dominates with 1,680 clips, followed by French (890), Chinese (733), Spanish (719), Japanese (645), and Italian (344). Clip lengths range from 2 seconds to 334.6 seconds, with most clips short. The intent is that these videos were not seen during training, so scores reflect generalization rather than memorization.

Defining the metrics. Quality is measured by running off-the-shelf perceptual models — TOPIQ's Image Aesthetic Assessment for overall aesthetics, TOPIQ's image quality assessment on the face region, and MUSIQ on the mouth region — averaged across frames. Naturalness is measured kinematically using facial landmarks: Lip Dynamics takes 40 lip landmarks, computes pairwise Euclidean distances within frames, and averages their standard deviation over time; Head Motion Dynamics combines the standard deviation of pitch, yaw, and roll with the variance of their frame-to-frame differences and the variance of face-center translation; Eyebrow Dynamics takes the vertical eyebrow-to-eye distance normalized by inter-ocular distance and measures its standard deviation. Synchronization is measured against the audio: Silent Lip Stability uses a voice activity detector to find silent segments of at least 300 ms and applies a median absolute deviation to normalized mouth opening in those frames, while Lip-Sync normalizes mouth openness and RMS audio energy to [0, 1] over speech frames and takes the mean absolute difference between the two signals.

Tooling. Landmarks come from MediaPipe Face Mesh, head pose from FaceXFormer, and speech segment detection from Silero VAD.

Running the benchmark. Seventeen models — nine video-driven (Controltalk, LivePortrait, LIA, X-Portrait, LIA-X, EmoPortrait, MCNet, DaGan, FOM) and eight audio-driven (Hallo2, EchoMimic, Wav2Lip, SadTalker, Dimitra, OmniAvatar, Real3dPortrait, FLOAT) — are run with their default hyperparameters and official weights on the same audio and reference frames, producing 85,000 videos. Ground-truth videos serve as the reference point so that each metric's deviation from real facial behavior is measurable in absolute terms.

Validating against humans. A user study hosted on Hugging Face Space presents paired videos with the instruction "Please watch both videos and select which one looks more realistic." Pairs include real-vs-generated and all combinations of the 17 methods; video selection is random and left/right placement is randomized to reduce positional bias. The resulting preference scores are then correlated (Spearman) against each metric.

Why This Matters

Evaluation is the bottleneck that decides which generative methods get adopted. If a metric can be gamed — as the authors show with Wav2Lip and SyncNet — then progress measured by that metric is partly illusory. THEval's value is that it separates concerns, so a reader can see why a model scores the way it does: is it failing at lip-sync, at head motion, at eyebrow expressiveness, or at raw image fidelity? That kind of diagnostic granularity is what lets researchers target fixes rather than optimize blindly.

The framework's ρ = 0.870 correlation with human judgment is also practically significant because user studies are slow and expensive. A validated automatic proxy lets labs iterate without running a new perceptual study for every model revision.

Real-world applications (implied by the domain rather than enumerated in the paper):

• Film and television dubbing and localization — replacing an actor's mouth region across languages while preserving head motion and expression requires exactly the quality/naturalness/synchronization split THEval provides.
• Virtual avatars and telepresence — avatar systems need believable eyebrow and head dynamics, not just correct lip-sync; THEval's naturalness dimension surfaces failures that lip-sync metrics miss.
• Digital humans in customer service, education, and gaming — long-form content exposes temporal drift and artifact accumulation, which the OmniAvatar analysis demonstrates and which short evaluation clips would hide.
• Synthetic media forensics and provenance — a benchmark that identifies characteristic artifact patterns (tearing under large head motion, color cast over time) gives detectors concrete failure signatures to look for.

Industry relevance: companies building portrait animation products need a "good enough" threshold to ship against. THEval's Final Score (with LivePortrait at 0.9345 as the current top result) gives them a ranked target, and its public dataset, code, and live leaderboard give the field a shared comparison point.

Future Directions

• Extending to harder scenarios. The authors state future work will extend the benchmark to more diverse settings, including multiple humans and side views. The current dataset is restricted to single-speaker, frontal-face, 1080p clips.

• Handling long-form temporal degradation. The observed failure of OmniAvatar over longer videos — identity divergence, accumulating artifacts, and color cast appearing over time — is not fully captured by metrics averaged across frames. A drift-aware metric is an open problem.

• Reducing dependence on landmark and pose estimators. Metrics (4) through (8) all depend on facial landmark extraction (MediaPipe Face Mesh) and head pose estimation (FaceXFormer). How sensitive the framework is to those upstream estimators is not reported.

• Diagnosing the expressiveness ceiling in audio-driven methods. The paper identifies head motion and eyebrow expressiveness as persistent weaknesses for audio-driven approaches (head motion scores as low as 0.0895 and 0.1279 for Real3dPortrait and Dimitra), but the underlying cause and whether it is a data or architecture limitation is an open question.

Target Audience

This paper is most useful to researchers building or benchmarking talking head generation models, who need a replacement for the FID/FVD/SyncNet stack in their own evaluations. It also serves applied engineers and product teams deciding whether a generation method is production-ready, and benchmark and evaluation researchers interested in how to decompose a perceptual quality target into measurable components that correlate with human judgment. Readers focused on 3D Gaussian Splatting or NeRF-based head avatars will find THEval out of scope, since the authors explicitly exclude those methods for requiring multi-view inputs and fixed pretrained identities.

Authors’ abstract

Video generation has achieved remarkable progress, with generated videos increasingly resembling real ones. However, the rapid advance in generation has outpaced the development of adequate evaluation metrics. Currently, the assessment of talking head generation primarily relies on limited metrics, evaluating general video quality, lip synchronization, and on conducting user studies. Motivated by this, we propose a new evaluation framework comprising 8 metrics related to three dimensions (i) quality, (ii) naturalness, and (iii) synchronization. In selecting the metrics, we place emphasis on efficiency, as well as alignment with human preferences. Based on this considerations, we streamline to analyze fine-grained dynamics of head, mouth, and eyebrows, as well as face quality. Our extensive experiments on 85,000 videos generated by 17 state-of-the-art models suggest that while many algorithms excel in lip synchronization, they face challenges with generating expressiveness and artifact-free details. These videos were generated based on a novel real dataset, that we have curated, in order to mitigate bias of training data. Our proposed benchmark framework is aimed at evaluating the improvement of generative methods. Original code, dataset and leaderboards will be publicly released and regularly updated with new methods, in order to reflect progress in the field.