Now You See Me, Now You Don't: A Unified Framework for Expression Consistent Anonymization in Talking Head Videos

AI summary

Overview

Research area: Computer vision, specifically privacy-preserving face and talking-head video anonymization using diffusion-based generative models.

Technical level: Intermediate. The paper assumes familiarity with diffusion models, ControlNet conditioning, face embeddings, and standard face-recognition evaluation protocols, but the pipeline itself is described stage by stage.

Scope: The paper introduces AnonNET, a multi-stage pipeline that replaces facial identity in talking-head videos with a newly synthesized identity while preserving age, gender, race, pose, and expression, and evaluates it on CelebA-HQ, LFW, CelebV-HQ, VoxCeleb2, and HDTF.

What This Paper Is About

Facial video is increasingly recorded and shared, and regulations such as the GDPR and the European Union's AI Act (adopted in 2024) restrict how biometric identifiers like faces may be collected and processed. Common anonymization techniques such as pixelation, blurring, and masking degrade the video so much that downstream tasks like expression recognition, tracking, and action recognition become unreliable, and prior work has shown these techniques can be partially reversible.

The goal of this work is a single framework that removes identity information from talking-head videos while keeping the remaining high-level attributes (age, gender, race), the surrounding scene, the original audio, and the person's expressions and head motion intact.

Key Contributions

1. A novel multi-stage framework for video anonymization that synthesizes new facial identities while preserving age, gender, race, and expressions.
2. A new dataset of anonymized videos derived from the VoxCeleb, CelebV, and HDTF datasets, presented as a resource for future research such as deepfake detection.
3. A comprehensive evaluation of the pipeline against state-of-the-art models at the image level and, at the video level, with respect to re-identification, identity consistency, and expression-aware downstream utility.
4. A modular design in which each stage (scene detection, face detection, inpainting, motion transfer) can be independently improved or replaced.

Main Findings

• Identity obfuscation (images): On CelebA-HQ and LFW, AnonNET at 35 denoising steps reaches rank-1 re-identification (Re@1) values of 0.041 (VGGFace2) on CelebA-HQ and 0.042 on LFW. The paper reports that CIAGAN and DeepPrivacy2 achieve the lowest scores overall, that AnonNET provides a consistent drop in re-identification compared to CIAGAN and DeepPrivacy while retaining attribute fidelity, and that it performs favorably relative to recent diffusion-based anonymization baselines FALCO and CAMOUFLaGE.

• Perceptual quality and aesthetics: AnonNET outperforms all baselines on Q-Align quality and aesthetic scores on both image datasets. On CelebA-HQ it reaches the highest quality (4.164) and aesthetics (3.332) scores, exceeding both DeepPrivacy2 and the ground truth. On LFW it leads with 2.887 (quality) and 1.939 (aesthetics). CIAGAN yields lower perceptual scores than both AnonNET and DeepPrivacy2.

• Privacy–utility trade-off: The authors state that AnonNET does not achieve the absolute lowest Re@1 but provides superior image quality.

• Video-level results: Compared with ground-truth videos, AnonNET improves quality and aesthetics across all three video datasets while maintaining comparable identity suppression. On HDTF it raises quality from 4.045 to 4.153 and aesthetics from 2.938 to 3.021. On CelebV-HQ quality goes from 3.800 to 3.907 and aesthetics from 2.718 to 2.886. On VoxCeleb2 — described as low resolution and challenging — quality rises from 2.493 to 2.859 and aesthetics from 1.606 to 1.949, with identity preservation essentially unchanged (0.021 to 0.022).

• Pose and gaze preservation: On CelebA-HQ, AnonNET guided by OpenPose achieves the best pose preservation in the comparison, with a pose error of 0.014 at 20 steps and 0.015 at 35 steps, versus 0.140 for DeepPrivacy2, 0.088 for FALCO, and 0.048 for FAMS. Gaze errors are 0.187 (20 steps) and 0.172 (35 steps) versus 0.244, 0.258, and 0.161 for the respective baselines.

• Attribute preservation and failure rates: On CelebA-HQ, 29,997 of 30,000 images were successfully anonymized with 3 failures; on LFW, 12,912 of 13,233 succeeded with 321 failures. No face detection failures occurred on either dataset. Gender was preserved at 99.4 percent (CelebA-HQ) and 99.3 percent (LFW), race at 79.5 percent and 87.1 percent, expression at 74.7 percent and 52.9 percent, with age mean absolute deviations of (1.87, 4.23) and (2.69, 6.29). Expression accuracy is lower on LFW due to greater variability and resolution constraints.

• Computational cost: Anonymizing the 50,000-clip VoxCeleb2 subset takes approximately 160 hours with LIA and 185 hours with LivePortrait on a single A100 GPU.

• Limitations: Because attribute guidance depends on pretrained recognition networks, errors in gender, expression, or gaze estimation can propagate into the anonymized output (shown as gender mismatch, expression mismatch, and gaze inconsistency). Multi-person scenarios are deferred to future work, and scenes with no faces are left untouched.

Methodology in Plain English

The framework works as a fixed sequence of seven stages.

1. Scene detection and identity clustering. The input video is split into scenes using FFmpeg's scene-change filter based on frame-wise histogram differences, then refined by mean RGB differences to avoid over-segmentation. Faces are clustered across scenes using VGG-Face2 embeddings with cosine-distance thresholds, so the same person keeps the same anonymized identity throughout the video.

2. Face detection and frontal selection. RetinaFace locates the face region. For each scene, one representative frame with a frontal head pose is selected. Head pose is estimated with the face_alignment library in 2D landmark mode, extracting 68 landmarks per frame; six key points (nose tip, chin, eye corners, mouth corners) are matched to a predefined 3D face model and the perspective-n-point problem is solved with OpenCV's solvePnP. Frames where fewer than 80 percent of landmarks fall inside the image are discarded, and the frame with the smallest absolute pitch and yaw is chosen. Using one frame per scene reduces flickering and computation.

3. Attribute recognition. DeepFace estimates age, gender, race, and emotion on the selected face. These are treated as non-identifying, high-level cues.

4. Diffusion-based inpainting. The masked face is regenerated with Realistic Vision V5.0, a publicly available checkpoint based on Stable Diffusion v1.5. Three ControlNets — segmentation mask, lineart, and openpose — provide structural guidance so synthesis stays inside the face region and keeps geometry and expression. An attribute-conditioned prompt (the paper's example is "A photorealistic portrait of a middle-aged Asian female, with a neutral expression") sets the target appearance, and a negative prompt suppresses artifacts such as distortions, unrealistic textures, or cartoon-like features. The DPMSolver++ scheduler is used over typically 20 to 70 steps with a guidance scale between 8 and 20 depending on the dataset, and a VAE with perceptual reconstruction loss maps between pixel and latent space.

5. Anonymity verification. VGG-Face2 embeddings of the original and anonymized images are compared. If the cosine distance score falls below a threshold of 0.3, suggesting possible identity leakage, inpainting is re-run with greater stochasticity: a higher prompt guidance scale, reduced ControlNet conditioning strength, and 5 additional denoising steps.

6. Landmark-free motion transfer. The anonymized face is animated frame by frame using LIA and LivePortrait rather than explicit keypoint tracking. The original source frame is encoded into a latent motion representation capturing pose and expression, framewise motion offsets are predicted for each target frame, a learned flow map warps the anonymized source frame accordingly, and LivePortrait optionally refines eye and lip dynamics such as blinks and speech. Decoupling motion from identity in latent and flow-guided spaces avoids landmark detection errors and improves robustness to rapid motion and partial occlusion.

7. Video reassembly. Processed frames are merged back into the original scene structure using original timestamps; the audio is unaltered and resynchronized, and background context is retained.

Evaluation uses several metrics: Re@1 with VGGFace2 and CASIA-WebFace embeddings; Q-Align/One-Align for perceptual quality and aesthetics; pose angles from Dlib's face pose estimator with MTCNN landmarks and gaze direction from L2CS-Net measured as mean absolute error; expression labels predicted pre- and post-anonymization with DeepFace; and, for video, average cosine distance of DINO embeddings between consecutive frames to measure temporal identity consistency.

Why This Matters

Impact on research. The paper targets a gap between anonymization methods that destroy usable information and generative methods that produce convincing faces but lack control over attributes such as age, gender, race, and expression. It also contributes a dataset of anonymized videos derived from VoxCeleb, CelebV, and HDTF, which the authors propose as a resource for areas such as deepfake detection, and it reports results on video-level identity consistency and downstream expression utility rather than only image-level realism.

Real-world applications:

• Medical therapy sessions recorded for research, where patient identity must be protected but related expressions and emotions must survive for analysis.
• Journalism, where footage may need to be published without exposing subjects.
• Human-computer interaction, where video data may be processed while preserving non-identifying facial behavior.
• Downstream computer vision pipelines such as expression recognition, people tracking, action recognition, and video reenactment, which the paper argues are impeded by pixelation, blurring, and masking.

Industry relevance. The motivation is explicitly regulatory: the GDPR imposes strict constraints on the collection, processing, and dissemination of personal data including biometric identifiers, and the European Union's AI Act, adopted in 2024, introduced a tiered risk-based framework placing heightened scrutiny on AI systems handling biometric and identity-sensitive data. Methods that anonymize while preserving data utility are directly relevant to organizations that must comply with these rules without losing the analytical value of video. The paper notes that the code of AnonNET will be publicly released.

Future Directions

• Extending AnonNET to full-body video anonymization, rather than restricting modification to the face region.
• Incorporating audio-driven synchronization to improve lip consistency.
• Improving expression preservation in more dynamic conversational settings.
• Handling multi-person scenarios, which the authors explicitly defer because of the complexity of simultaneous multi-face motion transfer.
• Developing more robust or fine-tuned attribute predictors, since errors in gender, expression, or gaze estimation propagate to the anonymized output, particularly for edge cases and underrepresented demographics.

Target Audience

Researchers and graduate students working on face anonymization, privacy-preserving computer vision, diffusion-based generative models, and video synthesis will find the pipeline design and evaluation protocol most useful. Practitioners who need to de-identify recorded video while retaining analyzable facial attributes — for example in medical research, journalism, or human-computer interaction — and researchers building or evaluating deepfake detection systems, who may use the released anonymized video dataset, are also well served. The paper is intermediate in difficulty: readers should be comfortable with diffusion models, ControlNet conditioning, and standard face-recognition metrics to follow the evaluation in detail.

Authors’ abstract

Face video anonymization is aimed at privacy preservation while allowing for the analysis of videos in a number of computer vision downstream tasks such as expression recognition, people tracking, and action recognition. We propose here a novel unified framework referred to as Anon-NET, streamlined to de-identify facial videos, while preserving age, gender, race, pose, and expression of the original video. Specifically, we inpaint faces by a diffusion-based generative model guided by high-level attribute recognition and motion-aware expression transfer. We then animate deidentified faces by video-driven animation, which accepts the de-identified face and the original video as input. Extensive experiments on the datasets VoxCeleb2, CelebV-HQ, and HDTF, which include diverse facial dynamics, demonstrate the effectiveness of AnonNET in obfuscating identity while retaining visual realism and temporal consistency. The code of AnonNet will be publicly released.