APPLE: Attribute-Preserving Pseudo-Labeling for Diffusion-Based Face Swapping

AI summary

Overview

Research area: Computer vision, specifically generative face swapping — transferring a source person's identity onto a target image while keeping the target's pose, expression, lighting, skin tone, and makeup intact.

Technical level: Advanced. The paper assumes familiarity with diffusion models, rectified flow, diffusion inversion, classifier-free guidance, LoRA adapters, and standard face-swapping evaluation metrics (FID, ArcFace identity similarity, pose and expression error).

Scope in one sentence: The paper proposes APPLE, a diffusion teacher–student framework that generates attribute-preserving pseudo-labels so that a student face-swapping model can be conditioned on clean images instead of masked ones.

What This Paper Is About

Real ground-truth data for face swapping does not exist, so models are trained with proxy objectives. The dominant proxy is conditional inpainting: mask the target's face and reconstruct it using the source identity. Masking blocks identity leakage, but it also destroys attribute cues such as lighting, skin tone, makeup, and accessories, so the model produces plausible yet misaligned attributes. APPLE instead trains a teacher to produce high-quality pseudo-labels that stay aligned with the target's attributes, then trains a student model on those clean pseudo-labels under a direct image-editing objective.

Key Contributions

1. A diffusion-based teacher–student framework (APPLE) in which improved pseudo-label quality, rather than a new architecture, is the mechanism for better attribute preservation.
2. A conditional deblurring objective that replaces conditional inpainting during teacher training. Instead of masking the face, the target face region is replaced with a blurred version of the same image (downsampled to 8×8 and upsampled back), removing high-frequency identity detail while retaining low-frequency attribute cues such as pose, lighting, and expression.
3. An attribute-aware inversion scheme used at inference time to build pseudo-labels. The target is inverted under attribute-only conditioning, producing noise that carries target attributes but no target identity bias, which helps recover fine-grained details such as makeup and accessories that blurring removes.
4. A student model conditioned on raw, clean pseudo-label images, trained to reconstruct the original target. This removes the need for auxiliary networks or complex attribute-conditioning preprocessing at inference.

Main Findings

• Deblurring beats inpainting for the teacher. In the teacher ablation, the inpainting baseline scores FID 11.00, ID Sim. 0.54, ID Retrieval 92.80/96.90, Pose 3.37, Expr. 1.01. Switching to deblurring gives FID 4.20, ID Sim. 0.53, ID Retrieval 89.50/96.20, Pose 2.58, Expr. 0.79 — large gains in pose and expression at a small cost in identity retrieval.

• Attribute-aware inversion adds a further gain. Adding inversion to deblurring yields FID 3.68, ID Sim. 0.54, ID Retrieval 90.40/96.70, Pose 2.07, Expr. 0.70, improving FID, pose, and expression while restoring identity similarity to 0.54.

• Attribute-only conditioning is the best inversion configuration. Among four configurations, attribute-only gives FID 3.68, ID Sim. 0.54, ID Retrieval 90.40/96.70, Pose 2.07, Expr. 0.70. No conditioning gives FID 6.20, ID Sim. 0.52, Pose 2.03, Expr. 0.74; identity-only gives FID 10.02, ID Sim. 0.53, Pose 2.57, Expr. 0.83; full conditioning gives FID 10.51, ID Sim. 0.53, Pose 3.13, Expr. 0.99. The paper attributes the artifacts from full conditioning to residual identity information in the inverted noise.

• The APPLE teacher produces better pseudo-labels than FaceDancer. Two students, each trained on 50K pseudo-triplets from VGGFace2-HQ and evaluated at 5K training steps, compare as follows. FaceDancer teacher: FID 2.47, ID Sim. 0.53, ID Retrieval 87.50/95.60, Pose 2.07, Expr. 0.65. APPLE teacher: FID 1.98, ID Sim. 0.53, ID Retrieval 88.60/95.80, Pose 1.77, Expr. 0.62. The FaceDancer-based student has slightly higher identity similarity, but the APPLE-teacher student wins on pose, expression, and fine-grained attribute metrics.

• The student outperforms its teacher. On FFHQ, APPLE (Teacher) scores FID 3.68, ID Sim. 0.54, ID Retrieval 90.40/96.70, Pose 2.07, Expr. 0.70, while APPLE (Student) scores FID 2.18, ID Sim. 0.54, ID Retrieval 90.50/97.00, Pose 1.85, Expr. 0.64.

• State-of-the-art attribute preservation on FFHQ against eleven baselines. APPLE (Student) reaches the lowest FID (2.18) and lowest pose (1.85) and expression (0.64) error among the compared methods, at ID Sim. 0.54 and ID Retrieval 90.50/97.00. Baselines include SimSwap (FID 18.54, ID Sim. 0.55, 94.10/99.00, Pose 3.11, Expr. 1.73), CSCS (11.00, 0.65, 99.00/99.50, 3.64, 1.44), MegaFS (12.83, 0.49, 79.6/86.3, 4.40, 1.11), FaceDancer (3.80, 0.51, 89.70/96.50, 2.23, 0.74), HiFiFace (11.81, 0.50, 85.40/93.40, 3.20, 1.34), InfoSwap (5.00, 0.54, 91.40/97.50, 4.33, 1.40), E4S (12.13, 0.49, 78.30/87.80, 4.39, 1.29), DiffSwap (6.84, 0.34, 41.92/63.09, 2.63, 1.20), DiffFace (8.59, 0.54, 90.70/95.90, 3.67, 1.24), FaceAdapter (13.03, 0.52, 87.00/93.20, 5.12, 1.38), and REFace (7.22, 0.60, 97.60/99.40, 3.67, 1.08).

• Identity and attribute fidelity trade off. The paper states that CSCS and REFace report unusually high identity similarity (0.65 and 0.60) at the cost of attribute preservation, producing copy-paste-like artifacts, whereas APPLE reports a more balanced trade-off.

• Guidance scale gives an inference-time knob. On FFHQ, APPLE (Student) at w=1.0 scores FID 2.19, ID Sim. 0.54, ID Retrieval 90.40/96.50, Pose 1.84, Expr. 0.63; at w=1.5, FID 3.45, ID Sim. 0.58, ID Retrieval 94.60/98.30, Pose 2.15, Expr. 0.72; at w=2.0, FID 4.74, ID Sim. 0.60, ID Retrieval 96.00/98.40, Pose 2.38, Expr. 0.79. Main-paper results use w=1.0 without hyperparameter tuning, while other diffusion baselines are evaluated with their respective tuned w.

• Cross-dataset results on CelebA-HQ. APPLE (Student) at w=1.0 scores FID 3.24, ID Sim. 0.58, ID Retrieval 95.80/98.60, Pose 1.68, Expr. 0.56; at w=1.5, FID 4.99, ID Sim. 0.62, ID Retrieval 97.00/99.30, Pose 1.96, Expr. 0.64; at w=2.0, FID 6.55, ID Sim. 0.63, ID Retrieval 97.30/99.50, Pose 2.17, Expr. 0.70. Baselines on CelebA-HQ include SimSwap (9.99, 0.61, 97.70/99.00, 2.18, 1.23), CSCS (10.97, 0.67, 97.70/99.50, 3.36, 1.31), FaceDancer (5.47, 0.53, 92.60/98.50, 2.18, 0.67), InfoSwap (5.44, 0.55, 93.10/97.50, 3.85, 1.37), E4S (14.41, 0.52, 83.50/92.30, 4.15, 1.14), DiffSwap (20.08, 0.23, 18.94/35.07, 7.54, 2.08), DiffFace (18.75, 0.31, 46.30/47.30, 10.84, 1.88), FaceAdapter (11.50, 0.51, 85.10/90.80, 4.88, 1.30), and REFace (8.04, 0.69, 99.10/99.90, 3.33, 1.00). The paper notes some baselines are trained on CelebA-HQ while APPLE is trained solely on VGGFace2-HQ.

• FaceForensics++ results are partially reported in the provided content. Baselines shown are SimSwap (6.16, 0.67, 97.89/99.00, 1.94, 0.78), CSCS (13.58, 0.72, 94.70/97.50, 2.86, 1.23), FaceDancer (5.48, 0.57, 90.77/98.40, 2.16, 0.72), InfoSwap (12.62, 0.62, 97.20/98.90, 3.74, 1.72), E4S (19.29, 0.63, 94.30/97.50, 3.48, 1.34), and DiffSwap (9.53, 0.36, 10.10/57.10). The APPLE rows for this table are not present in the truncated content.

• Training configuration. FLUX.1-Krea [dev] is the backbone, with PulID as the identity encoder and OminiControl as the attribute conditioning branch at LoRA rank 64. Training data is VGGFace2-HQ filtered by AES with a threshold of 5.1. The teacher trains for 15K iterations without the identity loss and 50K more with it; the student trains for 15K iterations resuming from the teacher. Identity losses apply to the late 35% of diffusion timesteps for the teacher and the late 50% for the student. AdamW with learning rate 1e-4 and weight decay 1e-3 is used, with 4 NVIDIA A6000 GPUs, batch size 1 per GPU, gradient accumulation 4, effective batch size 16, 28 inference/inversion steps, and 512×512 resolution.

Methodology in Plain English

The team starts from the observation that masking a face before inpainting throws away exactly the information face swapping is supposed to preserve. Their fix is to make the training signal easier first, then transfer it.

Step 1 — Teach the teacher to deblur, not to inpaint. During training, the target's face region is replaced with a heavily blurred version of itself rather than a blank mask. Blurring a face at 8×8 resolution and scaling it back up removes identity-specific high-frequency detail but keeps color, tone, and lighting. The teacher is also given extra structural hints: 3DMM landmarks, eye landmarks from a gaze estimator overlaid onto the condition, and glass segmentation masks from face parsing.

Step 2 — Improve the pseudo-labels at inference. Even a deblurring teacher has to guess fine details like makeup that blurring erased. The authors take advantage of a known property of diffusion inversion: inverted noise is not truly Gaussian and retains residual semantic content. Instead of suppressing that residue, they exploit it. They invert the target image using only the attribute condition (no identity condition), which produces noise carrying target attributes but no target identity bias. The paper compares PCA visualizations of noise from four conditioning setups and finds that configurations with attribute conditioning show clear facial semantics, while non-conditioned and identity-only noise looks like random noise. In swapped outputs, attribute-only conditioning preserves makeup without the artifacts that full conditioning introduces.

Step 3 — Train the student on clean pseudo-labels. The teacher generates a pseudo-label by swapping a target image of identity A with a different subject B. This produces a pseudo-triplet: source image of A, pseudo-swapped label, and original target of A. The student takes identity features from the source and attribute features from the pseudo-label, in raw image form, and is trained to reconstruct the original target. Because the student sees clean images instead of degraded masked inputs, it learns better preservation and ends up outperforming its teacher. It also needs no auxiliary network or preprocessing pipeline for attribute conditioning at inference.

Evaluation uses 1,000 source and 1,000 target faces from FFHQ, producing 1,000 swapped results. FID measures fidelity against real FFHQ images; HopeNet measures pose and Deep3DFaceRecon measures expression via L2 distance to the target; ArcFace embeddings give identity similarity by cosine distance; and ID Retrieval reports Top-1 and Top-5 accuracy from cosine similarity matching.

Why This Matters

Impact on research. The paper reframes the face-swapping proxy-objective problem. Rather than asking how to condition a model on a masked input better, it asks how to build a substitute supervision signal good enough that masking becomes unnecessary. The finding that inversion noise's residual semantic content — usually treated as a nuisance that harms editability — is actually useful when the goal is to change identity but keep attributes, is a transferable insight for other inversion-based editing tasks. The result that a student trained on clean pseudo-labels can surpass the teacher

Authors’ abstract

Face swapping aims to transfer the identity of a source face onto a target face while preserving target-specific attributes such as pose, expression, lighting, skin tone, and makeup. However, since real ground truth for face swapping is unavailable, achieving both accurate identity transfer and high-quality attribute preservation remains challenging. Recent diffusion-based approaches attempt to improve visual fidelity through conditional inpainting on masked target images, but the masked condition removes crucial appearance cues, resulting in plausible yet misaligned attributes. To address this limitation, we propose APPLE (Attribute-Preserving Pseudo-Labeling), a fully diffusion-based teacher-student framework for attribute-preserving face swapping. Our approach introduces a teacher design to produce pseudo-labels aligned with the target attributes through (1) a conditional deblurring formulation that improves the preservation of global attributes such as skin tone and illumination, and (2) an attribute-aware inversion scheme that further enhances fine-grained attribute preservation such as makeup. APPLE conditions the student on clean pseudo-labels rather than degraded masked inputs, enabling more faithful attribute preservation. As a result, APPLE achieves state-of-the-art performance in attribute preservation while maintaining competitive identity transferability.