Preserving Source Video Realism: High-Fidelity Face Swapping for Cinematic Quality

AI summary

Overview

Research area: Computer vision, specifically video face swapping for film and entertainment production, combining diffusion-based video generation (DiT + Rectified Flow) with reference-guided editing.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers, rectified flow training objectives, VAE latent encoding, and standard face-swapping evaluation metrics.

Scope: The paper introduces LivingSwap, a video-reference-guided face-swapping pipeline that injects target identity through edited keyframes while preserving the source video's expressions, lighting, and motion, plus the Face2Face training dataset and CineFaceBench evaluation benchmark.

What This Paper Is About

Existing video face-swapping methods struggle to meet cinematic production standards: GAN-based approaches work frame-by-frame and produce flickering and temporal artifacts over long sequences, while diffusion-based inpainting methods discard the original facial pixels and rely on intermediate representations such as facial landmarks or 3D faces, losing fine detail. The paper's goal is to build a face-swapping model that directly references the raw source video's facial region so that identity is replaced while lighting, expression, subtle nuances, and background are faithfully preserved across long, complex shots. The authors propose LivingSwap, which they describe as the first video reference guided face swapping model.

Key Contributions

1. LivingSwap, a controllable long-video face-swapping pipeline. The method uses keyframes as conditioning signals for identity injection and combines them with video reference guidance and temporal stitching, which the authors report reduces the need for frame-by-frame human editing by a factor of 40.
2. The Face2Face paired dataset. A paired face-swapping dataset constructed by applying Per-frame Edit (via Inswapper) and then reversing the roles of the data pairs so that the GAN-swapped video becomes the model input and the original video supplies the ground-truth supervision and keyframes.
3. CineFaceBench. A cinematic-scenario benchmark containing 400 target–source test pairs covering long-take shots, complex lighting, exaggerated expressions, heavy facial makeup, and semi-transparent occlusions, with each video paired with an easy and a hard target image selected by identity similarity scores.
4. State-of-the-art empirical results. Quantitative comparisons on both FF++ and CineFaceBench show LivingSwap achieving the best average rank among the compared methods.

Main Findings

• Best average rank on FF++: Among Deepfakes, FaceShifter, InfoSwap, SimSwap, BlendFace, CanonSwap, DiffSwap, Face-Adapter, Inswapper, and LivingSwap, the proposed method achieved the best average rank (3.17) on the FaceForensics++ benchmark, which the authors characterize as containing relatively simple scenarios such as interviews and livestreams.
• Inswapper retains higher ID similarity but ranks lower overall: On FF++, Inswapper scored 0.636 ID Similarity versus LivingSwap's 0.592, and on CineFaceBench it scored 0.567 (easy) and 0.422 (hard) versus LivingSwap's 0.532 and 0.367. However, Inswapper's average rank was 2.500 on CineFaceBench versus LivingSwap's 1.667, and on FF++ it was 3.83 versus 3.17.
• Best FVD on both benchmarks: LivingSwap recorded FVD 54.32 (easy) and 63.97 (hard) on CineFaceBench, and 19.29 on FF++, the lowest among compared methods in both tables.
• Model design ablation confirms each component matters: Removing keyframe guidance dropped ID Similarity to 0.281 (from 0.536) and degraded temporal consistency; removing the target image lowered ID Similarity to 0.515; replacing the video reference with inpainting reduced fidelity metrics (Light 0.292 versus 0.285, Pose 2.87 versus 2.84).
• Data filtering provides no benefit: Comparing training on the top 70%, bottom 70%, and full Face2Face dataset gave nearly identical results (0.532, 0.540, and 0.536 ID Similarity respectively), leading the authors to use all data because diversity improves overall fidelity.
• The model surpasses its own training data quality: On manually selected noisy Face2Face pairs containing residual beards, artifacts, and misaligned expressions, LivingSwap produced results with improved expression alignment, greater visual realism, and fewer local artifacts than the Inswapper-generated pairs it was trained on.
• Keyframe guidance prevents accumulated identity drift: Using only the first frame for identity injection caused ID errors to accumulate as the video progressed; injecting swapped keyframe identities corrected each chunk individually.
• Keyframe selection matters in the worst cases: On the 10 worst-performing CineFaceBench cases, rule-based frontal-frame selection (yaw within ±30°, pitch within ±20°) achieved 0.240 ID Similarity and 2.393 Expression Error, versus 0.073 and 2.441 for a fixed-interval keyframe policy and 0.224 and 2.609 for Inswapper.
• Overlap is necessary between chunks: Generating chunks independently without temporal overlap produced noticeable frame discontinuities and temporal jumps at boundaries.

Methodology in Plain English

The task takes a source video, a mask marking the face region, and a target identity image. LivingSwap splits the work into four parts.

First, keyframe identity injection: a small set of frames exhibiting large changes in pose, expression, or illumination is selected as keyframes, and each is face-swapped with a high-quality image-level method (Inswapper), optionally refined manually in tools such as Adobe Photoshop. Each neighboring keyframe pair acts as a temporal boundary condition for the diffusion model.

Second, video reference completion: instead of masking the face and regenerating it from landmarks or 3D face representations, the model receives the complete source video segment as a visual reference, so lighting, subtle expressions, and background details survive intact. An optional target identity image improves identity in hard cases such as occlusions or closed eyes. All inputs are encoded with a VAE encoder and concatenated as temporally aligned latent tokens, alongside a binary mask marking the editable region. An attribute encoder built from DiT blocks, mirroring the diffusion backbone and initialized with matching pretrained weights, injects features into each backbone layer by element-wise addition.

Third, temporal stitching: long videos are divided into fixed-length chunks processed in chronological order. The first chunk takes both guidance frames from keyframes; later chunks use the final output frame of the previous chunk as start guidance and the next keyframe as end guidance. Auxiliary techniques include frame interpolation, temporal reverse playback, frame skipping, and multi-pass inference.

Fourth, training data: because paired source–target video data is scarce, the authors built Face2Face from CelebV-Text and VFHQ, applied Inswapper to generate swapped videos, then reversed the roles so the swapped video is the input and the original video supplies ground truth. Training uses the rectified flow loss on top of a 14B pretrained VACE initialization, 10,000 steps, AdamW at learning rate 1e-5, batch size 16, 640 resolution, 81 frames, and 8 NVIDIA H200 GPUs for approximately 14 days. Ablation models were trained for 2,000 steps.

Why This Matters

Impact on research. The paper reframes video face swapping as a reference-guided generation problem rather than a frame-by-frame or inpainting problem, providing a paired dataset (Face2Face) and a cinematic benchmark (CineFaceBench) that other researchers can use. It also demonstrates that reversing data pair roles and relying on strong pretrained priors lets a model learn from noisy, imperfectly generated supervision.

Real-world applications:

• Film and television post-production, where a performer's face must be replaced throughout long takes while preserving the original performance lighting and expression.
• Visual effects work requiring character de-aging or identity substitution across lengthy sequences with complex lighting, makeup, and partial occlusion.
• Advertising and entertainment content production that needs rapid, controllable identity swaps without frame-by-frame manual editing.
• Industrial editing workflows where artists refine a small number of keyframes, optionally in tools such as Adobe Photoshop, and leave the rest to the model.

Industry relevance. The authors frame the 40× reduction in manual labor and the model's robustness to imperfect keyframes as directly relevant to professional film and television deployment. The involvement of Ant Group as an author affiliation, alongside Zhejiang University, reflects this industrial orientation. The work was supported by the National Natural Science Foundation of China (No. 62576315, No. 62506338).

Future Directions

• Better keyframe selection. The authors show a simple rule-based frontal-frame rule (yaw within ±30°, pitch within ±20°) substantially improves the worst cases, implying automated or learned keyframe selection is an open opportunity.
• Replacing the Per-frame Edit module. Appendix C examines robustness to keyframe quality across different image-level face-swapping models, suggesting the pipeline's dependence on an external per-frame editor is a candidate for improvement.
• Robustness under identity and attribute variation. Appendices E and F explore identity differences and source-video attribute variations, pointing to generalization as a continuing question.
• Limitations. The supplementary material includes a dedicated limitations section (Appendix K), and comparison with closed-source methods is listed as Appendix J—taken together, these indicate known gaps that future work would need to address.

Target Audience

Researchers and practitioners in generative video editing, video face swapping, and visual effects. It is most useful to readers already comfortable with diffusion transformers, rectified flow, and standard face-swapping metrics who want a reference-guided alternative to inpainting pipelines, and to industry engineers evaluating face-swapping models for cinematic production where temporal consistency over long sequences is the primary constraint.

Authors’ abstract

Video face swapping is crucial in film and entertainment production, where achieving high fidelity and temporal consistency over long and complex video sequences remains a significant challenge. Inspired by recent advances in reference-guided image editing, we explore whether rich visual attributes from source videos can be similarly leveraged to enhance both fidelity and temporal coherence in video face swapping. Building on this insight, this work presents LivingSwap, the first video reference guided face swapping model. Our approach employs keyframes as conditioning signals to inject the target identity, enabling flexible and controllable editing. By combining keyframe conditioning with video reference guidance, the model performs temporal stitching to ensure stable identity preservation and high-fidelity reconstruction across long video sequences. To address the scarcity of data for reference-guided training, we construct a paired face-swapping dataset, Face2Face, and further reverse the data pairs to ensure reliable ground-truth supervision. Extensive experiments demonstrate that our method achieves state-of-the-art results, seamlessly integrating the target identity with the source video's expressions, lighting, and motion, while significantly reducing manual effort in production workflows. Project webpage: https://aim-uofa.github.io/LivingSwap