FaceCam: Portrait Video Camera Control via Scale-Aware Conditioning

AI summary

Overview

Research area: Computer vision / generative video — camera-controllable portrait video generation via diffusion transformers.

Technical level: Intermediate. The paper assumes familiarity with camera intrinsics/extrinsics, multi-view geometry (fundamental/essential matrices, epipolar constraints), and diffusion/flow-matching video generation, but its core idea is intuitive.

Scope: A single paper (with supplementary material) proposing a scale-aware facial-landmark camera representation plus a data-generation pipeline for camera-controlled portrait video generation, validated on Ava-256 and in-the-wild videos.

What This Paper Is About

Given a single monocular portrait video, the goal is to re-render the same person from a user-specified camera trajectory while preserving identity, facial expression, head motion, and hair dynamics. Existing camera-control methods fail on portraits because extrinsic-parameter representations suffer from monocular scale ambiguity (the same parameter change produces wildly different visual effects depending on scene scale), and reconstruction-based methods propagate 3D estimation errors into visible facial distortions. FaceCam's answer is to condition the generator on rendered facial landmarks — an image-space representation of camera motion that never exposes the unobservable global scale.

Key Contributions

1. FaceCam, a portrait video camera-control system with a face-tailored, scale-aware camera representation that replaces scene-agnostic extrinsic parameters with rasterized 2D facial landmark maps, resolving monocular scale ambiguity and enabling intuitive camera-trajectory authoring.

2. A data generation and training pipeline built from static-camera multi-view studio captures plus unlabeled in-the-wild videos, using synthetic camera motion (zoom and pan) and multi-shot stitching, which supports continuous target camera motion at inference without any 4D synthetic data.

3. Empirical validation on the studio-captured Ava-256 dataset and 100 in-the-wild portrait videos, showing strong camera-control adherence alongside identity and motion preservation, with ablation studies isolating each training-data component and the choice of proxy 3D head.

Main Findings

• Ava-256 quantitative results: FaceCam reaches PSNR 15.85, SSIM 0.7208, LPIPS 0.2521, and ArcFace 0.8574, versus ReCamMaster (9.73 / 0.5570 / 0.5809 / 0.7014) and TrajectoryCrafter (10.32 / 0.5462 / 0.5673 / 0.5220). The variant FaceCam*, which uses a generic rendered 3D Gaussian head for conditioning when the target video is unavailable, reaches 9.83 / 0.5816 / 0.5494 / 0.8073.

• In-the-wild results: FaceCam scores 97.00 on camera correctness, 83.94 ArcFace Similarity, 73.49 Imaging Quality, 59.91 Aesthetic Quality, 94.77 Subject Consistency, 94.98 Background Consistency, 99.05 Motion Smoothness, and 96.00 Dynamic Degree. ReCamMaster scores 83.00 / 78.92 / 69.05 / 55.85 / 93.26 / 93.02 / 99.30 / 90.00; TrajectoryCrafter scores 99.00 / 49.79 / 71.37 / 55.76 / 92.23 / 92.25 / 98.97 / 97.00.

• Baseline failure modes observed: ReCamMaster struggles under large pose changes because of scale ambiguity, pushes the head out of frame, produces hallucinated backgrounds, and blurs under zoom-in. TrajectoryCrafter shows facial distortions from dynamic point-cloud estimation errors, flattened faces with weak facial texture, and high ArcFace loss.

• In-the-wild data ablation: Removing in-the-wild videos (FaceCam w/o In-the-wild Videos) yields 100.00 camera correctness but lower identity preservation and quality (77.73 ArcFace, 70.71 Imaging Quality, 55.73 Aesthetic Quality, 89.00 Dynamic Degree). The full model trades a small amount of correctness (97.00) for better identity and visual quality.

• Synthetic camera motion ablation: Removing it (keeping only multi-shot stitching) gives 96.00 camera correctness, 81.19 ArcFace, 72.03 Imaging Quality, 55.10–adjacent values, and 83.00 Dynamic Degree — the lowest Dynamic Degree among the compared variants.

• Multi-shot stitching ablation: Removing it (keeping only synthetic camera motion) drops camera correctness to 86.00 and ArcFace to 76.38, with 70.73 Imaging Quality, 55.10 Aesthetic Quality, and 80.00 Dynamic Degree — the worst camera correctness of all ablated variants.

• Proxy head insensitivity: Two additional proxy 3D heads give 97.00 camera correctness with ArcFace 84.45 and 84.74 respectively, versus 83.94 for the standard proxy, and near-identical VBench scores — indicating the landmark conditioning carries camera pose rather than identity or expression.

• Training generalizes from discontinuous to continuous motion: Although multi-shot stitching produces only discrete camera-pose changes during training, the model handles smooth continuous camera trajectories at inference.

• Geometric foundation of the representation: With at least seven 2D correspondences the fundamental matrix between two uncalibrated views can be estimated, and with known intrinsics the relative pose is recovered up to a global scale. Scaling 3D landmarks and translation by any factor s > 0 leaves the 2D projections unchanged, matching the scale invariance of monocular capture.

• Generalization beyond training distribution: The supplementary material reports the same pipeline extends to stylized inputs such as cartoon characters, and that FaceCam synthesizes realistic outpainted regions (for example bun hairstyles) and renders co-occurring objects such as headsets, confetti, cosmetics, jewelry, headbands, microphones, and glasses.

Methodology in Plain English

The authors treat a portrait video as a recording of a dynamic 4D scene along a per-frame camera trajectory, and the task as generating that same scene from a new trajectory given only the source video and the target trajectory. Instead of telling the model "the camera is at this rotation and translation," which is ambiguous for monocular input because absolute scale cannot be observed, they tell it where facial landmarks land in pixel space.

Concretely, they take landmarks from the anchor frame, define a head-centric coordinate system, and rasterize the projected 2D landmarks of the desired target camera pose into pixel-space image channels. Because scaling the 3D landmarks and the translation by the same positive factor leaves the 2D projections identical, this conditioning signal is inherently scale-invariant and never exposes the unobservable metric scale. At inference the user only needs a target trajectory: a proxy 3D Gaussian head is rendered along that trajectory, MediaPipe detects landmarks in each rendered frame, and those landmark maps become the condition. The proxy head can be of any identity and is unrelated to the input video.

Training data is the hard part. The authors start from NeRSemble, a studio multi-view dataset with 425 subjects, 16 synchronized viewpoints, and roughly 9.4K video sequences — but its cameras are static and all captures share identical studio lighting. They therefore augment: randomly scaling clips with factor s in [0.75, 1.25] and replacing backgrounds with random colors shared between source and target; synthesizing zoom (start/end scale ratios sampled from [1.0, 1.25], linearly interpolated per frame, restored to original resolution by cropping or padding) and pan (linearly interpolated cropping/padding offsets per frame); and multi-shot stitching, which picks 1–4 clips from different camera poses, trims them to different temporal segments, and concatenates them so one sequence shows changing viewpoints. To break out of the studio domain they add roughly 800 collected monocular in-the-wild portrait videos, applying synthetic camera motion to create virtual target views since no second viewpoint exists. The final training set is about 9.1K videos (8.9K from NeRSemble plus about 200 in-the-wild).

The model itself is built on the open-source Wan video foundation model. The source video latent is concatenated with the noise latent through frame conditioning, and the camera conditioning latent is applied through channel conditioning. The authors fine-tune the 3D attention layers and projection layers of the diffusion model with a flow-matching loss, training for 3K steps on 24 NVIDIA A100 GPUs at a constant learning rate of 5e-5 and batch size 24.

Evaluation uses two benchmarks. For the static camera setting, 10 identities from Ava-256 each get 10 input-output camera pairs, giving 100 videos; baseline camera extrinsics are converted into each method's coordinate system. For the dynamic setting, 100 in-the-wild videos are each assigned one of 10 canonical camera motions (Pan Left/Right/Up/Down, Zoom In/Out, Arc Left/Right/Up/Down), 10 videos per motion. Since ground-truth target videos do not exist in the wild, camera-following is measured by detecting MediaPipe landmarks in the last generated and input frames, estimating head-pose difference, and assigning a binary correctness label based on whether the measured shift matches the intended trajectory.

Why This Matters

Impact on research: The paper reframes camera control for portrait video as a correspondence problem rather than a parameter-prediction or reconstruction problem, sidestepping both monocular scale ambiguity and 3D estimation error amplification. The data-generation strategies matter independently of the representation: they show that static-camera multi-view captures and unlabeled in-the-wild videos can be bootstrapped into a model that handles continuous trajectories, without any 4D synthetic data.

Real-world applications:

• Social media and short-form video, where creators want multi-angle re-framing from a single phone recording.
• Post-production, where a single locked-off take can be re-angled or extended without a reshoot.
• Telepresence and video conferencing, for virtual camera movement around a speaker.
• AR/VR avatar pipelines, which need re-rendered portrait viewpoints that preserve expressions and hair.

Industry relevance: The system builds on existing foundation video models and off-the-shelf components (FaceLift for the proxy 3D Gaussian head, MediaPipe for landmark detection), and its inference-time conditioning requires no per-video optimization or 3D-aware instrumentation — a practical profile for deployment. The demonstrated robustness to outpainted regions and co-occurring objects like headsets and microphones points at real videoconferencing and creator tooling rather than only studio footage.

Future Directions

• Removing the landmark-detection dependency: Inference relies on MediaPipe detection on rendered proxy frames; sensitivity to detection failure or unusual head geometry is not characterized in the reported results.
• Extending beyond the head: The conditioning is face-tailored. How the approach scales to full-body or multi-person camera control is not addressed.
• Data quality ceiling: Ablations show in-the-wild data improves identity and quality while slightly reducing camera correctness (100.00 to 97.00), so the trade-off between domain coverage and control precision remains an open tuning question.
• Generalizing the proxy assumption: The paper shows insensitivity across three proxy 3D heads, but the design principle of "any identity, unrelated to input" invites further study on how far the conditioning signal can be abstracted from a rendered mesh.

Target Audience

Researchers and practitioners in generative video and novel view synthesis, especially those working on camera-controllable video diffusion models; graphics and AR/VR engineers building portrait re-rendering and avatar pipelines; and technical readers interested in how classical multi-view geometry constraints (scale ambiguity, epipolar correspondence, PnP) can be re-expressed as conditioning signals for modern diffusion transformers.

Authors’ abstract

We introduce FaceCam, a system that generates video under customizable camera trajectories for monocular human portrait video input. Recent camera control approaches based on large video-generation models have shown promising progress but often exhibit geometric distortions and visual artifacts on portrait videos due to scale-ambiguous camera representations or 3D reconstruction errors. To overcome these limitations, we propose a face-tailored scale-aware representation for camera transformations that provides deterministic conditioning without relying on 3D priors. We train a video generation model on both multi-view studio captures and in-the-wild monocular videos, and introduce two camera-control data generation strategies: synthetic camera motion and multi-shot stitching, to exploit stationary training cameras while generalizing to dynamic, continuous camera trajectories at inference time. Experiments on Ava-256 dataset and diverse in-the-wild videos demonstrate that FaceCam achieves superior performance in camera controllability, visual quality, identity and motion preservation.