Virtually Being: Customizing Camera-Controllable Video Diffusion Models with Multi-View Performance Captures

AI summary

Overview

Research area: Computer vision — controllable and customizable video diffusion models for virtual production, combining generative video with volumetric capture and 4D reconstruction.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers (DiTs), ControlNet-style conditioning, LoRA adaptation, DreamBooth fine-tuning, Plücker camera embeddings, and 4D Gaussian Splatting.

Scope: The paper introduces a data pipeline and two-stage training strategy that gives video diffusion models multi-view character consistency, precise 3D camera control, lighting adaptability, and multi-subject composition for filmmaking.

What This Paper Is About

Customized video generation typically conditions on a single reference image, which provides no view diversity and causes identity to break down when camera motion reveals the subject from new angles. The authors' goal is to generate videos of a specific person (or animal, or scene) that keep that identity consistent from any viewpoint while following an explicitly specified 3D camera path. They address this by repurposing 4D Gaussian Splatting (4DGS) reconstructions of professional volumetric capture performances as training data for video diffusion models.

Key Contributions

1. First framework to explicitly preserve multi-view identity under precise 3D camera control, enabled by a customization data pipeline that integrates professional volumetric capture, 4DGS reconstruction, and relightable rendering, combined with a two-stage training strategy (general camera-conditioned pretraining, then subject-specific customization).
2. A broad set of filmmaking-oriented generative capabilities: multi-subject generation via joint training and a noise-blending scheme that composes independently customized models at inference time, scene customization, real-life video-based customization, and control over subject motion and spatial layout.
3. Extensive benchmarking, ablations, and four user studies demonstrating improvements in multi-view identity preservation, camera control accuracy, and lighting control.

Main Findings

• Multi-view identity preservation wins on the primary metric: the method achieves the highest AdaFace score of 0.351 in the text-to-video customization comparison, ahead of ConsisID (0.301), MagicMe (0.280), VideoBooth (0.279), DreamVideo (0.194), and MotionBooth (0.191). The authors attribute ConsisID's shortfall to its reliance on a single facial image at inference.
• Temporal quality trade-off is acknowledged: MagicMe scores higher on VBench subject consistency (0.978 vs. 0.933 for Ours), background consistency (0.967 vs. 0.946), and reduced temporal flickering (0.984 vs. 0.975), but the authors report that MagicMe suffers substantial identity degradation and generates notably less motion. MotionBooth, which supports only 2D camera control, fails to generate identity-preserving videos.
• User study favors the method: with 19 participants across 60 prompts and two reference identities (Emily or Alex), the method was preferred in 81.3% of responses for multi-view identity preservation, 70.6% for facial realism, and 74.1% for text alignment. The corresponding table values are 81.34%, 70.59%, and 74.07%.
• Multi-view training data matters: an ablation trained only on frontal-view images drops the AdaFace score to 0.327 versus 0.351 for the full multi-view dataset, with visibly poorer identity preservation from side-view angles.
• Relit data improves lighting realism: in a user study across 60 prompts with 18 participants, the model trained with relit data was preferred in 83.9% of cases; without it, illumination appears flatter.
• Camera control errors are lowest: the pretrained camera-conditioned model reaches a translation error of 0.267 and rotation error of 0.047, versus AC3D (0.310 / 0.112) and CameraCtrl (0.522 / 0.163).
• Moving cameras during customization help: customization with dynamic trajectories gives 0.324 translation error and 0.086 rotation error, while a static-camera variant degrades to 0.482 and 0.125.
• Joint-subject data improves interaction: adding videos containing both subjects together raises interaction realism, with 72.9% of 18 participants across 60 prompts preferring it.
• Noise blending is a close, more flexible alternative: composing independently customized models at inference reaches an AdaFace score of 0.320 versus 0.337 for joint training, but requires no retraining for new subject combinations.
• Image-to-video customization is necessary: with a customized I2V model, AdaFace rises to 0.350 versus 0.324 for a non-customized model, and 65.43% of 18 participants across 60 prompts preferred the customized version (34.57% for non-customized).
• Stated limitations: fine-tuning is required to fully exploit high-quality multi-view 4DGS data, and the low resolution of the CogVideoX backbone underuses the higher-resolution inputs.

Methodology in Plain English

The authors start by recording real performances in a studio. A facial rig of 75 synchronized cameras arranged in a cylinder 2.5 meters tall and 2.7 meters in diameter captures face performances, while a 160-camera rig on a 4-meter-wide cylinder captures full-body motion. Each of the two subjects, "Alex" and "Emily," performs 3–6 multi-view sequences of roughly 50 to 180 frames at 24 frames per second, lit with multiple strobe lights for flat, diffuse illumination.

Rather than training directly on this footage, they reconstruct each sequence with 4D Gaussian Splatting and then re-render it from new viewpoints, randomly sampling start and end positions within a 2–10 meter radius and interpolating smoothly between them. Because 4DGS renders accurately but cannot invent new content, while video diffusion models can invent content but drift in identity, the rendered footage becomes a supervision signal that carries both multi-view identity information and exact camera annotations. A generalizable video relighting model, driven by HDRI maps from Poly Haven, adds lighting variety.

Training proceeds in two stages. First, a ControlNet-style branch learns camera-conditioned generation on general data — RealEstate10K for static scenes and HumanVid for dynamic, human-centric video — with the main DiT frozen. Camera information enters as Plücker coordinates, is applied only during the first 40% of denoising timesteps, and is injected into the first 25% of DiT blocks. Second, the model is customized DreamBooth-style: fine-tuned on the subject-specific data with each subject tied to a unique text token, while a regularization set sampled from the pretraining data protects general capability and camera control.

For image-to-video, initial frames come from a DreamBooth-fine-tuned FLUX.1-dev for single subjects or MuDI for multiple subjects, then a camera-conditioned I2V model fine-tuned on the same data animates them.

Multi-subject generation has two routes. Joint training simply mixes single-entity datasets (plus a small set of 27 joint-subject videos across 3 sequences). Noise blending instead composes independently customized models at inference: a generic prompt first produces a coarse layout video, SA2VA (built on SAM2) segments each subject into spatio-temporal masks, then denoising restarts from the same seed, runs the first 10% of steps without customization to lock the layout, and afterwards each subject's model predicts its own next latent, which are merged through the masks. Additional sources — 10 artist-designed CG indoor scenes rendered in Blender Cycles (24 trajectories each, 16 valid ones retained after filtering) and handheld iPhone videos of two indoor environments plus a cat, split into 20 two-second clips with poses from CUT3R — extend the framework to scene and real-life customization.

Why This Matters

Impact on research. The paper reframes 4DGS as a data generator rather than only a reconstruction technique, showing how accurate multi-view rendering can supply the view diversity that single-image customization lacks. It also extends camera-conditioned diffusion to customized subjects, an area where prior work such as MotionBooth was limited to 2D translation.

Real-world applications:

• Virtual production and previsualization, where directors need identity-consistent characters shot from arbitrary camera moves before principal photography.
• Visual effects and post-production, including replacing performers in existing footage while preserving the source video's motion and spatial layout via optical-flow control.
• Personalized content creation, generating videos of specific individuals in novel scenes and contexts while keeping their identity intact across viewpoints.
• Subject–scene composition for advertising or games, placing customized characters into customized CG or real-world environments with controllable interactions.

Industry relevance. The author list spans Eyeline Labs (United States, United Kingdom, Canada, Germany) and Netflix, and the work was published at SIGGRAPH Asia 2025 Conference Papers, reflecting direct production-pipeline interest. Features like inference-time composition of independently customized models and real-life video customization target practical requirements: avoiding retraining per subject combination and working with accessible handheld footage.

Future Directions

• Reducing or removing the need for fine-tuning so that high-quality multi-view 4DGS data can be leveraged more directly.
• Moving beyond the low-resolution CogVideoX backbone so that the higher-resolution capture inputs are not underused.
• Extending the framework to more subjects and richer inter-subject interactions, given that joint data was limited to 27 videos across 3 sequences and is reported to improve interaction realism.
• Closing the residual gap between joint training (AdaFace 0.337) and modular noise blending (0.320) while keeping the flexibility of inference-time composition.
• Broadening validation beyond the two studio subjects plus one real-life cat and the small set of 10 CG scenes.

Target Audience

Researchers and engineers working on video diffusion, controllable generation, and character customization; computer graphics and virtual production practitioners at visual effects and streaming studios; and graduate students with a background in diffusion models, 3D/4D reconstruction, or performance capture who want to understand how capture data can be converted into generative supervision.

Authors’ abstract

We introduce a framework that enables both multi-view character consistency and 3D camera control in video diffusion models through a novel customization data pipeline. We train the character consistency component with recorded volumetric capture performances re-rendered with diverse camera trajectories via 4D Gaussian Splatting (4DGS), lighting variability obtained with a video relighting model. We fine-tune state-of-the-art open-source video diffusion models on this data to provide strong multi-view identity preservation, precise camera control, and lighting adaptability. Our framework also supports core capabilities for virtual production, including multi-subject generation using two approaches: joint training and noise blending, the latter enabling efficient composition of independently customized models at inference time; it also achieves scene and real-life video customization as well as control over motion and spatial layout during customization. Extensive experiments show improved video quality, higher personalization accuracy, and enhanced camera control and lighting adaptability, advancing the integration of video generation into virtual production. Our project page is available at: https://eyeline-labs.github.io/Virtually-Being.