Beyond Inpainting: Unleash 3D Understanding for Precise Camera-Controlled Video Generation

AI summary

Overview

• Research area: Computer vision, specifically camera-controllable video generation with video diffusion models (VDMs), novel-view video synthesis, and 3D/4D scene understanding.
• Technical level: Advanced. The paper assumes familiarity with diffusion transformers, rectified flow / flow matching, LoRA adapters, monocular depth estimation, and 3D reconstruction from monocular video.
• Scope: The paper introduces DepthDirector, a video re-rendering framework that uses warped depth video rendered from an explicit 3D mesh as camera-control guidance, trained on a newly built Unreal Engine 5 multi-camera dataset called MultiCam-WarpData.

What This Paper Is About

The authors target a specific failure mode in camera-controlled video generation: most precise methods warp a reconstructed 3D representation of the input video into the target camera trajectory and then "repaint" (inpaint) the warped RGB result. Because monocular depth estimation is imperfect, the warped RGB video carries distortions and artifacts, so the generated output inherits those errors — the paper calls this the Inpainting Trap, describing it as a shortcut in which the model bypasses understanding of 3D/4D world structure and camera transformations. The goal is to keep precise camera controllability while faithfully preserving subject identity, fine detail, and dynamic content.

Key Contributions

1. Warped depth as camera-view condition. Instead of guiding the model with warped RGB, DepthDirector renders the depth of a dynamic 3D mesh under the target camera trajectory and uses that as the geometric guidance signal, which the authors argue is more robust to per-pixel errors of monocular depth than RGB-level guidance.
2. View–Content Dual-Stream Condition mechanism. The warped depth (plus occlusion mask) is encoded as a view branch and added into the noise latent, while the source video is concatenated frame-wise with the target tokens as a content branch, so the model receives both target viewpoint layout and source appearance/motion.
3. Lightweight LoRA-based video diffusion adapter. The pretrained video diffusion backbone stays frozen and only a small-rank LoRA adapter (rank 32) is trained, preserving the model's knowledge priors and keeping training cost low.
4. MultiCam-WarpData dataset. A large-scale multi-camera synchronized dataset built in Unreal Engine 5, containing 8K videos across 1K dynamic scenes, with ground-truth depth, rendered from 8 random camera trajectories per scene.

Main Findings

• The Inpainting Trap. The paper reports that warping-based methods are capped by the quality of the warped RGB video: unavoidable inaccuracies in 3D geometry from monocular video introduce distortions and artifacts, leading to subject inconsistency and degraded fidelity. The authors state this stems not from unrealistic rendering effects (e.g., flying pixels in point clouds in GEN3C and TrajectoryCrafter) but from unreliable depth information derived from monocular video, and that occlusion-aware masks from depth watertight meshes (EX-4D) did not solve the problem in essence.
• Camera accuracy. On Table 1, warping-based baselines achieve near-zero error because they inpaint directly on the warped video: TrajectoryCrafter (RotErr 0.828, TransErr 0.310, CamMC 1.171), GEN3C (1.166, 0.256, 1.649), EX4D (0.637, 0.167, 0.897). DepthDirector reaches RotErr 2.542, TransErr 0.388, CamMC 3.596 — described by the authors as comparable to warping-based methods and much lower than implicit camera-conditioned methods ReCamMaster (5.840, 1.055, 8.253) and CamCloneMaster (6.418, 1.027, 9.064).
• Identity preservation. DepthDirector reports the best Reference Similarity (0.6887) and InterFrame Similarity (0.9661), ahead of ReCamMaster (0.6403, 0.9519), EX4D (0.6269, 0.9279), GEN3C (0.6252, 0.9515), CamCloneMaster (0.5928, 0.9426), and TrajectoryCrafter (0.5672, 0.9169).
• View synchronization. DepthDirector reports Mat.Pix. of 988.7 and CLIP-V of 0.9197, versus EX4D (947.5, 0.9150), GEN3C (721.7, 0.9136), CamCloneMaster (507.9, 0.9057), ReCamMaster (479.4, 0.9201), and TrajectoryCrafter (275.3, 0.9119).
• Perceptual quality on VBench. DepthDirector reports the best score on nearly all VBench metrics: Subject Consistency 95.29, Background Consistency 94.66, Motion Smoothness 99.41, Aesthetic Quality 55.67, Imaging Quality 73.90. The next-best subject consistency is ReCamMaster at 95.17, and the next-best background consistency is GEN3C at 93.46.
• Ablation without the content condition. Removing the source-video injection lowers Reference Similarity to 0.5804 and Mat.Pix. to 772.0 (versus 0.6887 and 988.7 for the full model), and the qualitative figure shows the model failing to recover dynamic facial expression changes.
• Ablation with a lighter base model. Switching the base from Wan2.2-TI2V-5B to Wan2.1-T2V-1.3B at 843×480 resolution yields RotErr 2.366, TransErr 0.334 and Mat.Pix. 636.8; the authors state camera accuracy is not affected, while identity preservation, VBench consistency, and view synchronization are slightly lower due to weaker model capacity and lower resolution.
• Training cost. The authors report achieving superior performance with about 5.8% of the training cost of comparable approaches, using 8K video sequences versus, for example, the 136K videos cited for ReCamMaster.
• Dataset and training setup. MultiCam-WarpData contains 8K visually realistic videos from 1K dynamic scenes in 40 high-quality 3D environments with 8K different camera trajectories; each video is 576×1024 with 81 frames. Training uses 8 NVIDIA A100 GPUs and completes in 4 days; inference takes approximately 4 minutes per video with 50 denoising steps.
• Evaluation setup. 200 in-the-wild web videos sampled from the Koala dataset, each evaluated with two trajectories rotating ±30° around the main subject along the horizontal axis.
• Stated limitations. Concatenating source and target video tokens increases computational demands, and the framework cannot directly generate 360-degree rotation trajectories because warped depth video loses too much information at large viewpoint changes; the authors suggest autoregressive generation as a possible remedy.

Methodology in Plain English

The pipeline starts from an input monocular video. The authors estimate temporally consistent relative depth maps with a video depth estimator (DepthCrafter), and separately obtain camera extrinsics, intrinsics, and multi-view consistent per-frame depth from a 3D foundation model (Pi3). Because the two depth sources live in different coordinate frameworks, they solve for an optimal scale and shift factor to align relative depth with camera-space depth in inverse-depth space. The resulting point cloud is converted into a 3D mesh by connecting adjacent pixel points, following the approach of EX-4D, so that rendered depth has fewer noisy artifacts.

The mesh is then re-rendered under the target camera trajectory to produce a depth video and an occlusion mask (zero in occluded and out-of-frame regions). Raw depth values are clipped, normalized in log space into [0, 1], and mapped into RGB with a color map so the pretrained video VAE can encode them. Random scaling and shifting are applied before color mapping to make the model robust to the wide variation of depth scales across real input videos.

For conditioning, the encoded depth and mask are concatenated channel-wise, projected by a linear projector, and added to the noise latent tokens. The source video is compressed by the same VAE and concatenated with the target tokens along the frame dimension, doubling the token count; no additional attention layers are added because the existing 3D spatial-temporal self-attention already operates over all tokens. Only a LoRA adapter (rank 32) is trained on the frozen backbone using a flow-matching objective, with the AdamW optimizer at a learning rate of 1×10⁻⁴.

Training data comes from MultiCam-WarpData: environments and animated characters are assembled in Unreal Engine 5, and multiple cameras shoot each dynamic scene simultaneously along automatically generated trajectories, yielding paired source-view video, target-view video, and warped depth.

Why This Matters

• Research impact. The paper reframes camera-controlled video generation as a question of what geometric signal to condition on, rather than how to inpaint a warped image better. It argues that the RGB warp itself is the bottleneck, and that a depth-domain signal lets the diffusion model contribute its own 3D understanding. It also contributes a paired multi-camera dataset with ground-truth depth, addressing a stated shortage of suitable publicly available multi-view synchronized data.
• Real-world applications (potential, as the paper does not enumerate them):
  • Re-shooting existing footage from new angles for film, advertising, and short-form video production.
  • Virtual production and pre-visualization, where a director tests camera moves against already captured performances.
  • Viewpoint adjustment for VR/immersive playback and 3D scene reconstruction from casual video.
  • Robotic and embodied simulation, where changing viewpoints of a recorded scene supports policy learning or data augmentation.
  • Sports and surveillance review, where a fixed camera capture is re-rendered from a more informative angle.
• Industry relevance. The LoRA-based adapter keeps the frozen pretrained backbone intact and trains on 8 A100 GPUs in 4 days, with inference around 4 minutes per clip. The authors emphasize that this uses roughly 5.8% of the training cost of comparable approaches, which lowers the barrier for teams that cannot afford large-scale multi-view render dataset training.

Future Directions

• Handling large viewpoint changes. The paper reports that 360-degree rotation trajectories cannot be generated directly because warped depth loses too much information; autoregressive generation is proposed as a possible approach and left for future work.
• Reducing the token cost of dual-stream conditioning. Concatenating source and target tokens improves quality but increases computational demand, so more efficient conditioning schemes are an open problem.
• Mitigating dependence on depth estimator quality. The supplementary analysis shows that warps from all tested state-of-the-art video depth and 3D foundation models (VDA, DepthCrafter, GeometryCrafter, VGGT, Pi3, ViPE, DA3) contain distortion and artifacts without exception; further improving robustness to imperfect geometry remains open.
• Transferring the method to other base models and settings. The ablation with Wan2.1-T2V-1.3B indicates the approach is applicable to other video generation models, suggesting broader evaluation across backbones and resolutions.

Target Audience

Researchers and practitioners working on video diffusion models, camera-controllable video generation, novel-view synthesis, and 3D-aware generative modeling will gain the most. It is also relevant to graphics and VFX engineers interested in re-shooting captured footage, and to teams building multi-view synthetic datasets for video generation research. Readers need a working understanding of diffusion transformers, LoRA fine-tuning, and monocular depth estimation to follow the method and metrics in detail.

Authors’ abstract

Camera control has been extensively studied in conditioned video generation; however, performing precisely altering the camera trajectories while faithfully preserving the video content remains a challenging task. The mainstream approach to achieving precise camera control is warping a 3D representation according to the target trajectory. However, such methods fail to fully leverage the 3D priors of video diffusion models (VDMs) and often fall into the Inpainting Trap, resulting in subject inconsistency and degraded generation quality. To address this problem, we propose DepthDirector, a video re-rendering framework with precise camera controllability. By leveraging the depth video from explicit 3D representation as camera-control guidance, our method can faithfully reproduce the dynamic scene of an input video under novel camera trajectories. Specifically, we design a View-Content Dual-Stream Condition mechanism that injects both the source video and the warped depth sequence rendered under the target viewpoint into the pretrained video generation model. This geometric guidance signal enables VDMs to comprehend camera movements and leverage their 3D understanding capabilities, thereby facilitating precise camera control and consistent content generation. Next, we introduce a lightweight LoRA-based video diffusion adapter to train our framework, fully preserving the knowledge priors of VDMs. Additionally, we construct a large-scale multi-camera synchronized dataset named MultiCam-WarpData using Unreal Engine 5, containing 8K videos across 1K dynamic scenes. Extensive experiments show that DepthDirector outperforms existing methods in both camera controllability and visual quality. Our code and dataset will be publicly available.