MV-Performer: Taming Video Diffusion Model for Faithful and Synchronized Multi-view Performer Synthesis

AI summary

Overview

• Research area: Computer vision and computer graphics — specifically 4D (dynamic 3D) human novel view synthesis, video diffusion models, and multi-view video generation.
• Technical level: Advanced. The paper assumes familiarity with diffusion/flow matching, latent video autoencoders, DiT transformers, depth-based warping, and multi-view camera geometry.
• Scope: A single framework (MV-Performer) that converts one frontal monocular full-body video into multiple synchronized 360-degree novel-view videos of the same moving person, published at SIGGRAPH Asia 2025 (arXiv:2510.07190v1, 8 Oct 2025).

What This Paper Is About

Existing video diffusion models used for 4D novel view synthesis mostly re-aim the camera within the front view and cannot produce genuine 360-degree viewpoint changes, especially for people. MV-Performer targets the human-centric subdomain: given a monocular full-body video, it aims to synthesize synchronized videos from many viewpoints around the performer, including the back and sides that were never observed.

The core obstacles the authors identify are (i) insufficient 3D cues from monocular input, which makes front and back perspectives ambiguous, and (ii) intolerable "floater" artifacts from image warping under large viewpoint changes when monocular depth is inaccurate.

Key Contributions

1. First generative framework for monocular-to-dense-multi-view human video. The authors develop what they describe as the first framework for converting human-centric monocular video into dense multi-view videos, building on a cutting-edge video diffusion model and the MVHumanNet dataset.

2. A multi-view human-centric video diffusion model guided by camera-dependent normal maps. The model learns the joint distribution of multi-view human-centric videos and uses a normal map rendered from oriented partial point clouds as a geometric condition, extending the depth-based warping paradigm to large viewpoint changes.

3. A synchronization mechanism (Sync Attention). A frame-level spatial self-attention aggregates information across the reference view and all target views so that generated videos stay consistent with each other.

4. A robust in-the-wild inference procedure. By combining several state-of-the-art estimation methods into a depth-refinement pipeline, the framework greatly reduces artifacts caused by imperfect monocular depth estimation.

Main Findings

• Large margin over baselines on MVHumanNet (16-view training set of 5,400 subjects; 10-subject test set, even-numbered views). MV-Performer reaches PSNR 24.35, SSIM 0.926, LPIPS 0.066, FID 24.47, FVD 0.12. For comparison: Champ scores PSNR 11.23, SSIM 0.813, LPIPS 0.328, FID 55.92, FVD 5.54; ReCamMaster scores PSNR 6.97, SSIM 0.600, LPIPS 0.620, FID 154.03, FVD 10.78; ReCamMaster finetuned on MVHumanNet for 20 epochs reaches PSNR 11.62, SSIM 0.817, LPIPS 0.287, FID 26.44, FVD 2.17; TrajectoryCrafter scores PSNR 4.18, SSIM 0.493, LPIPS 0.722, FID 154.00, FVD 17.25.

• Large margin on DNA-Rendering (10 subjects sampled from the 8-camera-view subset). MV-Performer reaches PSNR 15.63, SSIM 0.861, LPIPS 0.152, FID 30.05, FVD 0.73. Champ: PSNR 9.08, SSIM 0.750, LPIPS 0.399, FID 58.59, FVD 4.73. ReCamMaster: PSNR 6.46, SSIM 0.595, LPIPS 0.602, FID 138.25, FVD 7.80. Finetuned ReCamMaster: PSNR 10.02, SSIM 0.769, LPIPS 0.342, FID 36.78, FVD 4.28. TrajectoryCrafter: PSNR 4.72, SSIM 0.498, LPIPS 0.758, FID 154.66, FVD 15.52.

• The camera-dependent normal condition matters. Removing it (ablation on DNA-Rendering) drops results to PSNR 15.61, SSIM 0.858, LPIPS 0.165, FID 36.60, FVD 0.837. Removing the sync module gives PSNR 15.38, SSIM 0.856, LPIPS 0.163, FID 38.96, FVD 0.898. Removing both gives PSNR 15.21, SSIM 0.850, LPIPS 0.169, FID 39.13, FVD 1.06. The full model scores PSNR 15.63, SSIM 0.861, LPIPS 0.152, FID 30.05, FVD 0.73.

• Sampling steps trade quality for cost. On MVHumanNet, FID improves from 55.54 at 5 steps to 43.43 at 10, 30.26 at 25, and 24.47 at 50 steps; PSNR shifts only slightly (24.90 at 5 steps versus 24.35 at 50). DNA-Rendering shows the same pattern (FID 54.85 at 5 steps down to 30.05 at 50). The authors report that 25–50 denoising steps strike a balance between quality and cost.

• Useful as a generative prior. Using MV-Performer to generate two side-view and one back-view video from frontal input to train GauHuman raises GauHuman's scores from PSNR 18.63, SSIM 0.866, LPIPS 0.179, FID 129.35, FVD 5.96 to PSNR 20.97, SSIM 0.901, LPIPS 0.146, FID 60.02, FVD 1.81. MV-Performer itself scores PSNR 24.35, SSIM 0.926, LPIPS 0.066, FID 24.47, FVD 0.12 in the same setting.

• Depth fidelity strongly affects output. Replacing the refined depth with the raw MegaSaM estimate produces noisy warping and unnatural bodies; the error grows with larger viewpoint changes.

• Frontal output is nearly pixel-aligned with ground truth, while back-side textures are synthesized and differ from ground truth but remain "reasonable and acceptable" per the authors.

Methodology in Plain English

The pipeline has three parts.

1. Turning the input into geometry. The system takes a frontal full-body video. In the wild, it estimates a coarse metric depth and camera parameters with MegaSaM, and high-quality relative depth plus normal maps with Sapiens. It aligns the relative depth to the metric depth by solving for a scale and shift with a least-squares closed-form solution, then refines the result using the normal map. From this depth, pixels are unprojected into a colored 3D point cloud and rendered into each target camera view, producing "partial renderings." Training pairs are constructed by RGBD-warping with the known camera parameters of the capture rig.

2. A geometric condition that removes front/back ambiguity. For each point, the dot product between the surface normal and the camera viewing direction is computed. If the result is positive the surface faces the camera; if negative it faces away. Normals are mapped from the [-1, 1] range into RGB [0, 1], and back-facing surfaces are painted black. This camera-dependent normal map is encoded by the 3D VAE alongside the partial rendering and concatenated with the noisy latents.

3. The diffusion model. The backbone is WAN 2.1 (Wan2.1-T2V-1.3B), a flow-matching video diffusion model whose 3D VAE compresses a video of f frames at (H, W) into a latent of shape [1 + f/4, H/8, W/8, C], with a 1.3B-parameter DiT running at 480px resolution. Two components are added to each DiT block: Ref Attention, a cross-attention where hidden latents query the reference-video latents (reusing the existing text cross-attention layer for simplicity), and Sync Attention, a frame-level spatial self-attention over the concatenated reference and target-view latents. No camera pose embedding is used, unlike some prior work.

Training is progressive. Stage one drops the sync module and trains everything else for 5 epochs (essentially video inpainting). Stage two freezes everything and trains only the sync module for 5 more epochs. AdamW is used with a learning rate of 1×10⁻⁴ decaying to 2×10⁻⁵, effective batch size 6×12 on 6 NVIDIA A100 GPUs, and K = 50 sampling steps. The full pipeline can generate about 10 videos with 49 frames on a 24G GPU such as an RTX 3090.

Evaluation uses PSNR, SSIM, LPIPS, FID, and FVD against ground-truth views. Baselines are TrajectoryCrafter, ReCamMaster (plus a version finetuned on MVHumanNet for 20 epochs), and Champ. Human4DiT and Disco4D are excluded because they focus on animation rather than 4D novel view synthesis, lack open-source code, or are too costly to reproduce. Five monocular videos from Bilibili and TikTok are collected to demonstrate generalizability.

Why This Matters

Impact on research. The paper argues that implicit camera embeddings are a poor fit for capture-rig datasets such as MVHumanNet, where the number of camera views is limited (open-source multi-view datasets typically use 32 to 60 fixed cameras). It shows that explicit depth-warping plus a camera-dependent normal map can push the warping paradigm to 360-degree synthesis, and that a video diffusion model can act as a generative 4D prior for downstream avatar reconstruction — an alternative to expensive synchronized multi-camera capture.

Real-world applications.

• Immersive VR/AR experiences where a viewer can move freely around a recorded performer.
• Free-viewpoint video for sports, performance, or broadcast content.
• Media content creation and movie production, where a single phone video could be expanded into multi-angle footage.
• Synthetic data generation and digital avatar creation, reducing reliance on studio capture cages.

Industry relevance. The method's ability to run on a 24G consumer GPU (RTX 3090) for roughly 10 videos at 49 frames is notable for practical deployment, though the paper notes that multi-step denoising still makes inference slow and computationally heavy. The code is released at https://github.com/zyhbili/MV-Performer.

Future Directions

• Face detail preservation. Despite WAN2.1's VAE, reconstruction errors still limit face-region quality and cap overall generation quality.
• Reducing reliance on depth estimation. Generated results fail when depth estimation is poor; the authors suggest finetuning the depth model using the metric human depth available in MVHumanNet++.
• Speed and compute. Distilling MV-Performer into a smaller, one-step denoising version is proposed as a path toward practical application.
• Fairness and data coverage. The model may degrade for untrained origin and certain skin tones, attributed to potential bias in WAN2.1 and the existing dataset; only the 1.3B version of WAN2.1 was tested due to computational limits.

Target Audience

Researchers and graduate students working on 4D human modeling, novel view synthesis, video diffusion models, and camera-controllable generation; graphics and vision engineers building free-viewpoint video or avatar pipelines; and practitioners in VR/AR, film, and media production who need multi-angle video from a single camera. Readers should already be comfortable with diffusion and flow-matching formulations, latent video representations, and multi-view geometry.

Authors’ abstract

Recent breakthroughs in video generation, powered by large-scale datasets and diffusion techniques, have shown that video diffusion models can function as implicit 4D novel view synthesizers. Nevertheless, current methods primarily concentrate on redirecting camera trajectory within the front view while struggling to generate 360-degree viewpoint changes. In this paper, we focus on human-centric subdomain and present MV-Performer, an innovative framework for creating synchronized novel view videos from monocular full-body captures. To achieve a 360-degree synthesis, we extensively leverage the MVHumanNet dataset and incorporate an informative condition signal. Specifically, we use the camera-dependent normal maps rendered from oriented partial point clouds, which effectively alleviate the ambiguity between seen and unseen observations. To maintain synchronization in the generated videos, we propose a multi-view human-centric video diffusion model that fuses information from the reference video, partial rendering, and different viewpoints. Additionally, we provide a robust inference procedure for in-the-wild video cases, which greatly mitigates the artifacts induced by imperfect monocular depth estimation. Extensive experiments on three datasets demonstrate our MV-Performer's state-of-the-art effectiveness and robustness, setting a strong model for human-centric 4D novel view synthesis.