Novel View Synthesis from A Few Glimpses via Test-Time Natural Video Completion

AI summary

Overview

• Research area: Sparse-input novel view synthesis (NVS), combining 3D Gaussian Splatting with pretrained video diffusion priors.
• Technical level: Advanced. The paper assumes familiarity with diffusion models, denoising score/ODE formulations, and 3D Gaussian Splatting.
• Scope: A zero-shot, test-time framework that treats sparse-view NVS as "natural video completion," using Stable Video Diffusion guided by 3D-GS-derived depth warps and uncertainty maps to synthesize pseudo-views that densify and constrain 3D-GS reconstruction, evaluated on LLFF, DTU, DL3DV, and MipNeRF-360.

What This Paper Is About

Reconstructing a 3D scene from only a few widely spaced photos usually produces blurry or artifact-ridden renderings, because NeRF and 3D Gaussian Splatting need dense supervision to resolve occlusions and geometric ambiguity. This paper reframes the problem: instead of only filling spatial gaps between views, treat the missing views as missing frames of a natural video unfolding through space, and let a pretrained video diffusion model hallucinate the in-between frames. Those hallucinated frames then become extra training supervision for a 3D-GS scene representation, iteratively improving both the generated views and the 3D geometry.

Key Contributions

1. A zero-shot, generation-guided 3D-GS pipeline that uses pretrained video diffusion models to improve novel view synthesis under sparse inputs, with particular attention to under-observed regions. The framework requires no scene-specific training or fine-tuning.
2. An uncertainty-aware modulation mechanism that couples 3D-GS with video diffusion for controllable pseudo-view generation. Guidance images are built by inverse warping pixels from the nearest input view using 3D-GS depth maps, and per-pixel uncertainty from a forward/backward cross-view consistency check controls how strongly the diffusion prediction is blended with the guidance.
3. A Gaussian primitive densification module that uses the generated pseudo-views together with a dense stereo model to add new Gaussian primitives in under-observed regions, improving scene completeness.
4. State-of-the-art results under extreme sparsity, including a reported gain of over 2.5 dB PSNR on DL3DV, plus strong results on LLFF, DTU, and MipNeRF-360. The approach is described as agnostic to the diffusion backbone, using Stable Video Diffusion primarily but compatible with alternatives.

Main Findings

• DL3DV (3, 6, 9 views): The method achieves 14.62 / 17.35 / 19.19 PSNR, 0.471 / 0.566 / 0.616 SSIM, and 0.491 / 0.396 / 0.335 LPIPS. The strongest listed baseline at 9 views is FSGS at 15.52 PSNR, 0.468 SSIM, 0.416 LPIPS; IPSM reaches 13.41 PSNR at 9 views and DNGaussian 13.44. The paper reports a gain of over 2.5 dB PSNR on DL3DV.
• LLFF (3 training views): 20.61 PSNR, 0.705 SSIM, 0.201 LPIPS, versus IPSM at 20.44 / 0.702 / 0.207, FSGS at 20.31 / 0.652 / 0.288, and plain 3D-GS at 17.43 / 0.522 / 0.321. The authors attribute the gain mainly to the extra supervision from pseudo-views, and highlight LPIPS as the metric most correlated with perceptual similarity.
• DTU (3 training views, object-centric): 20.51 PSNR, 0.840 SSIM, 0.137 LPIPS, described as outperforming the second-best approach by a significant margin across all three metrics (the next-best listed PSNR is FreeNeRF at 19.92; next-best SSIM is SparseGS at 0.834; next-best LPIPS is DNGaussian at 0.176).
• MipNeRF-360 (9 training views): 17.91 PSNR, 0.495 SSIM, 0.435 LPIPS, exceeding ViewCrafter (16.68 / 0.382 / 0.551), 3DGS-Enhancer (16.22 / 0.399 / 0.454), and MVSplat 360 (14.86 / 0.321 / 0.528).
• Rendering speed advantage: NeRF-based methods are noted at roughly 0.21 FPS, while the 3D-GS-based approach supports real-time rendering at around 430 FPS.
• Guidance image construction matters: Replacing 3D-GS color renderings with warping-based guidance lifts DL3DV 9-view PSNR from 16.59 (baseline 3D-GS) to 18.59 ("w/ GS interpolation") and to 19.19 ("w/ warping interpolation", the full model).
• Uncertainty components both help: Removing the geometric term drops PSNR to 18.21, removing the photometric term drops it to 18.93, compared with 19.19 for the full model. The paper states the video diffusion model cannot generate faithful images without the uncertainty mask.
• Densification and loss design matter: Removing GS densification drops PSNR to 18.23, removing point filtering gives 19.01, and removing the LPIPS loss for pseudo-views gives 18.81, against 19.19 PSNR, 0.616 SSIM, 0.335 LPIPS for the full model.
• Image-prior baselines struggle more than video-prior ones: IPSM, which uses an image diffusion model with Score Distillation Sampling, is reported to struggle with extremely sparse inputs because an image model lacks global scene context, whereas the video diffusion model can infer it from the reference frame.

Methodology in Plain English

The pipeline is iterative and involves no training of the diffusion model on the target scene.

1. Initialize: Train a 3D Gaussian Splatting model on the sparse input views.
2. Make guidance images: Pick camera poses between adjacent input views. Rather than rendering color from the (still imperfect) 3D-GS, project pixels backward from the nearest real input image into the target pose using depth maps the 3D-GS renders. The result is a structurally aligned but potentially incomplete guide image.
3. Score uncertainty: Project each guide pixel forward into the input image, then back again. Compare the round-trip position error and the color mismatch between the 3D-GS render and the input image. Large discrepancies mean high uncertainty — typically occlusions or poorly observed regions.
4. Modulate the diffusion model: Feed the guide images into a pretrained video diffusion model (Stable Video Diffusion). At each denoising step, solve a small weighted least-squares problem per frame that blends the model's own clean-latent prediction with the guidance latent. The weight is zero in high-uncertainty regions and near certain timesteps, so the diffusion model is free to invent content there; elsewhere the guide dominates to preserve structure.
5. Interpolate between two views: Run the diffusion model both forward and backward from the two flanking inputs and blend the two latent sequences with a linearly varying per-frame weight, which reduces drift under large viewpoint changes.
6. Densify geometry: Select a subset of pseudo-views with low mutual visibility overlap, run a dense stereo model to get a point cloud, filter outliers, and only add new Gaussian primitives where no existing primitive is nearby.
7. Retrain the 3D-GS: Optimize using both real views (L1, D-SSIM, and a depth regularization term) and pseudo-views (LPIPS, D-SSIM, and the depth regularization term). Each cycle trains 10K iterations, then refreshes the pseudo-views; the learning rate schedule is reset each cycle to avoid overfitting.
8. Repeat steps 2-7 so geometry and generated views improve each other.

Key hyperparameters reported: bandwidth parameters s1 = 100 and s2 = 0.25, uncertainty threshold delta = 0.5, and loss weights w1 = 0.8, w2 = 0.2, w3 = 1.0, w4 = 1.0, w5 = 0.2, w6 = 1.0. The frame-wise uncertainty threshold uses tau = (k / HW) * sum of U_i(p) + b, where k and b are described as tunable coefficients, with their values not given in the visible content.

Why This Matters

• Research impact: The work shows that zero-shot video diffusion priors — models trained on raw video without camera annotations — can serve as practical, generalizable supervision for sparse 3D reconstruction, avoiding the cost of fine-tuning view-conditioned video models. It also positions sparse NVS as a video-completion problem, which reframes how priors and geometry exchange information. The authors note the concurrent work of Zhong et al. (2025) is closely related but depends on a video diffusion model trained with camera poses and was not publicly available at submission time.
• Real-world applications:
  • Reconstructing walkable 3D scenes from casual handheld smartphone video taken while walking, where views are widely spaced and unconstrained.
  • Real estate, construction, and inspection capture, where a few photos of a room or site need to become a navigable 3D model.
  • AR/VR and gaming content pipelines that turn a small photo set into an explorable scene.
  • Robotics and simulation, where sparse real captures must be turned into dense, renderable environments for training or planning.
• Industry relevance: The real-time rendering speed (around 430 FPS reported, versus about 0.21 FPS for NeRF-based comparisons) and the absence of per-scene fine-tuning are practically attractive for interactive 3D products, mapping services, and content creation tools where compute budgets and turnaround time matter.

Future Directions

• Explicit future work is not reported in the visible portion of the paper (the text is truncated mid-ablation). The following are open questions the work raises:
• Extending the framework beyond the primarily used Stable Video Diffusion backbone to other video diffusion models, and quantifying how much pseudo-view quality and reconstruction accuracy depend on the backbone choice.
• Reducing or eliminating the iterative test-time cost, since each cycle retrains 3D-GS for 10K iterations before refreshing pseudo-views with the diffusion model.
• Handling residual temporal inconsistency in pseudo-views, which the paper says persists in distant regions with weak geometry and in fine-grained textures such as grass or tree leaves, and currently motivates using LPIPS rather than L1 loss.
• Improving generalization to highly unconstrained or dynamic camera trajectories, where the paper reports DL3DV as substantially more challenging than object-centric or forward-facing benchmarks.

Target Audience

Researchers and practitioners in computer vision, computer graphics, and 3D reconstruction who work on novel view synthesis, Gaussian Splatting, or diffusion-based generative priors. It is most useful to readers already comfortable with 3D-GS optimization and diffusion sampling; readers new to either area will need background reading. It will also interest engineers building photorealistic 3D capture or interactive rendering products from sparse photo or video inputs.

Authors’ abstract

Given just a few glimpses of a scene, can you imagine the movie playing out as the camera glides through it? That's the lens we take on \emph{sparse-input novel view synthesis}, not only as filling spatial gaps between widely spaced views, but also as \emph{completing a natural video} unfolding through space. We recast the task as \emph{test-time natural video completion}, using powerful priors from \emph{pretrained video diffusion models} to hallucinate plausible in-between views. Our \emph{zero-shot, generation-guided} framework produces pseudo views at novel camera poses, modulated by an \emph{uncertainty-aware mechanism} for spatial coherence. These synthesized frames densify supervision for \emph{3D Gaussian Splatting} (3D-GS) for scene reconstruction, especially in under-observed regions. An iterative feedback loop lets 3D geometry and 2D view synthesis inform each other, improving both the scene reconstruction and the generated views. The result is coherent, high-fidelity renderings from sparse inputs \emph{without any scene-specific training or fine-tuning}. On LLFF, DTU, DL3DV, and MipNeRF-360, our method significantly outperforms strong 3D-GS baselines under extreme sparsity.