Zero-Shot Novel Depth Synthesis Using 3D Foundation Models Scene Representations

AI summary

Overview

Research area: Computer vision — 3D scene understanding, novel-view synthesis, monocular/multi-view depth estimation, and generative diffusion models applied to 3D foundation model (3DFM) representations.

Technical level: Advanced. The paper assumes familiarity with diffusion models (DDPM, latent diffusion, flow matching), transformer architectures, 3D foundation models (DUSt3R, VGGT, WorldMirror), and affine-invariant depth evaluation protocols.

Scope: The paper asks whether pretrained 3D foundation models implicitly encode geometry that is not visible in their input images, and, having found evidence that they do, builds a diffusion model (Z3D) that generates depth maps for novel camera poses by denoising inside the 3DFM's internal token space.

What This Paper Is About

Given one or a few posed images of a scene, the paper aims to predict plausible and geometrically consistent depth for camera viewpoints that were never observed, including surfaces hidden behind foreground objects. Existing neural scene representations such as NeRF and 3D Gaussian Splatting interpolate between observed views and are not designed to hallucinate occluded geometry, particularly from sparse inputs; existing depth diffusion models operate in 2D image space and can only predict depth for surfaces visible in the input. The core hypothesis is that solving 3D reconstruction forces 3D foundation models to learn a representation containing general knowledge about full 3D scenes, which can then be reused to infer unseen geometry.

Key Contributions

1. Evidence that 3DFMs encode hidden surfaces. The authors probe the internal features of VGGT and WorldMirror and show that even a linear model can predict layered depth images (LDIs), which represent the first k surfaces each camera ray passes through — including occluded ones. They also expand the VGGT DPT head to output four depth layers plus confidence maps while keeping all other layers fixed.

2. The Z3D method. A framework that performs latent diffusion directly in the space of 3DFM patch tokens, conditioned on the source views' 3DFM features and the relative poses of the target views, then decodes the denoised tokens with the 3DFM's frozen DPT head into novel-view depth maps.

3. A computational strategy for high-dimensional latents. Because the aggregated multi-layer token representations used for depth are high-dimensional (2048-dimensional in both VGGT and WorldMirror), Z3D diffuses only tokens from layer 17 of the target views (identified as contributing significantly more to depth prediction than the other three aggregation layers), while conditioning on the outputs from all four aggregation layers of the source views. The authors apply a timestep shift to the noise scheduler to stabilize denoising in this high-dimensional latent space.

4. Evaluation across multiple domains and models, with purpose-built baselines. Since no existing method directly addresses this setting (and the code for the closest related work, MVGD, was unavailable), the authors construct LVSM+3DFM baselines and Depth Diffusion (DD) baselines that use the Z3D architecture but operate in depth-map space instead of latent space. They evaluate on indoor and outdoor data and also retrain Z3D on VGGT-Ω without algorithmic modification to test model-agnosticism.

Main Findings

• LDIs are linearly decodable from 3DFM features. Using a linear probe, the dataset-average baseline achieves AbsRel 0.319 and δ<1.25 of 0.564; VGGT-LDI achieves AbsRel 0.197 and δ<1.25 of 0.717; WM-LDI achieves AbsRel 0.167 and δ<1.25 of 0.789. The authors state this roughly halves the error rate compared to the dataset average, indicating hidden/unseen information is embedded in the 3DFMs.

• Z3D outperforms depth-space diffusion baselines in out-of-domain settings. For 1 source to 1 target on DTU, Z3D-VGGT reaches AbsRel 0.076 and δ<1.25 of 0.935, and Z3D-WM reaches AbsRel 0.075 and δ<1.25 of 0.939, versus VGGT-DD at 0.098 / 0.914 and WM-DD at 0.108 / 0.902. On 7-Scenes, Z3D-VGGT reports AbsRel 0.012 and δ<1.25 of 0.999.

• Baselines trained for perceptual quality do not preserve geometry. LVSM+VGGT on 7-Scenes (1→1) reports AbsRel 0.468 and δ<1.25 of 0.024, with point cloud metrics Accuracy mean 7.681 / median 6.503 and Completion mean 23.396 / median 11.958. The authors note LVSM sometimes struggles to generate novel views fully consistent with the target pose, suggesting optimizing for perceptual quality can come at the expense of faithful scene geometry.

• Latent-space generation is qualitatively smoother than depth-space generation. Depth Diffusion predictions show noticeably less smooth depth maps and pronounced gradient spikes relative to Z3D, and produce noisier point clouds with reduced geometric sharpness (Figures 4 and 5).

• Multi-view consistency holds with few source views. With 2 source views and 4 target views, Z3D-VGGT and Z3D-WM generate geometrically consistent depth maps across all target views. On 7-Scenes (2→4), Z3D-VGGT reports AbsRel 0.024 and δ<1.25 of 0.995, versus VGGT-DD at 0.055 / 0.986.

• The two 3DFMs perform on par with each other. Z3D-VGGT and Z3D-WM "perform on par with each other across all experiments," which the authors read as evidence that both 3DFMs learn similar geometric structures of the scene.

• Outdoor results. Pulled out separately from the in-domain grouping, MegaDepth (2→4) shows Z3D-VGGT at AbsRel 0.039, δ<1.25 of 0.976, point cloud Accuracy mean 0.091 / median 0.020 and Completion mean 0.213 / median 0.023, versus VGGT-DD at AbsRel 0.102, δ<1.25 of 0.905, and WM-DD at AbsRel 0.111, δ<1.25 of 0.896. Z3D has substantially higher point cloud accuracy but slightly lower completeness than the DD models.

• The framework transfers to a newer 3DFM. Z3D-VGGT-Ω, trained only under the 1-source → 1-target setting with no algorithmic changes, achieves performance comparable to Z3D-VGGT and Z3D-WM (for example, DTU AbsRel 0.096 and δ<1.25 of 0.925; 7-Scenes AbsRel 0.019 and δ<1.25 of 0.999).

• In-domain all models do well. The authors report that all models, including the DD baselines, achieve strong performance on their respective training distributions.

• Failure mode under low overlap. Performance degrades when source and target views have limited overlap; both Z3D-VGGT and Z3D-WM then struggle to produce accurate novel-view depth, finer details are missing, and the projected point clouds become inconsistent. The method also inherits the 3DFM's sensitivity to the choice of reference view, since the first source image is treated as the reference camera.

Methodology in Plain English

The authors begin with a probing experiment. Since a layered depth image records not just the first surface a ray hits but the second, third, and fourth as well, predicting LDIs requires knowledge of the parts of a scene that are hidden from a given camera. They train a simple linear model on the frozen features of 3DFMs to predict LDIs and find it does far better than predicting the dataset average, which suggests the models already "know" about hidden structure. They also attach a modified DPT head to VGGT to output four depth layers and confidence maps while freezing everything else.

Building on that, Z3D works in three parts. First, a frozen 3DFM backbone turns source images into patch tokens. Second, a diffusion model — a DiT-style transformer — learns to predict the noise that was added to the target view's tokens, given the clean source tokens (supplied through cross-attention blocks in every DiTBlock) and the target camera pose (added directly to the timestep embedding, via adaptive layer normalization). Third, the recovered clean target tokens are handed to the 3DFM's frozen DPT head, which decodes them into a depth map — analogous to how a VAE decoder turns latents into images.

Only target-view tokens are noised during training; source tokens stay clean and act purely as conditioning. Because the full set of aggregation-layer tokens is high-dimensional (2048 dimensions), the diffusion process is restricted to layer 17 for the target views, while the richer multi-scale outputs from all four aggregation layers of the source views are used as conditioning. Camera poses are encoded as 12-dimensional vectors per view, Fourier-encoded and passed through an MLP, with all poses expressed relative to the first source view.

Training uses a flow-matching noise scheduler with a flow-velocity prediction objective, following Stable Diffusion v3's formulation, with 1000 timesteps and a dimension-dependent timestep shift of α = sqrt(m/n), where n = 4096 is the reference base dimension and m is the effective data dimension of the 3DFM backbone. Inference uses only 50 sampling steps. Training happens in two stages: Stage 1 with one source and one target view at effective batch size 128 for 98k steps, then Stage 2 initialized from Stage 1 with two source views and four target views at effective batch size 32 for 156k steps. Training Stage 2 from scratch was observed to be unstable and slower to converge. Both stages use AdamW with β₁ = 0.9, β₂ = 0.95, no weight decay, and a learning rate of 2×10⁻⁴, with linear warmup over the first 10% of total steps and decay beginning at 30% of training progress.

Training data is mined from five datasets — MegaDepth, Hypersim, Taskonomy, Replica, and Habitat HM3D — by computing pairwise camera frustum overlap and extracting sequences with strong geometric overlap. Reported dataset statistics are: Taskonomy (Tiny) 35 scenes / 381,840 images; Replica 18 scenes / 104,397 images; Hypersim 457 scenes / 74,619 images; Habitat-Matterport 900 scenes / 900,000 images; MegaDepth 196 scenes / 130,000 images. Scenes with more than 1,000 images are evenly subsampled to 1,000 cameras. Evaluation uses DTU, NRGBD, 7-Scenes, and an in-domain set sampled from the training datasets' test splits, with depth aligned using least-squares fitting (Weiszfeld method) under the affine-invariant protocol.

Why This Matters

This work reframes novel-view depth synthesis from a reconstruction or interpolation problem into a conditional generation problem over a foundation model's learned scene representation. It suggests that the internal features of 3D foundation models — not just their outputs — are a reusable asset, and that decoupling representation learning (handled by the 3DFM) from generative completion (handled by diffusion) is a productive design. Practically, the ability to infer geometry behind occlusions from only one or a few images is what separates passive rendering from systems that can reason and act in partially observed spaces.

Real-world applications:

• Robot navigation in environments where only partial views are available and occluded geometry must be inferred rather than observed.
• AR/VR content creation, where consistent depth across unobserved viewpoints supports rendering and scene editing.
• Autonomous systems that need to reason about geometry outside the current sensor field of view.
• Physical interaction and planning in incomplete 3D environments, where missing geometry must be hallucinated rather than directly measured.

Industry relevance: The method is largely model-agnostic — Z3D was retrained on VGGT-Ω with no algorithmic changes and kept competitive performance — which matters for practitioners who want to build on whichever 3D foundation model is current. The approach requires no per-scene optimization, which is a practical advantage over NeRF-style pipelines for deployment.

Future Directions

• Reducing dependence on view overlap. The authors identify low overlap between source and target views as the primary failure mode, with fine-grained geometry in novel viewpoints being lost and point clouds becoming inconsistent; improving this regime is a natural next step.
• Handling reference-view sensitivity. Because the first source image is the reference camera and prior 3DFM work has shown the reference choice affects performance, finding ways to make Z3D robust to that choice remains open.
• Scaling to additional 3DFMs. The VGGT-Ω experiment suggests model-agnosticism, but how far this extends to other architectures — and whether the layer-17 restriction remains the right choice — is not settled.
• Improving completeness. On MegaDepth, Z3D has substantially higher point cloud accuracy but slightly lower completeness than the Depth Diffusion models, indicating a trade-off not yet resolved.

Target Audience

Researchers and graduate students working on 3D vision, novel-view synthesis, depth estimation, and generative models — particularly those interested in how pretrained 3D foundation models can be repurposed rather than retrained. It is also relevant to practitioners in robotics, AR/VR, and autonomous systems who need geometric inference beyond the visible surface, and to anyone studying probing and interpretability of large 3D models. Given the reliance on diffusion theory, DiT architectures, and affine-invariant depth protocols, the paper is best suited to readers with an advanced background in computer vision and generative modeling.

Authors’ abstract

3D Foundation Models (3DFMs) such as VGGT have recently pushed the boundaries of 3D vision by predicting rich unified representations with feed-foward transformers. The scene representations learned by these models enable strong performance on multiple 3D vision tasks. In this paper, we investigate using their internal representations to infer 3D in the scene from new views. Our hypothesis is that in order to solve the task of 3D reconstruction, these models need to learn a representation that includes a large amount of general knowledge about 3D scenes. After showing that it is possible to decode hidden surfaces from internal 3DFM representations, we propose a method, Z3D, that estimates pointmaps in unseen views by doing latent diffusion on 3DFM representation. We show that Z3D can predict realistic depth maps for new views across multiple datasets.