Video Generative Models as Geometry Learner

AI summary

Overview

Research area: Computer vision / 3D scene understanding — specifically zero-shot monocular depth and surface normal estimation using pretrained generative models.

Technical level: Advanced. The paper assumes familiarity with latent diffusion models, the EDM noise formulation, video diffusion backbones (Stable Video Diffusion), VAEs, and standard depth/normal evaluation metrics (AbsRel, δ₁, mean angular error, 11.25° accuracy).

Scope: The paper repurposes a pretrained image-to-video diffusion model (Stable Video Diffusion) into a unified geometry estimator called GeoNeXt, by reformulating depth and surface normal prediction as next-frame prediction.

What This Paper Is About

Estimating 3D geometry (depth and surface normals) from a single RGB image is normally solved either by large discriminative models trained on tens of millions of images, or by adapting pretrained image diffusion models that treat geometry as a "special image" to be generated. Both routes have drawbacks: task-specific diffusion models train a separate model per geometry target, while joint models that alter the diffusion architecture need much more labeled data.

The paper's goal is a single, data-efficient model that predicts depth and surface normals together, by borrowing the temporal priors of a pretrained video generative model and treating the geometry maps as the next frames that follow the input image.

Key Contributions

1. A new generative formulation. GeoNeXt reframes monocular geometry estimation as a next-frames prediction task inside a video diffusion model, giving a unified geometry learner for depth and surface normals rather than one model per target.
2. A studied fine-tuning protocol. The authors analyze the generative formulation, architectural choices (e.g., the CLIP conditioning branch, the image reconstruction branch) and the influence of the reconstruction order on robustness and performance.
3. Lightweight adaptation. Only the denoising U-Net is fine-tuned, with minimal architectural changes (removing the text/CLIP branch) on modest RGB–depth–normal triplets from Hypersim and Virtual KITTI 2.
4. Strong zero-shot results with little data. Extensive experiments across five depth and five normal benchmarks show GeoNeXt outperforming task-specific and unified generative competitors while using substantially less training data, and rivaling discriminative models trained on orders of magnitude more data.

Main Findings

• Data efficiency: GeoNeXt is trained on 59K samples. For comparison, GeoWizard is trained on 208K, Marigold and E2E-FT on 74K, DepthFM on 63K, Lotus-G on 59K, while large discriminative models use far more — MiDaS 2M, DPT 1.4M, Omnidata 12.2M, DA and DA-V2 62.6M. The text states discriminative models such as DepthAnything rely on 63.5M training images, and that GeoNeXt uses nearly 100× less data.

• Zero-shot depth (Table 1): GeoNeXt reports AbsRel / δ₁ of 5.3 / 96.8 on NYUv2, 8.2 / 92.6 on KITTI, 5.6 / 97.2 on ETH3D, 5.9 / 95.8 on ScanNet, and 22.6 / 74.3 on DIODE. On ETH3D this contrasts with AbsRel 13.1 for DA-V2 and 12.7 for DA in the same table.

• Beat the unified competitor on less data: The paper reports that GeoNeXt surpasses GeoWizard by 6.2 in AbsRel and 9.4 in δ₁ on KITTI, and by 10.4 in AbsRel and 1.8 in δ₁ on DIODE, despite GeoWizard using 208K training samples versus GeoNeXt's 59K.

• Zero-shot surface normals (Table 2): GeoNeXt reports mean angular error / 11.25° accuracy of 16.7 / 60.0 on NYUv2, 16.0 / 62.8 on ScanNet, 16.4 / 69.2 on iBims-1, 33.0 / 21.5 on Sintel and 22.8 / 30.8 on OASIS. The text highlights mean angular errors of 16.4 on iBims-1 and 33.0 on Sintel while jointly estimating depth and normals with significantly less training data.

• Reconstructing the image matters: Removing the image reconstruction branch (variant (a), geometry-only generation) degrades depth to AbsRel 6.5 on NYUv2 and 6.7 on ETH3D, and normals to mean 17.9 on NYUv2 and 18.5 on iBims-N — confirming image–geometry co-generation helps.

• CLIP conditioning slightly hurts: Removing the CLIP embedding branch gives AbsRel 5.6 on NYUv2 and 5.8 on ETH3D (versus 5.3 and 5.6 for the full model), and normal means of 16.9 and 16.5. The authors suggest resizing the image for CLIP features distorts structural information, so the full model disables this conditioning.

• Joint estimation beats separate models: A depth-only model reaches AbsRel 5.9 on NYUv2 and 6.4 on ETH3D; a normal-only model reaches mean 17.2 on NYUv2 and 17.3 on iBims-N. Both are worse than the unified model (5.3 and 5.6 for depth; 16.7 and 16.4 for normals).

• Reconstruction order is nearly irrelevant: Swapping which frame holds depth versus normals ("normal-depth" vs "depth-normal") produces nearly identical numbers (e.g., AbsRel 5.7 vs 5.3 on NYUv2; mean 16.6 vs 16.7), indicating robustness to ordering.

• Cost-effectiveness (Table 5): Measured on a single NVIDIA A5000 at 768×768 input, GeoNeXt with 1×1 NFEs runs in 1.0 s with 1.5B parameters; with 5×5 NFEs it takes 10.0 s. Marigold is 0.8 s at 1×1 and 180.5 s at 50×10 (1.9B parameters); GeoWizard is 1.2 s at 1×1 and 272.1 s at 50×10 (0.9B parameters); Lotus-G is 0.8 s (1.9B). The paper notes I/O overhead from loading separate checkpoints is not included in the reported runtimes.

Methodology in Plain English

The authors start from Stable Video Diffusion, a model that generates a video sequence from a single conditioning image. Their key trick is to relabel the problem: instead of asking for a new video, they ask the model for the next frames after the input image, where those "frames" are a depth map and a surface normal map.

Concretely, an RGB image, its depth map and its normal map form a three-item triplet. All three are encoded into a compact latent space using a frozen Stable Diffusion VAE (single-channel depth is tiled to three channels and averaged back at decoding; depth and normals are normalized to [0,1] and then rescaled to [-1,1]). The image latent is replicated into the geometry slots as a strong conditioning signal, and the geometry latents are concatenated with it. The model then denoises the whole triplet together — image included — rather than generating geometry alone, which keeps the image and its geometry consistent. Noise follows the EDM formulation, and only the denoising U-Net is fine-tuned; the CLIP cross-attention conditioning path is disabled because resizing images for CLIP distorts geometry.

Training uses standard EDM noise sampling with σ drawn from N(0.7, 1.6), the Adam optimizer at a learning rate of 5×10⁻⁶ and random horizontal flipping. Depth is handled in disparity space (d = 1/d′). Training data is Hypersim (approximately 54K official samples filtered to about 39K valid ones, resized to 576×768) and Virtual KITTI 2 (four of five scenes, roughly 20K samples, cropped to 352×1216 with an 80 m far-plane limit), mixed at a 9:1 ratio following Marigold.

At inference, latent variables for image, depth and normal are initialized from Gaussian noise, the encoded input image is concatenated in, and the U-Net iteratively denoises all three. The image latent is then discarded and the depth and normal latents are decoded by the VAE. The default setup uses 5 denoising steps with an ensemble of 5 runs using different noise seeds.

Why This Matters

Impact on research. The paper shows that video generative priors — temporal attention and richer generative knowledge from large-scale video pretraining — transfer usefully to a static 3D perception problem. It suggests an alternative axis to scaling labeled data: reusing a different class of pretrained generative model and reformulating the task so the pretraining and fine-tuning objectives are close. It also argues that jointly modeling image and geometry is better than geometry-only generation, and that a unified model beats two separate task-specific models.

Real-world applications (the ones the paper names or implies):

• Autonomous driving (listed as an application domain, and Virtual KITTI is one of the training datasets).
• 3D surface reconstruction and mesh reconstruction from single images.
• Inverse rendering.
• Controllable image generation and image relighting, using predicted geometry as structural cues.

Industry relevance. A single 1.5B-parameter checkpoint that produces both depth and normals in about 1.0 s at 1×1 NFEs on an NVIDIA A5000 is attractive for deployment, since separate depth and normal models mean two checkpoints, extra storage, and repeated GPU loading I/O. Data efficiency also matters commercially: 59K training samples is a far cheaper fine-tuning budget than 208K or tens of millions of images.

Future Directions

• Broader generative backbones. The paper states the method is architecture-agnostic and reports results with more advanced DiT-based models such as WAN in supplementary material, leaving the systematic comparison outside the main text.
• Beyond depth and normals. GeoNeXt models two geometry targets; whether the next-frame formulation extends to other geometric quantities (the paper does not report such experiments) is an open question.
• Temporal geometry on video. The approach borrows temporal priors but is evaluated on single images; applying it to frame sequences for temporally consistent geometry is a natural extension the paper does not evaluate.
• Metric geometry and intrinsics. The paper notes that metric depth depends on camera intrinsics and that most approaches, including theirs, predict affine-invariant depth; conditioning on known intrinsics is left as an unaddressed limitation.
• Closing the gap with large discriminative models. GeoNeXt rivals but does not uniformly beat discriminative models trained on over 100× more data — for example DSINE (160K) reports 16.4 on NYUv2 normals versus GeoNeXt's 16.7 — so the remaining gap and how to close it is an open area.

Target Audience

Researchers and graduate students in computer vision working on 3D geometry estimation, diffusion-based generative modeling, or transfer learning from pretrained generative backbones. It is also relevant to practitioners who need a single, compact, fast model for depth and normal prediction in robotics, AR/VR, autonomous driving or content-creation pipelines, and to anyone interested in how video-model priors can be repurposed for non-video tasks. Readers without a background in diffusion models and geometry benchmarks will find the method sections demanding.

Authors’ abstract

Recent generative approaches to geometry estimation adapt pretrained image diffusion models and treat the task as image-conditioned generation. Leveraging off-the-shelf image diffusion models, they either (i) train task-specific geometry models (for depth and surface normal estimation) independently, losing the opportunity of exploring the intrinsic correlation of these geometric targets, or (ii) jointly fine-tune modified image diffusion backbones (e.g., altered self-attention), which typically demands substantial labeled data. To overcome these limitations in a principled fashion, we repurpose pretrained video generative models as a unified and data-efficient framework for geometry estimation, formulated innovatively as a next-frames prediction task. Our method, GeoNeXt, inherits naturally structured knowledge and richer priors from the video model, while further adapting them for joint modeling of images and geometry targets (image <-> geometry), enabling more data efficient and effective learning of geometry. Extensive experiments validate our method for zero-shot monocular depth and surface normal estimation across diverse datasets, outperforming both previous task-specific and unified generative competitors while using substantially less training data. Notably, our method rivals discriminative state-of-the-art approaches trained on over 100x more data and even standouts on several benchmarks.