GeoDiff: Geometry-Guided Diffusion for Metric Depth Estimation

AI summary

Overview

• Research area: Computer vision — monocular and stereo metric depth estimation, diffusion models, inverse problems.
• Technical level: Advanced (assumes familiarity with diffusion models, score matching, and stereo geometry).
• Scope: The paper proposes GeoDiff, a training-free framework that adds stereo geometric guidance to a pretrained diffusion-based monocular depth model to produce metric depth in meters.

What This Paper Is About

Diffusion-based monocular depth estimation (DB-MDE) models such as Marigold predict relative depth that is invariant to scale and shift, so a single image alone cannot yield absolute distances in meters. GeoDiff reframes depth estimation as an inverse problem: given a stereo pair (or two views with known poses), it uses a pretrained latent diffusion model conditioned on the left RGB image plus stereo-based geometric constraints to learn the per-scene scale and shift needed to recover metric depth. The goal is accurate metric depth without retraining the underlying depth model.

Key Contributions

1. A framework that combines diffusion-based MDE priors with stereo settings to achieve metric depth estimation, rather than only affine-invariant relative depth.
2. An inverse-problem-based approach that introduces a plug-and-play module, described as seamlessly integrating with any pretrained diffusion-based depth model that uses iterative updates.
3. Extensive zero-shot experiments across indoor, outdoor, and challenging translucent/specular scenes, comparing against monocular, metric-depth, and stereo baselines.
4. The paper states it is the first training-free approach combining monocular and stereo information to solve metric depth estimation.

Main Findings

• Outperforms baselines on KITTI-Stereo (metric, no alignment): AbsRel 0.07, δ1 0.91, RMSE 3.94, versus Marigold (aligned) 0.13 / 0.85 / 4.81 and DepthPro (metric) 0.16 / 0.81 / 4.43.
• Best aligned numbers on KITTI-Stereo: 0.09 / 0.91 / 3.72, ahead of MiDas (0.63 / 0.24 / 11.72), Marigold (0.13 / 0.85 / 4.81), and GeoWizard (0.18 / 0.75 / 5.7).
• Booster (specular/transparent objects) metric: 0.11 AbsRel, 0.81 δ1, 0.18 RMSE, versus DepthPro 0.34 / 0.51 / 0.67, and ZoeDepth and UniDepth which the paper lists with AbsRel values of 7.37 and 4.55 respectively.
• Booster aligned: 0.04 / 0.98 / 0.06 — matching Marigold's aligned 0.04 / 0.98 / 0.06 and edging GeoWizard's 0.04 / 0.96 / 0.08.
• Middlebury metric: 0.11 AbsRel, 0.85 δ1, 2.31 RMSE; the paper notes it leads in AbsRel and δ1 but falls short on RMSE among the compared methods. The aligned result is 0.11 / 0.87 / 1.41.
• Stereo comparison on Booster (Table 2): GeoDiff metric gets 0.11 / 0.81 / 0.18, on par with IGEV-Stereo on AbsRel and δ1 (0.10 / 0.94) while beating it on RMSE (0.31). RAFT-Stereo is reported at 5.4 AbsRel and 18.89 RMSE.
• Arbitrary two-view setting on Tanks and Temples: metric 0.47 / 0.38 / 0.62 versus Dust3r's 0.51 / 0.35 / 0.56; with alignment 0.08 / 0.92 / 0.18 versus Dust3r's 0.07 / 0.93 / 0.17.
• Ablation — reprojection loss placement matters: optimizing reprojection loss directly in pixel space over 50 iterations produces noisy depth, while using it as guidance inside diffusion sampling yields sharper results with no explicit smoothness regularizer.
• Ablation — the loss must affect the latents: restricting the reprojection loss to learning only scale and shift gives AbsRel 0.14 and δ1 0.80 on KITTI-Stereo, versus 0.07 and 0.91 for the full model.
• Implementation specifics: built on Marigold's public pretrained model with weights frozen, learning rate 1e-2, an ensemble of 10 depth samples for all metrics, γ = 1e-2 regularization, and a single NVIDIA RTX A6000 GPU.

Methodology in Plain English

The method starts from an off-the-shelf diffusion depth model that already knows how to turn an RGB image into a plausible relative depth map. Because that map has no absolute scale, the authors add two things.

First, a learnable linear transformation — a scale and a shift, both passed through a softplus so they stay positive — that converts the relative depth into metric depth. These two parameters are optimized separately for each scene.

Second, stereo geometry as a training signal. During the iterative diffusion sampling, the one-step depth prediction is scaled to metric units, then used to warp the right image back into the left camera's viewpoint using the known camera intrinsics and relative pose. If the depth is correct, the re-rendered left image should match the actual left image, so the difference (a combination of SSIM and L1 losses) becomes a geometric loss. Its gradient is added to the diffusion model's own score function, nudging the sampling trajectory toward depth that is both visually plausible and geometrically consistent. The scale and shift parameters are updated by the same gradient. Backward warping is used instead of forward warping to avoid holes and extra computation.

Two practical additions make this stable. A global scale hyperparameter is pre-selected per scene, because otherwise the optimizer can keep pushing depths farther away to reduce the warping error and derail the sampling trajectory. An L2 regularizer on the scale and shift parameters further stabilizes optimization. The authors argue the diffusion process itself acts as an implicit regularizer, replacing the smoothness constraints that pixel-space reprojection optimization normally needs.

For scenes without calibrated stereo, the framework accepts any two views with known relative pose, and camera intrinsics and extrinsics can be estimated using foundation models for dense 3D reconstruction.

Why This Matters

• Research impact: The paper claims the first training-free route from monocular diffusion priors to metric depth using stereo guidance, positioning depth estimation as an inverse problem in latent space rather than a task requiring a new trained model.
• Real-world applications:
  • Autonomous driving, where absolute distances matter and transparent or reflective surfaces are common.
  • 3D reconstruction and point-cloud generation at true scale, unlike methods such as Dust3r that recover geometry only up to an unknown scale factor.
  • AI-generated content pipelines that need consistent spatial structure.
  • Spatial measurement and scene understanding from two-view captures, including handheld or uncalibrated image pairs when poses are estimated.
• Industry relevance: The approach is plug-and-play and requires no retraining or domain-specific data, which lowers the cost of deploying metric depth. The stated trade-off is slow inference from iterative diffusion sampling, which constrains latency-sensitive deployment.

Future Directions

• Improving robustness by substituting a stronger or more expressive pretrained MDE prior, since the method's ceiling depends on that prior.
• Accelerating inference through faster sampling techniques or lighter diffusion architectures, addressing the admitted slow speed of iterative diffusion sampling.
• Generalizing the geometric guidance beyond two views, since the current work focuses specifically on the two-view paradigm while multi-view stereo methods exist.
• Reducing reliance on estimated camera parameters for in-the-wild pairs, where intrinsics and extrinsics must come from separate foundation models.

Target Audience

Researchers and practitioners working on depth estimation, diffusion-based dense prediction, or inverse problems in latent space, plus engineers building 3D reconstruction, autonomous driving, and spatial-AI systems who need metric depth without training a new model. The paper is best suited to readers already comfortable with diffusion sampling, score functions, and stereo geometry.

Authors’ abstract

We introduce a novel framework for metric depth estimation that enhances pretrained diffusion-based monocular depth estimation (DB-MDE) models with stereo vision guidance. While existing DB-MDE methods excel at predicting relative depth, estimating absolute metric depth remains challenging due to scale ambiguities in single-image scenarios. To address this, we reframe depth estimation as an inverse problem, leveraging pretrained latent diffusion models (LDMs) conditioned on RGB images, combined with stereo-based geometric constraints, to learn scale and shift for accurate depth recovery. Our training-free solution seamlessly integrates into existing DB-MDE frameworks and generalizes across indoor, outdoor, and complex environments. Extensive experiments demonstrate that our approach matches or surpasses state-of-the-art methods, particularly in challenging scenarios involving translucent and specular surfaces, all without requiring retraining.