Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation

AI summary

Overview

• Research area: Computer vision, specifically monocular depth estimation with generative diffusion models, with connections to computational photography, scene reconstruction, and graphics.
• Technical level: Intermediate. The paper builds directly on the Marigold line of work and assumes familiarity with diffusion models, VAEs, LoRA/quantization, and standard depth metrics such as AbsRel and δ1.
• Scope: A single paper describing a two-stage fine-tuning protocol that turns an open-source image-editing diffusion transformer (Qwen-Image-Edit-2509) into a monocular depth estimator, plus evaluations on depth, edge sharpness, and transfer to other dense regression tasks.

What This Paper Is About

Monocular depth estimation — recovering a depth value for every pixel from a single image — remains hard to generalize out of distribution and tends to produce blurry, oversmoothed depth maps that lose thin structures and fine detail. The authors revisit the Marigold recipe for repurposing pretrained generative models into depth estimators, and update it for modern diffusion transformer (DiT) backbones, specifically an image-editing model rather than a text-to-image model. Their goal is a depth predictor that is sharper, generalizes better, and is still cheap enough to train on a single 32GB GPU in under a week.

Key Contributions

1. Marigold V2 protocol. A lightweight two-stage fine-tuning protocol that converts an image-editing DiT into a monocular depth estimator (or another dense modality regressor), using 4-bit quantization with QLoRA so that training fits on a single consumer 32GB GPU with a modestly sized dataset over a few days.
2. iREPA-depth. A variant of representation alignment applied in Stage 1 that aligns the model's internal representations toward semantic features extracted from the ground-truth depth map using a frozen DINOv3 encoder, rather than from the RGB input as in prior alignment work.
3. SinkLoss. A novel Sinkhorn–Knopp matching-based objective used in Stage 2 that tiles the image into non-overlapping blocks and enforces a soft one-to-one assignment between predicted and ground-truth depths within each block, improving edge sharpness and reducing flying pixels while preserving detail such as fur, hair, and thin structures.
4. Transfer to other dense tasks. The recipe is reported to adapt to depth completion, see-through depth, surface normal estimation, and intrinsic image decomposition, with state-of-the-art results claimed on each (quantitative tables for these tasks are not included in the truncated content).

Main Findings

• Best zero-shot accuracy among comparable-data methods. On the five evaluation datasets, Marigold V2 (depth, ours, trained on 74K images) reports NYUv2 AbsRel 3.6 / δ1 98.0, KITTI AbsRel 5.4 / δ1 97.4, ETH3D AbsRel 2.8 / δ1 99.2, ScanNet AbsRel 3.7 / δ1 97.9, and DIODE AbsRel 5.2 / δ1 97.1. The paper states this is the best zero-shot performance across all evaluated datasets among methods trained on comparable data.
• Large gains on the stated benchmarks. The abstract reports a 16–26% improvement in AbsRel over the previous best on KITTI and ETH3D. On ETH3D specifically, the best model reaches AbsRel 2.8 versus the strongest baseline result of 3.8.
• Better edge preservation. On the HyperSim test set using Soft Edge Error (SEE3, SEE5, SEE7), Marigold V2 scores 0.352, 0.333, 0.320, compared with PPD at 0.404, 0.385, 0.371 and InfiniDepth at 0.470, 0.451, 0.436.
• iREPA-depth beats iREPA on RGB in Stage 1 ablations (30K steps). Unregularized baseline: NYUv2 4.53 / 97.60, KITTI 7.26 / 94.94, ETH3D 4.25 / 98.25, ScanNet 4.59 / 97.41, DIODE 6.82 / 95.81. With pixel and gradient losses plus iREPA on RGB: 4.50 / 97.81, 6.89 / 96.22, 3.64 / 98.86, 4.38 / 97.65, 5.72 / 96.54. With iREPA-depth instead: 4.36 / 98.01, 6.72 / 96.38, 3.62 / 98.90, 4.22 / 97.91, 5.55 / 96.82.
• iREPA's quantitative effect shrinks with longer training but visual gains persist. At 160K steps, iREPA on RGB gives NYUv2 3.62 / 98.01, KITTI 5.27 / 97.39, ETH3D 2.84 / 99.08, ScanNet 3.84 / 97.88, DIODE 5.16 / 97.07; iREPA-depth gives 3.68 / 98.03, 5.30 / 97.43, 2.68 / 99.18, 3.81 / 97.82, 5.02 / 97.02. The paper concludes iREPA-depth mainly accelerates convergence and retains a qualitative benefit.
• Log-depth is the best target parameterization. Averaged over test sets: Linear Depth 5.04 / 97.10, Disparity 5.28 / 97.15, Log Depth 4.72 / 97.71. The authors note that an L1 penalty on log-depth coincides with per-pixel AbsRel error to first order.
• SinkLoss transfers across backbones. AbsRel averaged over five test datasets and SEE on HyperSim: Stable Diffusion V1.5 goes from 7.51 to 7.12 AbsRel and from 0.553 / 0.531 / 0.514 to 0.485 / 0.464 / 0.449 SEE3/5/7; FLUX.2 klein goes from 4.88 to 4.82 and from 0.491 / 0.472 / 0.457 to 0.377 / 0.359 / 0.345; Qwen-Image-Edit-2509 goes from 4.10 to 4.12 and from 0.449 / 0.429 / 0.414 to 0.352 / 0.333 / 0.320.
• Pixel-space losses mainly help δ1, not AbsRel. Adding image-space supervision on top of latent MSE has limited effect on AbsRel but consistently improves δ1 on all datasets.
• LPIPS underperforms iREPA. A direct LPIPS perceptual loss between prediction and ground truth does not deliver the same improvement in visually dense regions as iREPA-depth.
• Inference stays single-step. The deployed model requires just one forward pass through the VAE and the DiT.

Methodology in Plain English

The authors start from Qwen-Image-Edit-2509, a large pretrained image-editing diffusion transformer, and teach it to output depth instead of edited images. To keep it affordable, they quantize the pretrained weights to 4-bit precision and train only rank-128 QLoRA adapters, with batch size 1 and gradient clipping, on a deliberately compact mix of HyperSim and vKITTI (HyperSim samples filtered to those with at most 0.1% invalid depth pixels; vKITTI at 1216×352, HyperSim at 768×512; batches sampled 90% from HyperSim and 10% from vKITTI).

The depth target is converted to an affine-invariant normalized log-depth and replicated across three channels so the RGB image-editing backbone can consume it. Because the timestep is fixed at t = 0.5, the rectified-flow formulation collapses into a direct latent regression: the model predicts the difference between the RGB latent and the depth latent, and inference is a single pass.

Training happens in two stages. Stage 1 combines the latent regression loss with pixel-space L1 reconstruction and gradient losses, and adds iREPA alignment that pulls the DiT's internal features toward DINOv3 features computed from the ground-truth depth map. Stage 2 continues fine-tuning with the SinkLoss and unfreezes the VAE decoder. SinkLoss cuts each image into non-overlapping K×K blocks (K = 5), builds a cost matrix of absolute differences between predicted and ground-truth depths, handles invalid pixels by penalizing their rows and columns (B = 10⁶), and solves an entropy-regularized optimal transport problem with Sinkhorn–Knopp iterations (τ = 0.1, 5 iterations) to obtain a soft one-to-one assignment. The rationale is that thin and transparent objects have noisy ground truth even in synthetic data such as HyperSim, where V-Ray's quasi-Monte Carlo sampling randomly assigns foreground or background depth to those pixels; requiring only that the block contain the same set of depth values up to permutation, rather than exactly at the same pixels, produces cleaner depth maps.

Ablations run 30,000 steps (about one day each) on a single 32GB GPU; Stage 2 adds 30,000 more steps (about one day). The final Stage-1 model trains for 160,000 steps and takes slightly more than five days. Loss weights are λlatent = 1.0, λpix = 1.0, λgrad = 5.0, λiREPA = 0.2 for Stage 1; λiREPA = 0.2 and λSinkLoss = 1.0 for Stage 2.

Why This Matters

Impact on research. The paper argues that the field's strongest discriminative depth models (MoGe, π³, Depth Anything) are limited by scarce ground-truth depth data — million-scale versus the billion-scale data available to generative models — and by sensor noise and poor handling of non-Lambertian, transparent, or reflective surfaces. It shows that a generative DiT can be repurposed for dense regression at a cost accessible to individual practitioners and small labs, rather than the tens of GPUs needed to train from scratch or the 96 GPU-days reported for DICEPTION and the 8 GPUs used by Lotus-2. The SinkLoss idea also addresses a general problem: how to supervise dense regression when the ground truth itself is noisy or ambiguous.

Real-world applications (as motivated by the paper):

• Image-based rendering and novel view synthesis.
• Bokeh simulation and computational refocusing.
• Portrait relighting and matting.
• Single-image 3D reconstruction, scene lifting, object insertion, augmented reality compositing, and 3D content creation from casual photographs.

Industry relevance. The work comes from HUAWEI Bayer Lab with EPFL and the University of Bologna, and is published in ACM TOG. The emphasis on a single 32GB GPU and less than a week of training, plus single-step inference with quantized weights, targets practical deployment constraints rather than benchmark-only performance. The paper also reports the recipe transferring to depth completion, see-through depth, surface normals, and intrinsic image decomposition, which matters for photography and graphics pipelines.

Future Directions

• Extending and quantifying the transfer to further dense regression tasks beyond the ones named (depth completion, see-through depth, surface normals, intrinsic image decomposition), which the truncated content does not report numerically.
• Testing whether the SinkLoss-style permutation-tolerant supervision generalizes to other settings where ground truth is ambiguous or noisy, given the paper shows it transfers to Stable Diffusion V1.5 and FLUX.2 klein.
• Closing the remaining gap between quantitative and qualitative effects of iREPA-depth: at 160K steps its AbsRel and δ1 advantage largely fades, yet the visual benefit persists, leaving open how to preserve or exploit that gain.
• Revisiting the trade-off between the single-step, VAE-based design and alternatives such as pixel-space diffusion (Pixel-Perfect Depth) and implicit-field methods (InfiniDepth) that make different choices about detail and resolution.

Target Audience

Researchers and practitioners in computer vision and graphics working on monocular depth estimation, generative model repurposing, or dense prediction; engineers who need strong depth predictions under tight compute budgets; and readers interested in optimal-transport-based losses for noisy supervision or in representation-alignment techniques for diffusion transformers.

Authors’ abstract

Monocular depth estimation is a ubiquitous yet highly ill-posed computer vision task, with downstream applications in scene reconstruction, computational photography, and robotics, among others. Despite the field's maturity, recent models still struggle to generalize to out-of-distribution inputs and to produce sharp and detailed depth maps. In this paper, we revisit Marigold, a set of techniques for repurposing modern image generation and editing models, powered by the diffusion transformer (DiT) architecture, into state-of-the-art monocular depth estimators. Our recipes target single-step inference from pretrained multi-step flow-matching models, with quantization where needed, preserving model capacity while remaining cheap to run. We analyze the artifacts of naive training and identify two effective remedies: aligning the model's internal representations with semantic features extracted from ground-truth, and adopting a 2-stage fine-tuning protocol built around a novel Sinkhorn-based loss. The results are crisper, cleaner depth maps that generalize well out-of-distribution, with 16-26% improvement in AbsRel over the previous best on KITTI and ETH3D. Qualitatively, our model resolves fur, foliage, and hair-thin edges that have eluded prior models. Furthermore, Marigold V2 achieves state-of-the-art results when applied to other dense regression tasks, such as surface normals estimation and intrinsic image decomposition. Project website: https://hf.co/spaces/huawei-bayerlab/marigold-v2-web