ReDepth Anything: Test-Time Depth Refinement via Self-Supervised Re-lighting

AI summary

Overview

Research area: Monocular depth estimation, test-time adaptation, and generative diffusion priors for 3D vision.

Technical level: Intermediate — the reader needs familiarity with diffusion models, score distillation, and depth estimation architectures, though the paper explains its core ideas clearly.

Scope: A test-time, label-free refinement framework that uses re-lighting and a 2D diffusion prior to correct depth predictions from feed-forward foundation models such as Depth Anything V2 and Depth Anything 3.

What This Paper Is About

Monocular depth foundation models like Depth Anything V2 generalize broadly but still produce wrong or overly smooth geometry on images far from their training distribution, such as an unusual animal, a textured object, or a cluttered street scene. The authors ask whether a model can be corrected at test time, for a single image, without any ground-truth depth, and without retraining. Their answer is to re-light the predicted 3D shape under random lighting and let a large pretrained image diffusion model judge whether the resulting shading looks realistic, using that judgment as the training signal.

Key Contributions

1. A test-time optimization framework (Re-Depth Anything) that adapts a pretrained feed-forward depth model to a single real-world image using only a 2D diffusion prior and no labeled depth. Reported on top of both DA-V2 and DA3, it produces consistent gains and reaches state-of-the-art on CO3D and ETH3D.

2. A differentiable single-image re-lighting module built on Blinn-Phong shading that links the predicted depth map to a re-illuminated version of the input image. This replaces classical photometric reconstruction for self-supervision and produces the rendering needed for the SDS loss.

3. A targeted optimization scheme that jointly updates only the intermediate feature embeddings fed to the DPT decoder and the decoder's weights, while keeping the ViT encoder frozen. Directly optimizing the depth tensor or fine-tuning the whole network is shown to fail through noise artifacts or geometry collapse.

4. A demonstration of generality across backbones by applying the same pipeline to Depth Anything 3 (a depth-predicting, ViT-L model rather than a disparity-predicting ViT-S model) and improving it as well.

Main Findings

• Consistent gains over DA-V2 on all nine metrics: Across CO3D, KITTI, and ETH3D, the method improves δ1/δ2/δ3, AbsRel, RMSE, log10, RMSE log, SI log, and SqRel — with up to 11.4% relative error reduction. KITTI SI log and RMSE log drop by 8.5%; ETH3D AbsRel drops by 8.4% and δ2 improves by 21.1%.

• State-of-the-art detail reconstruction with DA3: Applied to DA3MONO-LARGE, the method reduces CO3D AbsRel by 4.83%, SqRel by 7.39%, and normal MSE by 14.65%. ETH3D gains are smaller but positive (0.54%, 0.71%, 3.74%).

• Optimization target is decisive: Fine-tuning the entire DA-V2 network collapses the geometry even at a very low learning rate (2e-6); directly optimizing the depth tensor injects noise artifacts. Updating embeddings plus DPT weights preserves structure while adding detail. A two-stage variant that first optimizes embeddings then the decoder is close but slightly worse than the joint version.

• Ensembling helps, then saturates: A single run gives a 1.58% SI log improvement over DA-V2, three runs raise this to 2.22%, and ten runs give only 2.24%. The authors nonetheless report results with ten runs.

• Generative self-supervision beats classical shape-from-shading: Classical SfS fails visibly on the paper's test cases — for example, discoloration on a leather ball yields noisy, spurious normals — because it assumes constant albedo. The re-lighting formulation avoids that assumption and does not require pixel-perfect appearance reconstruction.

• Failure modes are limited and specific: Occasional small hallucinated edges (e.g., a sticker on a truck), geometry bleeding into the sky, and over-smoothing of fine detail in dark regions such as trees in KITTI. On CO3D the distribution of per-sample improvements is clearly skewed toward gains; on KITTI and ETH3D gains are more frequent and larger than losses, with a heavier positive tail.

• Cost profile: Each optimization run takes roughly 80 seconds on a single NVIDIA RTX 5000, with 1000 iterations of AdamW (learning rate 1e-3 for embeddings, 2e-6 for DPT weights) and Stable Diffusion v1.5 as the prior.

Methodology in Plain English

The pipeline starts with a pretrained depth model producing an initial depth (or disparity) map for one input image. Rather than trying to reconstruct the image photometrically — an ill-posed problem that requires estimating exact albedo, lighting, and materials — the authors augment the input. They treat the input image as a rough proxy for diffuse color, convert the depth map into a normal map via spatial gradients, and then re-shade the image under randomly sampled light directions and Blinn-Phong material parameters. The result is a set of artificially re-lit images whose shading is determined entirely by the predicted geometry.

A pretrained text-to-image diffusion model (Stable Diffusion v1.5, conditioned on a BLIP-2 caption of the original image) then scores how realistic each re-lit image looks, via the Score Distillation Sampling loss. Gradients flow back through the differentiable shading step into the depth model. Because naively optimizing the depth pixels or the full network misbehaves, the authors restrict updates to the intermediate feature embeddings produced by the frozen ViT encoder and the weights of the DPT decoder head, adding a smoothness regularizer on disparity gradients. Since the SDS loss is stochastic, the authors run the optimization ten times with different random seeds and average the resulting disparity maps.

Why This Matters

The work reframes how self-supervision for geometry can be obtained: instead of demanding photometric reconstruction against a real image, it uses a generative model to critique plausible shading, which relaxes the need for a physically accurate renderer. This is a small but meaningful shift in how diffusion priors are used for 3D tasks — as a scoring function for augmented views rather than a generator of novel ones.

Real-world applications include:

• Robotics and autonomous navigation, where an out-of-distribution scene (unusual terrain, novel obstacles) can degrade a depth network without warning, and a per-image correction pass can recover usable geometry.
• Augmented and virtual reality, where inserting virtual objects requires accurate surface normals and fine geometric detail around object boundaries, an area the paper shows clear improvements in.
• 3D content capture and reconstruction from casual single photographs, particularly for objects and scenes with complex shading, where the method adds missing detail and removes noise on flat regions.
• Consumer photography and computational imaging, where depth-aware effects such as portrait relighting or background blur benefit directly from refined normals and cleaner depth edges.

Industry relevance centers on deployment: the method is model-agnostic, applies on top of existing foundation models, requires no retraining or labeled data, and improves metrics that downstream 3D pipelines actually consume, notably normal accuracy.

Future Directions

• Reducing latency. Ten runs at ~80 seconds each is impractical for real-time use. The ensembling curve suggests most of the benefit arrives by three runs, but a principled single-pass variance reduction or a distilled refinement network is an open problem.
• Resolving the remaining failure modes. Hallucinated fine structure, geometry extending into the sky, and over-smoothing in dark regions point to weaknesses in the shading-based signal. Explicit handling of low-light regions, sky segmentation, or confidence-aware weighting could address these.
• Extending beyond static single images. The framework assumes one view with no temporal information. Combining it with video consistency cues or sparse multi-view input could strengthen the self-supervisory signal and stabilize optimization.
• Generalizing the prior and its use. The experiments rely on Stable Diffusion v1.5 and a single BLIP-2 caption. Whether stronger or task-specific generative priors yield larger gains, and whether the caption itself limits performance on unusual objects, remain untested.
• A unified benchmark protocol. The supplemental material notes that comparing depth methods is complicated by mismatched depth representations (relative disparity, relative depth, absolute depth) and evaluation protocols; the field would benefit from standardization, which this work motivates implicitly.

Target Audience

Researchers and graduate students in computer vision working on monocular depth estimation, test-time adaptation, or diffusion-based 3D reconstruction will find the core contribution directly relevant. Practitioners who deploy depth foundation models in robotics, AR/VR, or 3D capture will benefit from the practical insights, particularly the ablation showing which components to fine-tune and which to freeze, and the honest accounting of runtime and failure cases. Readers without a background in score distillation will need to consult the referenced DreamFusion and SDS literature to follow the technical details fully.

Authors’ abstract

Monocular depth estimation remains challenging, as foundation models such as Depth Anything V2 (DA-V2) struggle with real-world images that are far from the training distribution. We introduce Re-Depth Anything, a test-time self-supervision framework that bridges this domain gap by fusing foundation models with the powerful priors of large-scale 2D diffusion models. Our method performs label-free refinement directly on the input image by re-lighting the predicted depth map and augmenting the input. This re-synthesis method replaces classical photometric reconstruction by leveraging shape from shading (SfS) cues in a new, generative context with Score Distillation Sampling (SDS). To prevent optimization collapse, our framework updates only intermediate embeddings and the decoder's weights, rather than optimizing the depth tensor directly or fine-tuning the full model. Across diverse benchmarks, Re-Depth Anything yields substantial gains in depth accuracy and realism over DA-V2, and applied on top of Depth Anything 3 (DA3) achieves state-of-the-art results, showcasing new avenues for self-supervision by geometric reasoning.