Enhancing Generalization of Depth Estimation Foundation Model via Weakly-Supervised Adaptation with Regularization

AI summary

Overview

Research area: Computer vision — monocular depth estimation (MDE), specifically source-free / weakly-supervised domain adaptation of depth foundation models, with a focus on robustness under distribution shift and image corruption.

Technical level: Intermediate to Advanced. The paper assumes familiarity with teacher-student self-training, EMA updates, LoRA parameter-efficient fine-tuning, and depth normalization conventions (scale/shift ambiguity, median/MAD statistics).

Scope: The paper proposes WeSTAR, a parameter-efficient adaptation framework that combines dense self-training, semantic-aware hierarchical depth normalization, sparse pairwise ordinal weak labels, and LoRA weight regularization to improve Depth Anything v2 and MiDaS v3.1 on unseen realistic and corrupted depth benchmarks.

What This Paper Is About

Depth foundation models such as the Depth Anything series generalize well zero-shot, but their predictions still break down when the test images come from a different distribution — especially under corruption, bad weather, or night conditions. The authors ask whether — given only a small amount of unlabeled or weakly labeled data from the target domain — a pre-trained depth model can be adapted to perform better on that domain without losing its general knowledge. Their answer is WeSTAR, an adaptation recipe that keeps the model anchored near its original weights while nudging it with pseudo-labels and a few cheap relative-depth annotations.

Key Contributions

1. A low-rank regularized self-training framework with semantic-aware hierarchical normalization. The authors adapt depth foundation models to unseen distributions using LoRA adapters plus a weight-regularization loss that anchors updates to the pre-trained initialization, and they replace the content-agnostic grids of standard hierarchical depth normalization with semantic instance masks from SAM2.

2. Low-cost weak supervision via pairwise ordinal depth annotations. Instead of dense depth labels, the framework uses triplets of the form {p⁺, p⁻, l} where l ∈ {−1, 0, 1} encodes whether one pixel is farther, equal, or nearer than another, enforced with a margin ranking loss to fix local topological errors that pseudo-labels reinforce.

3. A structured pixel-pair sampling strategy that enforces transitivity. For each image the method runs 5 sampling iterations, picking an anchor pixel p_k plus a farther point p_k⁺ and a nearer point p_k⁻, yielding the two structured pairs (p_k⁺, p_k) and (p_k, p_k⁻) rather than independently random pairs.

4. Demonstrated robustness across realistic and corrupted out-of-distribution benchmarks. WeSTAR is evaluated on NYU-V2, KITTI, Sintel, DIODE, their corrupted counterparts NYU-C, KITTI-C, DIODE-C, Sintel-C (6 corruption types at severity 5), NuScenes night, and DrivingStereo (Sunny, Cloudy, Foggy, Rainy), on two backbones.

Main Findings

• Corruption severely degrades zero-shot depth models. On NYU-C the average over 6 corruption types drops from δ₁ 97.7 to 87.4 (a 10.5% decrease) and AbsRel rises from 4.6 to 10.6 (a 56.6% increase).

• WeSTAR leads on corrupted benchmarks (Depth Anything v2). Averaged over 6 corruptions: NYU-C 94.6 δ₁ / 7.1 AbsRel (vs. 87.4 / 10.6 for Source), KITTI-C 88.7 / 10.5 (vs. 83.2 / 13.2), Sintel-C 71.8 / 24.1 (vs. 60.3 / 30.6), DIODE-C 91.3 / 9.3 (vs. 88.0 / 11.1). The next strongest competitor on these benchmarks is SGRL (e.g., 92.4 / 8.4 on NYU-C, 66.5 / 29.9 on Sintel-C).

• WeSTAR helps most where the baseline has headroom (realistic datasets). On clean data the Depth Anything v2 baseline is already near ceiling on NYU (97.7 / 4.6) and DIODE (95.0 / 7.0), and WeSTAR reaches 98.2 / 4.3 and 95.2 / 6.5. The largest gains appear on Sintel (74.8 → 82.2 δ₁, 20.3 → 16.9 AbsRel), NuScenes (74.4 → 78.1, 18.5 → 16.2), and DrivingStereo Rainy (84.8 → 87.4, 12.1 → 10.6). One exception: on DrivingStereo Sunny, SGRL scores 83.4 / 13.5 versus WeSTAR's 82.8 / 13.6.

• The same pattern holds with a second backbone. With MiDaS v3.1, the Source model scores 43.2 / 40.5 on Sintel-C and 58.6 / 30.4 on Sintel; WeSTAR reaches 56.6 / 34.8 and 67.9 / 25.8. On DrivingStereo Sunny with this backbone, SGRL again leads (79.8 / 15.2) over WeSTAR (78.3 / 15.8). The authors excluded NYU and KITTI from the MiDaS evaluation because those clean datasets are part of MiDaS' training data.

• Dense self-training alone is not enough. Adding ST alone moves Sintel-C from 60.3 to 63.3 δ₁ and 30.6 to 27.6 AbsRel, but leaves Sintel essentially unchanged (74.8 → 74.9) and NuScenes unchanged (74.4 → 74.4). Weak supervision alone pushes Sintel δ₁ from 74.8 to 77.5 but worsens AbsRel from 20.3 to 24.1, which the authors attribute to its scale insensitivity. All three components together reach 71.8 / 24.1 on Sintel-C, 82.2 / 16.9 on Sintel, and 78.1 / 16.2 on NuScenes.

• LoRA beats full fine-tuning overall, and the encoder is where adaptation matters. Full parameter tuning gives 72.4 / 24.8 on Sintel-C, 80.8 / 19.5 on Sintel, 77.1 / 16.4 on NuScenes versus LoRA's 71.8 / 24.1, 82.2 / 16.9, 78.1 / 16.2. Tuning only the decoder leaves Sintel-C at 60.3 / 30.7 and NuScenes at 74.7 / 18.3, essentially no better than baseline.

• Semantic-aware normalization outperforms the alternatives. SA-HDN reaches 71.8 / 24.1 on Sintel-C and 82.2 / 16.9 on Sintel, against Global normalization's 70.9 / 24.2 and 80.8 / 16.4, and grid-based HDN's 68.9 / 28.1 and 78.6 / 22.4.

• WeSTAR is the most stable over training. The authors report that SGRL improves sharply at first but degrades over time (notably on DIODE Indoor-C), iBOT plateaus quickly, TTAC and FR gain inconsistently, and SSA is smooth but conservative, while WeSTAR improves consistently on both δ₁ and AbsRel.

Methodology in Plain English

The starting point is a publicly available depth foundation model. The authors freeze its original weights and inject small trainable low-rank matrices (LoRA, rank 8, alpha 16) into the encoder's attention layers, so only a tiny fraction of parameters move.

A teacher-student setup drives the adaptation. The teacher is an exponential moving average of the student (decay 0.996). Each training image is lightly and strongly augmented; the teacher predicts on the lightly augmented version to produce pseudo-labels, and the student predicts on the strongly augmented version. Because depth is only defined up to scale and shift, both prediction sets are normalized before comparison.

The normalization step is the paper's twist. Classic global normalization computes one median and one median-absolute-deviation over the whole image; hierarchical depth normalization does the same over fixed grids or depth bins. Both are blind to what is in the image, so a grid cell can straddle two objects with very different depths. The authors instead run SAM2 to produce instance masks on the fly and build a two-level context per pixel — the whole image plus the object instance the pixel belongs to — then average the normalized error across both contexts.

On top of this dense signal, a small number of ordinal pairwise labels is used. Each weak label says whether one pixel is farther, equal, or nearer than another, and a margin ranking loss (with a slack variable δ) penalizes violations. Because the pairs are built as anchor-plus-farther and anchor-plus-nearer, the supervision is transitive rather than arbitrary.

Finally, a weight-regularization term penalizes the magnitude of the LoRA product, keeping the adapted model close to its pre-trained prior. The total loss is a weighted sum: λ_st · L_st + λ_w · L_weak + λ_r · L_reg, with {λ_st, λ_w, λ_r} = {1.0, 0.001, 1.0} for the Depth Anything v2 backbone. Training uses AdamW (weight decay 0.0001), an initial learning rate of 0.1 scaled linearly with batch size as 0.1·(BS/256), cosine annealing, up to 100 epochs with early stopping after 30, batch size 4, on a single NVIDIA RTX 3090.

Why This Matters

Impact on research. The paper argues that self-training, which is well established for classification-based domain adaptation, does not transfer cleanly to depth regression: pseudo-labels are unreliable, the pre-trained geometric prior is already strong, and aggressive adaptation causes catastrophic forgetting. WeSTAR shows a way to get measurable gains from a handful of weak labels and a small amount of target data without access to source data — placing it in the source-free domain adaptation line of work. It also documents that corruptions are still a major weakness of current depth foundation models, which the authors frame as motivation for "principled" robustness methods rather than ad hoc fixes.

Real-world applications:

• Stereo conversion — converting 2D footage to 3D requires reliable relative depth on footage the model was never trained on.
• Augmented reality — placing virtual objects correctly requires metric-consistent structure in novel environments and lighting.
• 3D scene reconstruction — building geometry from photographs in conditions (night, rain, fog) that degrade zero-shot depth.
• Autonomous driving and robotics — the evaluation spans KITTI, NuScenes night, and DrivingStereo weather variants, which are exactly the sensor and weather conditions where depth models were shown to degrade.

Industry relevance. The setting is practical: no source data, no dense labels, a single RTX 3090, and LoRA-scale memory needs. That makes domain-specific adaptation feasible for teams that can collect a small amount of unlabeled or cheaply annotated data in a deployment environment (a particular camera rig, weather regime, or site) but cannot retrain a foundation model. The reliance on SAM2 for masks also means the method depends on another external foundation model being available at adaptation time.

Future Directions

• Reducing dependence on the external segmentation model. SA-HDN requires SAM2 to generate instance masks on the fly; whether the gains survive with a weaker or no segmentation model, and what the compute overhead is, is not addressed in the reported content.

• Closing the remaining gap on corrupted and synthetic scenes. Sintel-C remains the hardest benchmark, with WeSTAR at 71.8 δ₁ / 24.1 AbsRel on the Depth Anything v2 backbone and 56.6 / 34.8 on MiDaS v3.1.

• Understanding when weak supervision hurts. Weak supervision alone improved δ₁ but degraded AbsRel on Sintel (20.3 → 24.1) due to scale insensitivity; characterizing when ordinal constraints induce distortion versus correction would help practitioners decide how much weak labeling to collect.

• Characterizing the amount of data needed. The paper states it uses "a small set" of unlabeled or weakly labeled samples and warns that overly aggressive adaptation risks over-specialization, but the reported content does not include a data-scaling analysis of how much target data or how many pixel pairs are sufficient, nor a sensitivity analysis of λ_w or the ranking margin δ.

Target Audience

Researchers and engineers working on monocular depth estimation, domain adaptation, or foundation-model fine-tuning. It will be most useful to readers already comfortable with self-training and parameter-efficient tuning who want a concrete, low-resource recipe for adapting a depth model to a new domain, and to practitioners evaluating whether a depth foundation model can be trusted under corruption, night, or adverse weather — plus anyone interested in how weak ordinal labels can supplement pseudo-label-based adaptation in a regression setting.

Authors’ abstract

The emergence of foundation models has substantially advanced zero-shot generalization in monocular depth estimation (MDE), as exemplified by the Depth Anything series. However, given access to some data from downstream tasks, a natural question arises: can the performance of these models be further improved? To this end, we propose WeSTAR, a parameter-efficient framework that performs Weakly supervised Self-Training Adaptation with Regularization, designed to enhance the robustness of MDE foundation models in unseen and diverse domains. We first adopt a dense self-training objective as the primary source of structural self-supervision. To further improve robustness, we introduce semantically-aware hierarchical normalization, which exploits instance-level segmentation maps to perform more stable and multi-scale structural normalization. Beyond dense supervision, we introduce a cost-efficient weak supervision in the form of pairwise ordinal depth annotations to further guide the adaptation process, which enforces informative ordinal constraints to mitigate local topological errors. Finally, a weight regularization loss is employed to anchor the LoRA updates, ensuring training stability and preserving the model's generalizable knowledge. Extensive experiments on both realistic and corrupted out-of-distribution datasets under diverse and challenging scenarios demonstrate that WeSTAR consistently improves generalization and achieves state-of-the-art performance across a wide range of benchmarks.