RoSAMDepth: Robust Self-supervised Depth Estimation Leveraging Segment Anything Model

AI summary

Overview

Research area: Computer vision — self-supervised monocular depth estimation, with a focus on robustness under adverse conditions (night, rain) and on injecting object-level priors from the Segment Anything Model (SAM).

Technical level: Intermediate. The core ideas (photometric self-supervision, smoothness losses, teacher–student distillation) assume some familiarity with depth-estimation literature, but the paper's three main components are conceptually describable without heavy mathematics.

Scope: The paper proposes RoSAMDepth, a training framework that uses pre-generated SAM masks as object-level priors to make self-supervised depth estimation sharper at object boundaries and more stable across day, night, and rain, evaluated on nuScenes and Oxford RobotCar.

What This Paper Is About

Self-supervised depth networks learn by reconstructing one camera view from another, which works well in clear daytime scenes but degrades at night or in rain, where photometric consistency breaks down. Existing robust methods treat degraded image regions uniformly, ignoring that depth behaves differently for different objects, so predictions blur across object boundaries and drift within objects. RoSAMDepth's goal is to bring object-level structure — obtained from SAM — into both the feature representation and the supervision signal of a robust self-supervised depth model.

Key Contributions

1. First integration of SAM into robust self-supervised depth estimation, using only pre-generated SAM masks during training as object-level priors (SAM is run offline and is not part of inference).

2. Segment-guided representation contrasting (SRC), which builds a per-segment feature prototype at each decoder scale and applies an InfoNCE-based contrastive loss, so pixels of the same SAM segment cluster together in feature space — deliberately at the feature level rather than as a direct constraint on predicted depth.

3. Adaptive regional outlier masking (AROM) with a regional Gaussian likelihood loss, which models inverse teacher depth within each SAM segment as a Gaussian and suppresses intra-object outliers using a threshold that adapts to each object's depth variance, replacing conventional boundary-aware smoothness.

4. Object-level reliability estimation (ORE), which aggregates pixel-wise photometric error into per-segment errors and down-weights objects whose error deviates from the image mean, addressing the paper's finding that pixel-level reliability indicators fail to flag a wholly mispredicted object.

Main Findings

• nuScenes, single-frame: Ours-Single reaches AbsRel 0.1172 / SqRel 1.357 / RMSE 6.121 / δ₁ 86.13 on day-clear, 0.1742 / 1.865 / 7.966 / 72.52 at night, and 0.1298 / 1.612 / 6.801 / 82.30 on day-rain, versus Syn2Real-Depth's 0.1239 / 1.454 / 6.214 / 85.66, 0.1792 / 1.896 / 7.949 / 71.08, and 0.1331 / 1.616 / 6.926 / 81.68.

• nuScenes, multi-frame: Ours-Multi reaches 0.1131 / 1.281 / 5.992 / 86.95 (day-clear), 0.1667 / 1.775 / 7.729 / 73.95 (night), and 0.1236 / 1.521 / 6.613 / 83.53 (day-rain). Nighttime RMSE (7.729) is marginally above Syn2Real-Depth's 7.689, one of the few metrics where the method is not best.

• Average gains on nuScenes: Relative improvements of 2.8% (AbsRel), 2.7% (SqRel), 0.6% (RMSE), and 0.9% (δ₁) over Syn2Real-Depth, averaged over all conditions and both single- and multi-frame settings.

• Oxford RobotCar, single-frame: Day AbsRel 0.1006 / SqRel 0.595 / RMSE 3.022 / δ₁ 88.90 and night 0.1066 / 0.643 / 3.291 / 87.03, versus Syn2Real-Depth's 0.1063 / 0.646 / 3.171 / 88.29 and 0.1103 / 0.693 / 3.455 / 86.15 — relative improvements of 4.4%, 7.6%, 4.7%, and 0.9% on AbsRel, SqRel, RMSE, and δ₁.

• Component ablation (nuScenes night / day-rain): Adding components monotonically helps — SRC alone gives 0.1725 / 73.21 night and 0.1270 / 82.82 rain; SRC+AROM gives 0.1699 / 73.67 and 0.1259 / 83.17; the full four-component model gives 0.1667 / 73.95 and 0.1236 / 83.53. Removing either AROM or the regional Gaussian loss alone degrades results, showing they are complementary.

• Smoothness formulation matters: AROM + regional Gaussian loss (0.1699 AbsRel / 73.67 δ₁ at night) beats both image-edge-aware smoothness (0.1735 / 73.15) and a SAM-boundary-aware variant that swaps the image for SAM masks in the conventional loss (0.1732 / 73.17).

• SRC must act on features, not depth: Applying the contrastive objective directly to predicted depth Dₛ yields 0.1694 / 73.68 at night and 0.1261 / 83.05 in rain, worse than the feature-level version (0.1667 / 73.95 and 0.1236 / 83.53), attributed to SAM over-segmenting single objects.

• Qualitative behavior: Compared with md4all and Syn2Real-Depth, the method preserves sharper object boundaries in low-contrast rain, suppresses glare/reflection artifacts at night, and better delineates thin structures such as poles and street signs.

Methodology in Plain English

The authors build on the Syn2Real-Depth synthetic-to-real pipeline. A teacher depth network, already trained on synthetic data, is frozen; a student network is initialized from it and fine-tuned for 10 epochs on real data. SAM (the off-the-shelf ViT-H model, run in "segment-everything" mode) produces masks for every training image in advance, and overlapping masks are collapsed so each pixel belongs to the largest mask covering it — the supplementary material merges small masks (under 10⁵ pixels, roughly 200×500) into larger parents when overlap exceeds 0.75.

Three mechanisms then use those masks. First, at each of three feature scales in the depth decoder, the network averages features within each segment to form a prototype, and a contrastive loss pulls each pixel's feature toward its own prototype and away from others — teaching the network object structure without forcing depth discontinuities at every SAM boundary. Second, for supervision, the teacher's inverse depth is summarized per segment by a mean and standard deviation; a deviation score relative to that mean, compared against a threshold that loosens for high-variance objects and tightens for flat ones, yields an outlier mask. That mask both drives a Gaussian-style loss pushing student depth toward the regional mean where outliers were found, and gates the distillation loss so outliers do not propagate. Third, per-object mean photometric error is compared with the whole-image mean, and objects that are much less consistent than average receive lower reliability weight in the distillation loss.

The final objective combines the gated distillation loss, the contrastive loss, the regional Gaussian loss, and auxiliary losses inherited from the baseline. Training used PyTorch, Adam, batch size 10, a learning rate of 8×10⁻⁵ decayed by 0.5 every 5 epochs, and a single NVIDIA 4090D GPU for about 10 hours. Evaluation covers nuScenes (21,476 training images: 15,129 day, 3,796 rain, 2,551 night; 6,019 test frame pairs, metrics within 0.1–80 m) and Oxford RobotCar (17,790 day and 19,162 night training images; 1,411 test images, metrics within 0.1–50 m), using AbsRel, SqRel, RMSE, and δ₁.

Why This Matters

Impact on research: The paper shows that a general-purpose, class-agnostic segmentation model can substitute for semantic segmentation priors, avoiding the fixed label sets and poor cross-domain generalization that limit semantic-segmentation-based depth methods. Its diagnosis that pixel-level photometric error is a poor reliability indicator at the object level is a transferable finding for any pseudo-label distillation pipeline.

Real-world applications:

• Autonomous driving, where depth must remain reliable at night and in rain, not just in clear daylight.
• Robotics, where object-level depth boundaries matter for grasping and obstacle avoidance.
• AR/VR, where consistent object geometry affects scene compositing and occlusion.
• Driver-assistance and mapping systems operating on the nuScenes and RobotCar-style urban driving data this paper evaluates on.

Industry relevance: Because SAM masks are pre-generated offline, the method adds no segmentation cost at inference. It is a fine-tuning recipe on top of an existing teacher model (10 epochs, one GPU, about 10 hours), which makes it relatively cheap to adopt, and the reported gains are largest in the hardest setting measured — nighttime RobotCar, where SquRel improves 7.6% over the baseline.

Future Directions

• Reducing dependence on offline SAM: Masks are pre-computed with ViT-H for every training image; substituting lighter segmentation models, or generating masks on the fly, would broaden practical deployment.

• Finer evaluation across degradations: The experiments cover day-clear, night, and day-rain on nuScenes plus day and night on RobotCar; fog, snow, and other corruptions mentioned in the related work are not evaluated, leaving the robustness claim only partially tested.

• Sensitivity to mask granularity and thresholds: The merge rule (10⁵ pixels, 0.75 overlap) and the AROM/ORE thresholds (base threshold τ₀, sensitivity λ, scaling β) are set as hyperparameters; a systematic study of how mask granularity affects depth sharpness is not reported.

• Temporal and video consistency: The method is evaluated per-frame (with a multi-frame input variant for nuScenes); extending object-level reliability across time, and testing whether SAM's over-segmentation can be exploited rather than corrected, remain open.

Target Audience

Researchers and graduate students working on self-supervised or robust monocular depth estimation; practitioners building perception stacks for driving or robotics who already use a teacher–student depth pipeline and want a low-inference-cost improvement; and anyone interested in how foundation-model segmentation priors can be transferred into dense regression tasks through feature-level contrastive learning and reliability-weighted distillation.

Authors’ abstract

Robust depth estimation aims to maintain high-quality depths across diverse conditions. However, most existing methods estimate depth without taking into account the object-level information. As a result, the predicted depth may easily deviate within objects and become blurred under adverse conditions. To overcome this weakness, we propose RoSAMDepth, a novel framework that can assist robust self-supervised depth estimation in leveraging rich and diverse object-level priors from the Segment Anything Model (SAM). We focus on incorporating object-level information across three key aspects: a segment-guided representation contrasting method that injects object-level awareness into the feature representation space; an adaptive regional outlier masking strategy combined with a regional Gaussian likelihood loss that enforces regional depth smoothness; and an object-level reliability estimation strategy that mitigates the influence of unreliable supervision. Extensive experiments across multiple datasets and diverse weather conditions demonstrate that our method produces sharper, more accurate depth predictions, consistently outperforming state-of-the-art methods. Our code is available at https://github.com/xagao/RoSAMDepth.