Learning to Discriminate Scene Structures Makes Self-Supervised Depth Learning Scalable

AI summary

Overview

Research area: Computer Vision — self-supervised monocular depth estimation (SSMDE), with a focus on cross-domain scalability and zero-shot generalization.

Technical level: Advanced. The paper assumes familiarity with self-supervised depth from monocular video, vision transformers (DINOv2), adapter architectures, and contrastive learning objectives.

Scope: The paper diagnoses why self-supervised depth models fail when trained on diverse, mixed-domain data, and proposes MaSS, an architecture-plus-objective framework that learns decoupled structural representations to make such training scalable.

What This Paper Is About

Self-supervised monocular depth estimation is attractive because it can learn from unlabeled video, but existing methods are almost always trained on one dataset and degrade when trained on many different scene types at once. The authors show that this happens because the network latches onto dataset-specific structural shortcuts (for example, a "vanishing point" prior learned from driving data) instead of genuine, transferable scene geometry, so different domains pull the optimization in conflicting directions. Their goal is to force the model to learn a generalized scene-structure representation before decoding depth, so that adding more diverse data actually helps rather than hurts.

Key Contributions

1. Diagnosis of cross-domain optimization conflict. The authors show that the coarsest, high-level feature of the depth encoder does not encode depth cues such as foreground–background relations or perspective geometry, but instead reflects dataset-specific structural bias — which explains why mixed-domain self-supervised training causes performance declines.

2. The MaSS framework. A bidirectional interaction architecture bridges a trainable conventional depth encoder with a frozen pretrained vision foundation model (DINOv2 ViT-B/14, following the ViT-Adapter design), injecting learnable geometric cues into frozen VFM features.

3. Anchored–decoupled feature projectors. Two projectors produce collapse-free representations: one anchored to the VFM's semantic relationships across samples (contextual), the other explicitly decoupled from it (structural), enforced by the anchoring–decoupling loss.

4. A scene-structure discrimination objective. A contrastive objective with a custom structural discrepancy metric (soft-histogram depth distributions compared by symmetric KL divergence) organizes the structural feature space by scene-layout similarity rather than generic instance identity.

Main Findings

• State-of-the-art single-dataset results: Trained and evaluated individually, MaSS beats prior self-supervised methods on all four datasets — KITTI (Abs Rel 0.094, δ₁ 0.909), DDAD (0.128, 0.845), NYUv2 (0.109, 0.868), and nuScenes (0.139, 0.851). The strongest prior results on those datasets were 0.095/0.902 (DCPI-Depth on KITTI), 0.134/0.831 (DiMoDE on DDAD), 0.113/0.871 (Gas-Mono on NYUv2), and 0.139/0.836 (DiMoDE on nuScenes).

• First self-supervised method to benefit from mixed-domain training: Under a mixed four-dataset setting (random sampling from KITTI, DDAD, NYUv2, nuScenes per batch, one-third the epochs), MaSS improves to KITTI 0.092/0.913, DDAD 0.125/0.862, NYUv2 0.104/0.890, and nuScenes 0.134/0.852, while reproduced SoTA baselines (Lite-Mono, Dynamo-Depth, SC-DepthV3, DCPI-Depth) all degrade — for example DCPI-Depth on NYUv2 falls to 0.331/0.472 and Lite-Mono on nuScenes to 1.147/0.816.

• Strong zero-shot generalization across eight unseen datasets: The mixed-dataset model scores Waymo Open 0.123/0.867, DrivingStereo 0.100/0.905, Cityscapes 0.103/0.896, DIML 0.104/0.890, iBims-1 0.107/0.883, SUN-RGBD 0.127/0.893, 7-Scenes 0.120/0.861, and DIODE 0.289/0.736. Mixed-domain training consistently beats the single-dataset MaSS variant, indicating mutual enhancement across domains.

• Competitive with far less overhead and data: MaSS† uses 93.2M parameters, 81.6 GFLOPs, and 150K training samples, versus DA V2 at 97.5M / 89.5 GFLOPs / 62M samples, and KBR++ at 266.6M / 381.7 GFLOPs / 2.2M samples. MaSS† outperforms the KBR series despite this gap.

• Remaining gap on simple indoor benchmarks: MaSS still trails supervised depth foundation models on standard indoor datasets (e.g., DA V2 reaches 0.045/0.981 on iBims and 0.062/0.961 on SUN-RGBD versus MaSS† at 0.107/0.883 and 0.127/0.893), which the authors attribute to the homogeneous depth distributions of those datasets suiting models trained on large-scale indoor ground truth. The gap narrows on DIODE and DIML.

• Every component matters (ablation): Removing the contrastive objective gives nuScenes 0.145/0.850 and zero-shot Waymo 0.163/0.805; removing representation decoupling gives 0.144/0.840; removing the interaction architecture gives 0.147/0.837 and Waymo 0.190/0.750.

• Design beats plausible alternatives: Frozen DINOv2 + DPT reaches 0.186/0.719 on nuScenes, fully fine-tuning DINOv2–DPT reaches 0.197/0.709, and ViT-Adapter + DPT reaches 0.166/0.767 — all well behind MaSS† at 0.134/0.852.

• Backbone strength is not the driver: Swapping DINOv2 for DINOv3 changes results only marginally (nuScenes 0.136/0.855, NYUv2 0.100/0.897, Waymo 0.152/0.826, 7-Scenes 0.128/0.834).

• Scaling data further helps: Adding TartanAir (200K samples total) yields further gains on several zero-shot sets — Waymo 0.119/0.878, DrivingStereo 0.097/0.908, DIML 0.087/0.929 — with NYUv2 improving to 0.097/0.905. Results for the remaining zero-shot datasets in this table are not reported in the provided content.

Methodology in Plain English

The starting observation is that a depth network trained only on driving footage predicts depth that converges toward a vanishing point, while one trained on small rooms cannot handle large halls. The authors trace this to the encoder's highest-level features, which carry the dataset's structural fingerprint rather than real geometry.

To break that habit, MaSS keeps a standard depth encoder and pairs it with a frozen DINOv2 vision transformer. The two exchange information in both directions through cross-attention, so the depth encoder contributes geometric cues the frozen model lacks, while the frozen model contributes broad, generalizable semantics the depth encoder lacks — and the frozen weights are never damaged by noisy self-supervised signals.

The exchanged features pass into two separate projectors. One learns a contextual representation — object details and foreground–background relations — and is anchored so that relationships between samples stay close to those of the original DINOv2 features. The other is pushed to be orthogonal to the contextual one, so its capacity is spent on scene layout rather than redundant semantics.

To define what "scene structure" means without labels, the authors compare predicted depth maps. Each depth map is downsampled by a factor of 8, converted into a soft 20-bin histogram (a smooth binning function suppresses high-frequency artifacts), and compared to others using a symmetric KL divergence. Samples whose depth histograms are similar (similarity above 0.95) become positives in a multi-positive InfoNCE contrastive loss with temperature 0.1. This teaches the structural projector to group scenes by layout similarity rather than by superficial appearance — visually different scenes with the same layout end up nearby.

The two feature pyramids are concatenated, reduced with a 1×1 convolution, and fed to the depth decoder. Because the contrastive objective depends on depth predictions, the model needs a one-epoch warm-up stage with plain self-supervision first. Training fits on a single RTX 4090 with batch size 6 in roughly 2.5 days.

Why This Matters

The paper reframes data scaling for self-supervised depth: the bottleneck is not the volume of unlabeled video but the model's tendency to memorize per-dataset structure. If that tendency can be removed, the enormous supply of unlabeled driving, indoor, and in-the-wild video becomes usable in a single training run instead of many separate domain-specific models.

Real-world applications:

• Autonomous driving and ADAS, where a single depth model must work across cities, weather, lighting, and camera rigs without per-fleet retraining.
• Robotics and mobile manipulation, where robots encounter indoor rooms, corridors, and outdoor spaces and cannot be re-trained for each new environment.
• AR/VR and 3D content capture, where consumer devices need metric-aware depth from ordinary monocular video in unfamiliar spaces.
• Large-scale mapping and simulation, where mixed-source video (dashcams, phones, public datasets) must be pooled into one depth system.

Industry relevance: The overhead comparison matters commercially — MaSS achieves its results with 93.2M parameters, 81.6 GFLOPs, and 150K training samples, far below the 62M samples used by DA V2 and the 2.2M used by KBR++, and it trains on a single consumer-grade GPU. That places scalable self-supervised depth within reach of teams without large annotation budgets or GPU clusters.

Future Directions

• Combining MaSS with geometry or depth foundation models. The authors explicitly raise the question of whether MaSS can benefit from pretrained depth/geometry foundation models, or even improve them, and state that this is investigated in the supplementary material — a natural path to closing the indoor gap.

• Closing the indoor gap against supervised models. MaSS still underperforms supervised foundation models on homogeneous indoor benchmarks, and the paper's explanation (simple depth distributions favor large-scale ground-truth pretraining) suggests targeted work on indoor depth diversity.

• Pushing scale further. The TartanAir experiment (200K samples) shows continued gains, but the reported results are partial in the provided content; how far the trend holds at much larger scale, and where it saturates, remains open.

• Generalizing the structural discrimination idea. The soft-histogram discrepancy metric is defined over depth distributions; whether analogous layout-aware contrastive objectives could serve other self-supervised geometric tasks (surface normals, optical flow, camera pose) is unexplored.

Target Audience

Researchers and graduate students working on self-supervised depth estimation, domain generalization, and multi-dataset training; engineers building perception stacks who need one depth model to survive across domains; and practitioners interested in how frozen vision foundation models can be adapted without fine-tuning. Readers should be comfortable with monocular depth pipelines, transformer adapters, and contrastive losses — the paper is not an introductory read.

Authors’ abstract

Self-supervised monocular depth estimation is appealing for its potential to scale with unlabeled data, yet existing methods often degrade when trained on diverse scenes. This study finds that, under self-supervision, models tend to encode dataset-specific structural biases rather than transferable scene geometry, leading to cross-domain optimization conflicts. To address this issue, this study proposes the MaSS framework, aiming to make self-supervised depth learning scalable to diverse data. MaSS introduces a dedicated structural representation pathway explicitly decoupled from contextual features, which is guided by a novel scene-structure discrimination objective to learn generalized, layout-aware structural representations for depth decoding. Trained and evaluated on four diverse datasets, both individually and jointly, MaSS substantially outperforms prior self-supervised methods and is the first to consistently benefit from mixed-domain training. Zero-shot evaluation on eight additional datasets further demonstrates its remarkable generalizability, particularly on those entirely out-of-distribution scenes. The source code is publicly available at https://mias.group/MaSS.