Hybrid-grained Feature Aggregation with Coarse-to-fine Language Guidance for Self-supervised Monocular Depth Estimation

AI summary

Overview

Research area: Computer vision — self-supervised monocular depth estimation (MDE), with a focus on integrating vision-language and self-supervised foundation models (CLIP and DINO) into depth prediction, and on downstream 3D perception such as bird's-eye-view (BEV) prediction.

Technical level: Advanced. The paper assumes familiarity with self-supervised depth pipelines (DepthNet/PoseNet, photometric reprojection loss, edge-aware smoothness loss), contrastive learning, and prompt/instruction tuning of vision-language models.

Scope: The paper proposes a framework called Hybrid-depth that fuses CLIP's global semantic features with DINO's local spatial features under coarse-to-fine language guidance, and reports KITTI depth benchmarks plus nuScenes BEV perception results.

What This Paper Is About

Self-supervised monocular depth estimation normally trains a network to predict per-pixel depth from a single image by using camera motion and image reconstruction as supervision, but its accuracy is limited because the model never receives rich semantic or spatial prior knowledge. This paper asks whether large pre-trained foundation models — CLIP for global semantics and DINO for local spatial detail — can be adapted to supply that missing knowledge for self-supervised depth. The core obstacle the authors identify is a granularity mismatch: CLIP's representations are semantically rich but spatially imprecise, while DINO's are spatially detailed but lack contextual depth hierarchy, so naive fusion (e.g., direct channel concatenation) performs poorly.

Key Contributions

1. First use of CLIP and DINO together for self-supervised MDE. The authors state this is, to their knowledge, the first framework that leverages foundation models (CLIP and DINO) for self-supervised monocular depth estimation, transferring 2D pre-trained priors into 3D geometric perception through a coarse-to-fine scheme.

2. Language-guided aggregation of complementary features. They fuse CLIP's semantic context and DINO's spatial details using depth-related language prompts that act as a "granularity calibrator," which the authors claim resolves the feature-granularity mismatch that hurts naive concatenation.

3. A plug-and-play depth encoder. Hybrid-depth is modular and independent of typical self-supervised MDE pipelines, so it can be inserted into existing frameworks. The paper demonstrates this with Monodepth2, ManyDepth, and Mono-ViFI (also referred to as Mono-VIFI).

4. Demonstrated gains beyond depth. The framework improves all reported KITTI evaluation metrics for the methods it is plugged into, and also improves BEV perception methods (BEVDet and FB-BEV) on the nuScenes val set.

Main Findings

• Improvements on KITTI for every host method. With Hybrid-depth, Monodepth2 goes from Abs Rel 0.115 to 0.093, Sq Rel 0.903 to 0.596, RMSE 4.863 to 4.113, RMSE log 0.193 to 0.167, and δ<1.25 from 0.877 to 0.910 (δ<1.25² 0.959 to 0.970, δ<1.25³ 0.981 to 0.986). Manydepth improves from Abs Rel 0.098 to 0.096, Sq Rel 0.770 to 0.665, RMSE 4.459 to 4.192, RMSE log 0.176 to 0.170, δ<1.25 0.900 to 0.906. Mono-ViFI improves from Abs Rel 0.105 to 0.094, Sq Rel 0.708 to 0.658, RMSE 4.446 to 4.168, RMSE log 0.179 to 0.169, δ<1.25 0.887 to 0.906.

• Outperforms CLIP-based depth methods by a wide margin. On the same KITTI comparisons, DepthCLIP (0-shot) reports Abs Rel 0.473 and δ<1.25 of 0.281; Hu et al. (1-shot, 704×352) report Abs Rel 0.384; Auty et al. report Abs Rel 0.303. Monodepth2 with Hybrid-depth reports Abs Rel 0.093.

• Gains come from the coarse depth sensing stage, not just a stronger backbone. Removing that stage ("w/o co") while keeping the same backbone gives Monodepth2 Abs Rel 0.105 and Sq Rel 0.752, versus 0.093 and 0.596 with Hybrid-depth. The authors note that merely using a more powerful backbone does not achieve the same improvement.

• Both contrastive losses contribute. For Manydepth, adding only the intramodal loss L_i gives Abs Rel 0.098 / Sq Rel 0.717; only the cross-modal loss L_c gives 0.096 / 0.667; the full method gives 0.096 / 0.665. For Monodepth2, L_i alone gives 0.098 / 0.663, L_c alone 0.095 / 0.675, and the full method 0.093 / 0.596.

• Depth instruction as granularity calibrator matters in both stages. For Monodepth2, dropping calibration in the coarse stage ("w/o co-gc") gives Abs Rel 0.098 / RMSE 4.236; dropping it in the fine stage ("w/o fi-gc") gives 0.100 / 4.284; the full method gives 0.093 / 4.113.

• Both encoders are needed. With Monodepth2, using CLIP only gives Abs Rel 0.102 / RMSE 4.667, DINO only gives 0.104 / 4.685, while CLIP+DINO (Hybrid-depth) gives 0.093 / 4.113.

• Token count shows a non-monotonic effect. With 128 tokens: Abs Rel 0.095, Sq Rel 0.650, RMSE 4.199, RMSE log 0.170, δ<1.25 0.911. With 256 (the default used in all other experiments): 0.093, 0.596, 4.113, 0.167, 0.910. With 512: 0.097, 0.658, 4.182, 0.172, 0.908. With 1024: 0.095, 0.649, 4.154, 0.170, 0.911. The authors describe this as a capacity-overfitting trade-off.

• Downstream BEV perception improves. On nuScenes val at 256×704 input, BEVDet goes from mAP 0.283 / NDS 0.350 to 0.325 / 0.395 with Hybrid-depth; FB-BEV goes from mAP 0.312 / NDS 0.406 to 0.348 / 0.439. mATE, mASE, and mAOE also decrease for both.

• Qualitative improvement in edges and ground regions. The paper reports that Monodepth2 with Hybrid-depth produces smoother depth in ground regions while keeping sharper object boundaries and reducing depth artifacts around object edges compared with Manydepth and SQLDepth.

• Faster convergence. The authors note that Hybrid-depth promotes convergence of existing self-supervised MDE methods.

Methodology in Plain English

Stage 1 — Coarse depth sensing. The idea is to borrow a structural cue that is easy to obtain in driving scenes: along lane markings, depth increases consistently. The authors use lane detection data such as TuSimple (lane labels being more accessible than precise depth measurement) and crop a set of patches along the lane labels. They use N = 7 patches, chosen so that each patch is positioned higher in the image (smaller y coordinate) than the next, which serves as a proxy for "closer." Patches are sampled randomly along the column but regularly across the row to avoid patches at nearly identical depth.

Features come from two sources: multi-scale feature maps taken from four ResNet blocks of CLIP and DINO (the 2nd, 5th, 8th, and 11th layers). CLIP features are interpolated to DINO's spatial size and then concatenated. Each patch is paired with a text prompt formatted as "This patch appears to be [depth token]," where the token is drawn from a ranked set such as "very distant, …, close, very close." Because the patches have a known depth ordering, two loss terms enforce that ordering:

• An intramodal contrastive loss ensures a patch is more similar to itself than to patches at other depth levels, so the similarity matrix follows the expected ordinal pattern.
• A language-guided (cross-modal) contrastive loss ensures a patch's visual features are more similar to its own depth text embedding than to text embeddings for other depths.

Only the visual encoder is trained here; the text encoder stays frozen.

Stage 2 — Fine depth estimation. The coarse features are then refined into a dense depth map. The hand-crafted depth words are replaced with N learnable depth tokens, initialized by randomly sampling 512 elements from a normal distribution with mean 0 and standard deviation 0.02; these go through the frozen text encoder to produce depth text embeddings. The aggregated hybrid features are aligned with these embeddings through an inner-product operation that produces an N × HW matrix, which is reshaped back into a feature tensor. This tensor replaces the depth feature that the original DepthNet would produce, and depth is predicted by up-sampling following DPT, together with an auxiliary camera pose from a PoseNet.

Training uses the standard self-supervised recipe: warp one frame into another using the predicted depth and estimated 6-DoF pose, and minimize a photometric error combining an L1 pixel distance and SSIM over a 3×3 window with β = 0.85, plus an edge-aware smoothness loss with λ = 0.001. Only the visual encoder, the learnable tokens, and the up-sampling layers are trained; the text encoder is frozen.

Implementation. ResNet-50-based CLIP visual encoder, DINOv2 encoder, and a vanilla CLIP text encoder. PyTorch, 10 epochs, batch size 16, AdamW with initial learning rate 1e-4 and StepLR policy, on a single NVIDIA A800 GPU. The number of learnable depth and pose tokens is set to 256.

Why This Matters

Impact on research. The paper argues that prior CLIP-based depth methods underuse what vision-language models offer — either as zero-shot depth encoders or with learnable prompts that still trail conventional MDE methods. By showing that a coarse-to-fine, language-guided fusion of CLIP and DINO can be dropped into existing self-supervised pipelines and improve them, the work positions foundation-model priors as a practical upgrade path for self-supervised 3D perception rather than a replacement for it. The demonstration that the same encoder helps BEV perception links depth estimation to the broader 3D detection stack.

Real-world applications:

• Autonomous driving, which the paper lists among the primary application areas for MDE.
• Robotics, also listed by the authors as a key application domain.
• 3D reconstruction, the third application area named in the introduction.
• Bird's-eye-view perception for autonomous driving, where Hybrid-depth improved BEVDet and FB-BEV on nuScenes val.

Industry relevance. Self-supervised MDE avoids the cost of collecting per-pixel depth annotations, which the paper identifies as the main bottleneck of supervised depth. A plug-and-play encoder that improves Monodepth2, Manydepth, and Mono-ViFI without changing their pipelines is directly relevant to teams already running such pipelines in vehicles or robots. The BEV results indicate the improvement propagates to downstream perception modules used in autonomous-driving stacks.

Future Directions

• Extending to more and harder domains. The paper notes that self-supervised methods struggle in challenging conditions such as bad weather, night time, and indoor environments; whether Hybrid-depth transfers to these settings is not evaluated.
• Broadening the downstream task evaluation. BEV perception is the only downstream task tested (BEVDet and FB-BEV on nuScenes val at 256×704 input). Object detection, segmentation, and planning are untested.
• Understanding and tuning the token capacity trade-off. The paper reports a non-monotonic relationship between learnable token count and accuracy and attributes it to a capacity-overfitting trade-off, but a principled way to choose the count is not provided.
• Testing other foundation-model pairings and backbones. The study uses ResNet-50-based CLIP, DINOv2 ViT-B, and a vanilla CLIP text encoder; whether other encoders or prompt designs change the conclusions is not reported.

Target Audience

Researchers and engineers working on self-supervised monocular depth estimation, foundation-model adaptation, vision-language prompt/instruction tuning, and autonomous-driving perception — particularly those who already run pipelines such as Monodepth2, Manydepth, or Mono-ViFI and want a drop-in encoder upgrade. Readers need a working understanding of self-supervised depth training, contrastive learning, and CLIP/DINO-style models to follow the method and ablation sections; the paper is not introductory.

Authors’ abstract

Current self-supervised monocular depth estimation (MDE) approaches encounter performance limitations due to insufficient semantic-spatial knowledge extraction. To address this challenge, we propose Hybrid-depth, a novel framework that systematically integrates foundation models (e.g., CLIP and DINO) to extract visual priors and acquire sufficient contextual information for MDE. Our approach introduces a coarse-to-fine progressive learning framework: 1) Firstly, we aggregate multi-grained features from CLIP (global semantics) and DINO (local spatial details) under contrastive language guidance. A proxy task comparing close-distant image patches is designed to enforce depth-aware feature alignment using text prompts; 2) Next, building on the coarse features, we integrate camera pose information and pixel-wise language alignment to refine depth predictions. This module seamlessly integrates with existing self-supervised MDE pipelines (e.g., Monodepth2, ManyDepth) as a plug-and-play depth encoder, enhancing continuous depth estimation. By aggregating CLIP's semantic context and DINO's spatial details through language guidance, our method effectively addresses feature granularity mismatches. Extensive experiments on the KITTI benchmark demonstrate that our method significantly outperforms SOTA methods across all metrics, which also indeed benefits downstream tasks like BEV perception. Code is available at https://github.com/Zhangwenyao1/Hybrid-depth.