MetricAnything: Scaling Metric Depth Pretraining with Noisy Heterogeneous Sources

AI summary

Overview

Research area: Computer vision — monocular metric depth estimation, large-scale pretraining, 3D perception foundation models.

Technical level: Advanced. The paper assumes familiarity with Vision Transformers, DPT heads, scale-and-shift-invariant losses, distillation, and depth estimation metrics (AbsRel, Log10, δ thresholds).

Scope: The paper proposes a minimally engineered pretraining paradigm that learns metric depth from roughly 20 million noisy, heterogeneous image–depth pairs using randomly masked depth maps as "Sparse Metric Prompts," then distills the result into a prompt-free student model for downstream tasks.

Authors are affiliated with Li Auto Inc. The paper is released under CC BY 4.0, and code is open-sourced at https://metric-anything.github.io/metric-anything-io/.

What This Paper Is About

Relative depth estimation has scaled successfully by mixing datasets and distilling pseudo-labels, but metric depth estimation — predicting absolute, physically meaningful distances — has not shown comparable scaling trends, because real-world 3D annotations come from different sensors (LiDAR, RGB-D, stereo) and algorithms (SfM, MVS, SLAM) that each bring their own noise patterns, artifacts, and camera-dependent biases.

The goal is to build a general-purpose metric depth foundation model that learns from that "data soup" directly, without hand-crafted prompts, camera-specific modeling, or task-specific architectures.

Key Contributions

1. Metric Anything pretraining paradigm. A minimalist prompt-based pretraining framework in which a Sparse Metric Prompt — produced by randomly masking depth maps — acts as a universal interface that decouples spatial reasoning from sensor and camera biases, enabling learning from heterogeneous 3D sources.

2. Demonstrated scaling in metric depth. Aggregation of approximately 20M image–depth pairs spanning reconstructed, captured, and rendered 3D data across 10,000+ camera models, with the paper reporting a stable scaling trend (larger training data yields higher zero-shot δ1 accuracy) previously unseen in this domain.

3. Universal generalization without task-specific engineering. Both the pretrained teacher and its distilled prompt-free student are reported to reach state-of-the-art performance across ten downstream tasks, including depth completion, super-resolution, radar-camera depth estimation, monocular depth estimation, camera calibration, single/multi-view metric 3D reconstruction, and VLA planning.

4. Pretrained ViT as a visual encoder for MLLMs. Using the pretrained Metric Anything ViT as a visual encoder is reported to significantly boost Multimodal Large Language Model capabilities in spatial intelligence.

Main Findings

• Scaling trend in metric depth: Figure 2(a) shows that a larger training dataset yields consistently higher zero-shot δ1 accuracy, which the authors frame as evidence that metric depth estimation can follow the same scaling laws as modern foundation models.

• Zero-shot depth super-resolution and completion: With the pretrained model used directly and zero-shot (no task-specific finetuning), Ours-Pretrain reports AbsRel (%) on NYUv2 of 1.53 (8×), 1.86 (16×), 1.70 (LiDAR-like), 2.08 (extreme, 100 points); on ETH3D of 1.46 (8×), 2.03 (16×), 0.87 (LiDAR-like), 0.84 (extreme); and on KITTI of 2.34 (8×), 3.53 (16×), 3.53 (LiDAR-like), 3.36 (extreme). Comparison baselines include DAv2, DepthPro, Omni-DC, Marigold-DC, DepthLab, PromptDA, PriorDA, and LingBot-Depth. Unlike prior work that simulates test-time prompts, the model is trained once with simple sparsely sampled prompts.

• Radar-camera depth estimation on nuScenes: Radar data was deliberately excluded from pretraining to avoid information leakage, making this a transfer test to an unseen sensor. Finetuning the pretrained teacher reports MAE/RMSE of 651.4/2084.4 mm (0–50 m), 863.6/2771.6 mm (0–70 m), and 934.5/3057.5 mm (0–80 m). The from-scratch counterpart reports 1335.4/2958.8, 1622.9/3788.1, and 2101.7/4033.2 mm respectively — the paper describes finetuning as nearly doubling the accuracy of its from-scratch counterpart and surpassing all prior fusion methods, including TacoDepth (1492.4/3324.8 mm at 0–80 m in the plug-in setting) and RadarCam-Depth. Radar point clouds are noted as over a thousand times sparser than LiDAR.

• Monocular depth estimation — δ1 accuracy (%): Student-DepthMap reports 79.9 (ETH3D), 59.5 (Booster), 88.1 (NuScenes), 97.7 (Sun-RGBD), 27.7 (Sintel), and 65.8 (Middlebury), with an overall rank of 1.50 — the best rank among compared methods including DepthAnything, DAV2, Metric3D, Metric3D-v2, PatchFusion, UniDepth, ZoeDepth, and DepthPro.

• Detailed zero-shot metrics (Table 4): Student-Depthmap reports AbsRel / Log10 / δ2 / δ3 of 0.147 / 0.064 / 97.71 / 99.90 on ETH3D; 0.152 / 0.063 / 96.56 / 98.26 on nuScenes; 0.792 / 0.227 / 50.01 / 70.57 on Sintel; 0.085 / 0.033 / 99.31 / 99.65 on Sun-RGBD; 0.200 / 0.082 / 96.17 / 99.93 on Middlebury; and 0.282 / 0.100 / 84.11 / 96.83 on Booster. All six test datasets were excluded from training, validation, and hyperparameter tuning for a strict zero-shot protocol.

• Domain-specific strengths: On Sun-RGBD the method reports AbsRel 0.085 versus 0.156 for Metric3D v2. On Booster it reports AbsRel 0.282, described as outperforming all baselines. On nuScenes and ETH3D it ranks 2nd in absolute error metrics while often achieving the highest accuracy in stricter thresholds (δ3); the paper argues δ3 is more reasonable given inherent noise in those ground-truth datasets.

• Student point maps: Pseudo-labels from the pretrained model were used to finetune MoGe-2, denoted Student-PointMap, reported as consistently achieving state-of-the-art performance across Fig. 8, Fig. 9, Table 5, and Table 6. The specific numeric values of Tables 5 and 6 are not included in the available content.

• DepthAnything3 comparison: The GIANT-LARGE and DA3MONO-LARGE variants of DepthAnything3 were evaluated, and the paper states their performance was not particularly satisfactory in this setting, conjecturing that DepthAnything3 depends on inferring matching relationships across multiple views.

• Loss and architecture findings: For real-world data, dropping the top 20% highest-loss regions per image is used as a robust MAE. The student uses a distance-balanced inverse-depth loss, D_log = 1 − ln(x)/ln(C), where C trades off long-range and short-range supervision, plus an inverted skip-connection scheme that feeds deep ViT features into deeper decoder layers near the output.

• Parameter efficiency of prompt injection: The conditioned DPT head adds only about 5% extra parameters relative to the depth backbone.

Methodology in Plain English

The authors first gather depth data from three kinds of sources: reconstructed 3D data (from SfM, SLAM, MVS, stereo, and temporal stereo), captured 3D data (from LiDAR, ToF, and RGB-D sensors), and a smaller amount of rendered 3D data from virtual engines. Everything is standardized as per-pixel metric depth maps along the camera z-axis, together with a validity mask marking which pixels have real measurements. The total is approximately 20 million image–depth pairs across more than 10,000 camera models.

Instead of designing task-specific prompts, they create a Sparse Metric Prompt by randomly sampling between 2,000 and 40,000 valid pixels (roughly 1% of the image) from each depth map. To keep prompts in a consistent format, they use a pretrained depth model to produce a prior depth map, then apply Pixel-wise Depth Scale Alignment and Global Metric Depth Recovery (borrowed conceptually from See3D) to fill gaps, concatenating the results with the prompt mask into a three-channel input. A conditioned DPT head injects this prompt into the network, adding only about 5% extra parameters, while the backbone itself is a standard, widely used architecture with the patch encoder and image encoder merged into one shared ViT.

Training uses MAE and scale-and-shift-invariant mean absolute gradient error for synthetic data, and a robust MAE for real-world data that discards the top 20% highest-loss regions per image to reduce sensitivity to noise.

Finally, the trained teacher generates dense pseudo depth labels from RGB images, which are used to train a prompt-free student. Because those pseudo-labels stay reliable at both near and far distances — unlike most real sensor data, which is biased toward short range — the authors replace the standard skip connections with an inverted scheme and use a distance-balanced inverse-depth loss, so the student can lean on deep semantic features rather than shallow texture cues.

Why This Matters

Impact on research. The paper argues that metric depth estimation can benefit from the same scaling laws that drive NLP and 2D vision foundation models, and that general-purpose data-driven methods systematically outperform hand-crafted designs — an argument the authors link explicitly to "The Bitter Lesson." It challenges the prevailing practice in metric depth of carefully curating small, clean datasets or building elaborate prompt-engineering pipelines, and suggests future work should focus on general pretraining paradigms rather than single-sensor training and testing.

Real-world applications:

• Autonomous driving, where radar-camera fusion depth estimation matters for low-cost, all-weather, long-range perception.
• Robotics and Vision-Language-Action planning, where the model is reported to support VLA planning without task-specific design.
• Augmented reality, where accurate single-view and multi-view metric 3D reconstruction is needed for placing virtual content in physical space.
• Camera intrinsics recovery and calibration, useful for processing imagery from unknown or uncalibrated cameras.

Industry relevance. The work comes from Li Auto Inc., and the applications listed — driving perception, radar fusion, VLA planning — map directly onto automotive and embodied-AI product needs. The efficiency arguments (only ~5% extra parameters for prompt injection, a single shared ViT backbone) and the open-sourced model point toward practical deployment considerations alongside research novelty.

Future Directions

• Extending scaling further. The paper demonstrates a trend up to approximately 20M image–depth pairs and 10,000+ camera models; whether the trend continues with larger data or model sizes is left open.

• Broader sensor transfer. Radar was deliberately excluded from pretraining and used as a transfer test. The results suggest pretraining adapts to unseen modalities, but other sensors and sensing conditions are not explored in the available content.

• Task-specific numbers for the remaining downstream tasks. Results for camera intrinsics recovery, single/multi-view metric 3D reconstruction, and VLA planning are claimed but the detailed figures for some of these (including Tables 5 and 6) are not present in the available content.

• Understanding the limits of the distilled student. The paper notes that reusing the pretraining architecture directly as the student was suboptimal, hypothesizing causes related to depth distribution and skip connections. A more systematic account of when these design changes help or hurt would be a natural follow-up.

Target Audience

Researchers and engineers working on monocular depth estimation, depth completion, sensor fusion, 3D reconstruction, and vision foundation models, particularly those interested in scaling laws and pretraining data strategy. Also relevant to practitioners in autonomous driving, robotics, and AR who need metric-scale depth from limited sensors, and to teams building Multimodal Large Language Models that require stronger spatial reasoning encoders. Readers should be comfortable with depth estimation benchmarks and transformer-based vision architectures.

Authors’ abstract

Scaling has powered recent advances in vision foundation models, yet extending this paradigm to metric depth estimation remains challenging due to heterogeneous sensor noise, camera-dependent biases, and metric ambiguity in noisy cross-source 3D data. We introduce Metric Anything, a simple and scalable pretraining framework that learns metric depth from noisy, diverse 3D sources without manually engineered prompts, camera-specific modeling, or task-specific architectures. Central to our approach is the Sparse Metric Prompt, created by randomly masking depth maps, which serves as a universal interface that decouples spatial reasoning from sensor and camera biases. Using about 20M image-depth pairs spanning reconstructed, captured, and rendered 3D data across 10000 camera models, we demonstrate-for the first time-a clear scaling trend in the metric depth track. The pretrained model excels at prompt-driven tasks such as depth completion, super-resolution and Radar-camera fusion, while its distilled prompt-free student achieves state-of-the-art results on monocular depth estimation, camera intrinsics recovery, single/multi-view metric 3D reconstruction, and VLA planning. We also show that using pretrained ViT of Metric Anything as a visual encoder significantly boosts Multimodal Large Language Model capabilities in spatial intelligence. These results show that metric depth estimation can benefit from the same scaling laws that drive modern foundation models, establishing a new path toward scalable and efficient real-world metric perception. We open-source MetricAnything at http://metric-anything.github.io/metric-anything-io/ to support community research.