UniMetric: Universal Metric Scale Recovery for Visual Geometry Foundation Models

AI summary

Overview

• Research area: Computer vision / 3D visual geometry — recovering metric (physical) scale from monocular image sequences produced by visual geometry foundation models.
• Technical level: Intermediate. The central idea is intuitive, but the paper assumes familiarity with multi-view reconstruction, pointmaps, camera trajectories, depth evaluation metrics, and vision–language model prompting.
• Scope: The paper proposes a training-free method (UniMetric) that turns uncertain vision–language-model object-size priors into a single metric scale for a geometry foundation model's reconstruction, plus a twelve-subset benchmark (MetricBench) for measuring metric accuracy without ground-truth scale alignment.

What This Paper Is About

Visual geometry foundation models (GFMs) can reconstruct dense 3D structure and camera motion from ordinary video, but the reconstructions are only defined up to an unknown scale factor — they do not know whether a metre in the model equals a metre in the world. The paper asks whether the content of a scene (familiar objects such as cars, doors, and pedestrians, whose sizes are approximately known) provides enough information to recover that missing physical unit, without adding sensors and without training a metric predictor. The authors answer yes, and build a framework that combines many uncertain object-size estimates into one shared sequence-level scale.

Key Contributions

1. Recasting metric scale recovery as joint estimation from uncertain semantic size priors. The authors formalize how a semantic height estimate becomes a scale observation, prove a mean-square consistency result (Proposition 1) showing the error bound decreases as observations accumulate, and validate the benefit empirically with real VLM height intervals while holding relative object heights exact.

2. UniMetric, a universal and training-free scale recovery framework. It has three modules — Metric Prior Acquisition (a VLM returns object categories, bounding boxes, height intervals in metres, confidences, and a support surface), Semantic–Geometric Alignment (measuring reconstructed object heights along each frame's fitted support-plane normal and converting priors to admissible scale intervals), and Global Scale Recovery (minimizing a weighted convex interval objective to yield one scalar applied uniformly to pointmaps and camera translations).

3. MetricBench, a cross-platform benchmark for metric accuracy. Twelve dataset subsets totalling 694 sequences and 51,339 frames, split into driving-and-infrastructure and embodied-AI suites, with sensor-derived (LiDAR or depth camera) metric references and a protocol that allows rigid SE(3) registration but never ground-truth scale fitting.

4. A large paired evaluation across nine GFMs. VGGT, VGGT-Ω, π³, Depth Anything 3 (DA-3), Scal3R, LingBot-Map, 4RC, Déjà View (DVLT), and STream3R are each evaluated before and after UniMetric correction on depth, 3D reconstruction, and camera trajectory, using the same estimated scale for all three tasks.

Main Findings

• Aggregating uncertain priors improves scale: In a controlled study over 193 sequences from the nuScenes, JRDB, 3EED, and V2X-Radar-I subsets — with relative heights kept exact (h = 0.3·H, so the true recovery factor is 1/0.3) and height intervals supplied by a real VLM — mean absolute relative scale error (MARE) falls from 4.526–7.245% at a budget of K=1 observation to 1.086–2.175% at K=50. Relative reductions are 76.7% (nuScenes, 7.245% → 1.691%), 67.2% (JRDB, 6.627% → 2.175%), 79.8% (3EED, 5.435% → 1.097%), and 76.0% (V2X-Radar-I, 4.526% → 1.086%). Much of the gain is already present at K=10.

• Headline reductions across nine GFMs: UniMetric reduces mean depth Abs Rel by 54.2%, reconstruction Chamfer distance (CD) by 83.8%, and trajectory error (ATE Mean) by 65.6%, relative to uncorrected predictions, with no ground-truth scale alignment.

• Depth improves everywhere: Abs Rel improves in all 108 model–subset pairs. Suite-averaged mean Abs Rel falls from 0.892 to 0.338 in the driving-and-infrastructure suite and from 0.671 to 0.379 in the embodied-AI suite. Averaged over nine models, δ₁ rises from 3.7% to 55.4% (driving) and from 13.7% to 44.8% (embodied).

• Reconstruction improves in 107 of 108 pairs: Mean CD on the driving suite falls from 17.471 m to 2.528 m and on the embodied suite from 4.396 m to 1.008 m. For VGGT-Ω on FusionPortable, CD drops from 15.848 m to 0.653 m; on nuScenes from 14.594 m to 0.726 m. The single increase is π³ on DROID (0.090 m → 0.097 m), where the original reconstruction already had small metric error.

• Camera trajectories improve in 81 of 90 pairs: Mean ATE on the driving suite falls from 38.634 m to 12.119 m (a 26.515 m reduction); the embodied suite falls from 9.149 m to 3.938 m. For VGGT-Ω on FusionPortable, ATE falls from 39.863 m to 4.238 m under the same scale that improves its depth and CD. No separate trajectory scale is fitted.

• Where the method does not help is informative: Seven of the nine ATE pairs without improvement are on DROID, whose external camera is stationary and whose ATE values are at the millimetre-to-centimetre scale. The other two are VGGT on CEAR (8.296 m → 8.838 m) and on Humanoid Everyday (0.150 m → 0.158 m), despite improved depth on both. The authors state that a shared scale calibrates translation magnitudes but cannot fix errors in relative rotations or trajectory shape.

• Effect depends on the model's original metric error: On KITTI, π³ changes only from 0.190 to 0.188 Abs Rel, whereas VGGT-Ω on nuScenes improves from 0.979 to 0.152 — the corrected quantity is a scale factor, so models that already have small metric error gain less.

• Benchmark scale diversity: Median reference depth across MetricBench subsets ranges from 0.695 m (Humanoid Everyday) to 36.865 m (V2X-Radar-I), a more than 50× variation; the 90th-percentile reference depth reaches 81.245 m on DAIR-V2X-I, and mean trajectory length per sequence for moving cameras ranges from 1.305 m (Humanoid Everyday) to 251.145 m (FusionPortable).

Methodology in Plain English

1. Ask a vision–language model what it sees. For each frame, a pretrained VLM (Seed1.8 in these experiments) identifies familiar objects, returns bounding boxes, gives a plausible height range in metres with a confidence rather than a single nominal size, and marks the dominant support surface the objects stand on. A segmentation model (SAM 2.1) turns the boxes into pixel masks.

2. Measure the same objects in the reconstruction. The masks select points from the GFM's pointmaps, which are transformed into a common frame using the predicted camera poses. A plane is fitted to the support-surface points, and its unit normal defines the "up" direction for that frame — so the method does not assume a camera axis is vertical, and does not force one global ground plane on the whole sequence.

3. Turn each measurement into a scale interval. The reconstructed object height is the extent of its points along that support normal. Dividing the VLM's height interval by this reconstructed height yields a range of scales that would make that object plausible.

4. Reconcile all intervals into one number. Every observation is weighted by the VLM's confidence times a category-specific reliability weight. The objective is the weighted sum of squared distances from the candidate scale to each interval — a convex, piecewise-quadratic function solved by golden-section search. If all intervals overlap, any scale in their common intersection is a minimizer; if not, there is a unique compromise. When every interval collapses to a point with equal weights, the objective reduces to the simple mean analyzed in the theory section.

5. Apply the single scale. The recovered scalar multiplies the reconstructed points, the camera translations, and the depth maps. Rotations, intrinsics, and all relative geometry are untouched, so the method does not retrain or modify the underlying GFM.

6. Evaluate without hiding the error. MetricBench measures depth, 3D geometry, and camera pose against sensor references, permitting rigid SE(3) registration where coordinate frames differ but never fitting scale to ground truth — the standard practice that would erase exactly the error being studied.

Why This Matters

Impact on research. The paper separates two properties that are usually conflated — the quality of relative geometry and the correctness of its physical scale — and provides a benchmark protocol that keeps them distinct. It also supplies a theoretical argument (mean-square consistency with an error bound of σ²/N under zero-mean, uncorrelated, variance-bounded errors) for why many noisy semantic priors can jointly pin down one shared quantity, plus empirical evidence with real VLM intervals. The released code and data give the field a common way to compare scale-recovery approaches. The framing connects to enabling internet-video pretraining for embodied multimodal models, where reliable physical-scale annotation is a bottleneck.

Real-world applications (grounded in MetricBench's platforms):

• Autonomous driving and roadside infrastructure — the driving-and-infrastructure suite spans passenger vehicles and roadside units (KITTI, nuScenes, Waymo, Argoverse, V2X-Radar-I, DAIR-V2X-I) with median depths up to 36.865 m.
• Wheeled and legged robot navigation — CEAR (quadruped) and JRDB (wheeled robot) provide LiDAR references for platforms that need metric ego-motion.
• Robotic manipulation and humanoid control — DROID (Franka arm with external depth camera) and Humanoid Everyday (G1 humanoid) operate at close range, with median depths of 0.885 m and 0.695 m.
• Aerial and handheld mapping — 3EED (drone/quadruped) and FusionPortable (handheld/legged/UGV) cover much larger camera travel, with mean trajectory lengths up to 251.145 m per sequence.

Industry relevance. Any pipeline that turns consumer video or existing footage into metric 3D data — without LiDAR rigs, IMU calibration, or metric-labelled training sets — benefits from a training-free correction that can be bolted onto an existing GFM. The paper also reports a metric-depth calibration comparison (Table 7) that evaluates alternative sources of scale using the same backbone predictions, which is directly relevant to teams deciding between sensor-based, supervised, and prior-based scale strategies.

Future Directions

• Diagnosing the residual failures. The authors explicitly identify cases where a shared scale is insufficient — notably VGGT on CEAR and Humanoid Everyday for trajectory error, and π³ on DROID for CD — and note that relative rotation and trajectory-shape errors are outside what a scalar correction can fix. Characterizing and addressing those failure modes is a natural next step.
• Extending beyond a single per-sequence scalar. The paper applies one factor to depth, points, and translations; sequences with changing scale conditions, or compositions of multiple moving objects, are not addressed by this formulation.
• Dependence on VLM prior quality. The controlled study isolates semantic uncertainty with exact relative heights, but the full pipeline inherits VLM errors in object identification, height intervals, and support-surface localization. How sensitive the recovered scale is to prompt design, category coverage, and confidence calibration (the per-category reliability weights α_c are given as inputs, with prompts and settings left to the appendices) is an open practical question.
• Scaling and coverage. MetricBench covers twelve subsets, but its coverage of dynamic scenes, unusual objects, and very-long-duration streams — the settings the paper motivates for internet-video embodied pretraining — is not yet established.

Target Audience

Researchers and engineers working on 3D reconstruction, SLAM, depth estimation, and embodied AI who need physically scaled output from geometry foundation models; benchmark developers interested in metric evaluation protocols that avoid ground-truth scale fitting; and practitioners who want a training-free post-processing correction that works with released model weights and does not require additional sensors. Readers without background in multi-view geometry will find the conceptual framing accessible but may need to consult the appendices for implementation detail.

Authors’ abstract

Recent advances in multimodal foundation models, exemplified by GPT-6 Astra, raise the prospect of general-purpose embodied reasoning and planning. Using Internet videos for embodied pretraining calls for reliable physical-scale annotations. Yet visual geometry foundation models often recover scene structure and camera motion without reliable metric scale. We show that uncertain size priors of familiar objects can jointly constrain a reconstruction's shared scale. Building on this insight, we present UniMetric, a universal and training-free metric scale recovery framework for geometry foundation models, without additional sensors or task-specific metric supervision. Specifically, Metric Prior Acquisition elicits object height intervals from a vision-language model. Semantic-Geometric Alignment measures reconstructed object heights along the estimated support-surface normal in each frame and converts the priors into scale intervals. Global Scale Recovery reconciles these intervals through a convex objective, yielding a jointly feasible scale when they overlap and a unique compromise otherwise. The estimated sequence-level scale is applied uniformly to reconstructed points and camera translations. Controlled experiments with real VLM priors show that combining observations improves scale accuracy with exact relative object heights. To evaluate metric accuracy on GFM reconstructions without ground-truth scale alignment, we introduce MetricBench, covering twelve embodied, driving, and roadside subsets with sensor-derived ground truth. Across nine geometry foundation models, UniMetric reduces mean depth Abs Rel, reconstruction Chamfer distance, and trajectory error by 54.2%, 83.8%, and 65.6%, respectively, improving every model on all three tasks at the suite level. We release the UniMetric code and MetricBench data.