VidMap: Exploiting Temporal Structure for Video-Based Structure-from-Motion

AI summary

Overview

• Research area: 3D computer vision—specifically camera pose and calibration estimation from video, at the intersection of Visual SLAM and Structure-from-Motion (SfM).
• Technical level: Advanced. The paper assumes familiarity with bundle adjustment, rotation averaging, global positioning, loop closure, and monocular depth priors.
• Scope: VidMap is an offline system that reconstructs metric camera trajectories from long, uncalibrated videos by combining SLAM-style sequential tracking with SfM-style global optimization.

What This Paper Is About

Estimating camera poses and intrinsics from an arbitrary video is still unreliable because the two dominant families of methods each have a structural flaw. SLAM processes frames causally and incrementally, so early mistakes and tracking failures are locked in and it usually needs known calibration; SfM treats video as an unordered image set, which allows global optimization but throws away temporal ordering and is therefore vulnerable to visual symmetries. VidMap aims to get both: SLAM's sequential reliability and SfM's global, non-causal optimization, applied to long uncalibrated videos with severe motion and visual ambiguity.

Key Contributions

1. A hybrid SLAM/SfM pipeline for uncalibrated video. VidMap performs video-aware extraction (motion-triggered keyframing, dense matching, long sparse tracks) non-causally, then plugs those tracks into a global SfM back end (built on the GLOMAP framework), producing metric reconstructions of long videos with unknown intrinsics.
2. Provenance-aware track establishment and robust losses. Sequential correspondences and loop-closure (LC) correspondences are labeled separately. Instead of merging LC matches into tracks by transitivity the way GLOMAP does, VidMap attaches LC matches as separate observations, so an incorrect loop closure cannot fuse two sequential chains. Rotation averaging then applies a Huber loss to sequential edges and a Cauchy loss to LC edges, and global positioning uses provenance-selected robust losses.
3. Metric monocular depth priors with per-image scale estimation. Monocular depth maps with uncertainties are inserted into global positioning and bundle adjustment, with per-image scale factors optimized jointly with camera centers, 3D points, poses and intrinsics. Unlike MP-SfM, which fixes depth scales after incremental reconstruction, VidMap keeps optimizing them.
4. A dense-matching front end with covariance-aware drift correction. Tracks carry propagated 2D localization covariances; a multi-flow scheme evaluates direct flows from a window of preceding keyframes and accepts a candidate only if it agrees with the sequential prediction, reducing drift while guarding against aliasing. Calibration (GeoCalib perspective fields plus view-graph refinement) is estimated when intrinsics are unknown.

Main Findings

• LaMAR (long indoor phone/AR videos): VidMap reaches windowed AUC of 91.9 / 92.3 / 89.9 / 89.3 / 88.5 at 10 m, 25 m, 50 m, 100 m and full windows without calibration, and 93.2 / 93.0 / 90.5 / 89.9 / 89.7 with ground-truth calibration. The best uncalibrated baseline, DA3-Long, reaches 86.7 / 86.7 / 84.6 / 78.1 / 76.5, and the best calibrated baseline, ViPE, reaches 87.3 / 84.9 / 80.4 / 79.5 / 78.8. GLOMAP and DPV-SLAM drift by an order of magnitude more than VidMap.
• CroCoDL (disaster-site phone and robot videos): uncalibrated, VidMap scores 94.0 / 95.0 / 95.3 / 93.0 / 95.5 on phone sequences and 91.4 / 85.7 / 82.5 / 80.3 on robot sequences; calibrated, 94.7 / 95.8 / 96.3 / 95.6 / 96.6 and 91.5 / 85.2 / 80.6 / 78.2. GLOMAP-RoMA collapses to 12.7 / 10.0 / 10.3 / 20.3 on robot sequences. The paper attributes DA3-Long's degradation here to out-of-distribution depth models.
• ETH3D-SLAM: VidMap is 51.4 / 68.3 / 90.1 / 94.0 / 99.0 (5 cm, 10 cm, 50 cm, 1 m, 10 m) uncalibrated and 69.4 / 79.2 / 92.2 / 95.0 / 99.1 calibrated, ahead of the strongest calibrated baseline DPV-SLAM (62.3 / 72.5 / 87.4 / 91.8 / 98.4).
• EuRoC: VidMap scores 11.5 / 31.6 / 79.1 / 89.4 / 99.0 uncalibrated and 23.0 / 54.6 / 90.9 / 95.4 / 99.6 calibrated, essentially on par with DPV-SLAM (22.5 / 53.5 / 90.6 / 95.3 / 99.5) in the calibrated setting.
• Calibration is the remaining bottleneck on easier motion. The paper reports an 18-point gap between VidMap's calibrated and uncalibrated results on ETH3D-SLAM, and near-parity between the two settings on LaMAR, which it reads as evidence that VidMap estimates accurate calibration.
• Depth priors dominate long-range accuracy. In the ablation, removing depth in global positioning drops LaMAR from 91.9 / 92.3 / 89.9 / 89.3 / 88.5 to 67.6 / 42.8 / 24.9 / 13.6 / 37.7; removing scale optimization gives 69.3 / 44.7 / 25.6 / 13.9 / 39.7; removing metric scale gives 90.2 / 88.0 / 81.9 / 71.7 / 74.1; removing depth in bundle adjustment gives 91.5 / 91.1 / 88.0 / 86.5 / 85.5.
• Temporal structure matters too, but differently per dataset. Removing loop-closure edges costs little on LaMAR (90.8 / 90.3 / 87.6 / 86.6 / 85.5) but causes large drift on ETH3D-SLAM (32.9 / 47.4 / 77.5 / 85.9 / 97.9). Removing provenance-aware losses degrades LaMAR to 88.4 / 85.7 / 80.2 / 77.0 / 79.9, which the paper describes as catastrophic failure from visual aliasing.
• Runtime and memory scale acceptably. Matching is the most expensive component; total runtime grows linearly with the number of keyframes, peak RAM remains practical for longer videos, and VidMap is described as comparably efficient to GLOMAP and substantially faster than COLMAP.

Methodology in Plain English

VidMap runs in two stages. The first stage watches the video in order and decides what to keep. It picks keyframes based on how much the tracked image content has moved rather than at fixed time intervals, computes a monocular depth map per keyframe, and uses a learned dense matcher to move points from one keyframe to the next. Because chaining these flows accumulates drift, it also tests direct flows from several earlier keyframes: it picks the candidate with the smallest accumulated uncertainty, but only if that candidate agrees closely with the simple sequential prediction—otherwise it keeps the sequential one. Tracks are dropped when certainty falls, when they violate epipolar geometry, or when they cluster too tightly. Separately, the system uses image retrieval (MegaLoc descriptors) to find candidate loop closures, matches them densely, and verifies them geometrically. Crucially, these loop-closure matches are never merged into the sequential tracks; they are stored as separate observations with a "loop closure" label, while track points carry a "sequential" label.

The second stage is global optimization inherited from GLOMAP. Rotation averaging solves all camera orientations at once, but treats sequential edges and loop-closure edges with different robust loss functions so that bad loop closures are downweighted rather than allowed to corrupt everything. Global positioning then solves camera centers, 3D points and—this is the addition—a per-image scale factor that rescales the monocular depth prediction, with residuals that penalize disagreement between the optimized depth and the scaled monocular depth. A third term regularizes the log of each scale factor. Bundle adjustment then refines poses, points, intrinsics and the depth scales together, with robust losses annealed progressively across iterations, in the spirit of graduated non-convexity.

When intrinsics are unknown, VidMap estimates focal length with GeoCalib per keyframe, then re-optimizes a single shared focal length over the most confident keyframes, refines it using view-graph calibration on well-conditioned pairs, and falls back to GeoCalib when such pairs are scarce. The refined value initializes bundle adjustment, which optimizes intrinsics jointly with everything else.

Why This Matters

The paper's core claim is structural rather than incremental: it shows that the causal commitment of SLAM and the provenance-blindness of SfM are both avoidable, and that preserving the distinction between sequential and loop-closure evidence is what allows a global, non-causal system to survive visual aliasing and degenerate motion at the same time. Evaluations span four datasets with very different characteristics, including disaster-site robot footage that no baseline handles well, and all hyperparameters are reported as fixed across datasets.

Real-world applications implied by the paper:

• Navigation and robotics: the abstract frames large-scale training data for navigation as the motivation, and CroCoDL specifically covers disaster-site buildings where robots operate.
• Spatial computing and 3D scene modeling: the conclusion points to converting abundant video streams into accurate 3D models.
• Generative scene modeling: also named in the conclusion as a beneficiary of accurate reconstructions.
• Consumer and wearable capture: LaMAR consists of phone and AR device videos, and the paper emphasizes that videos come from non-expert users rather than expert operators.

Industry relevance: the system runs offline, parallelizes over long sequences, and is reported to be comparably efficient to GLOMAP and substantially faster than COLMAP, which matters for anyone processing video at scale. It also does not require calibrated capture hardware, and its accuracy in the uncalibrated and calibrated settings on LaMAR is nearly identical, which lowers the barrier for turning existing video archives into posed 3D data. The code is released at https://github.com/cvg/vidmap.

Future Directions

• Long-range drift. The conclusion identifies mitigating drift over increasingly long sequences as a formidable remaining challenge.
• Matching visual-inertial precision. The authors state that VidMap's geometric precision does not yet meet the strict tolerances of visual-inertial systems, defining a clear accuracy target for follow-up work.
• Where learned priors fail. The paper notes that on EuRoC, grayscale fisheye cameras and motion blur make learned depth and calibration priors less reliable and helpful, leaving open how best to handle such sensing conditions.
• Ablation-driven component design. The ablations show that depth priors dominate on LaMAR while loop closure dominates on ETH3D-SLAM, suggesting further work on adapting the balance between geometric and learned-prior constraints to sequence length and motion conditioning.

Target Audience

Researchers and engineers working on SLAM, structure-from-motion, visual odometry, or large-scale 3D reconstruction from video; practitioners who need metric camera trajectories from uncalibrated consumer or robot footage; and readers interested in how learned components (dense matching, monocular depth, perspective-field calibration)

Authors’ abstract

Accurately recovering the camera's calibration and metric poses for any unconstrained video would unlock large-scale training data for navigation and scene understanding. The dominant approaches to this problem are severely limited: Simultaneous Localization and Mapping (SLAM) is sensitive to initialization and transient failures due to its causal, incremental nature; it is often over-optimized for real-time operation and generally requires known camera calibration; while Structure-from-Motion (SfM) typically forgoes any image ordering, enabling optimal initialization and global optimization, but lacks robustness to visual symmetries and extreme motions. To bridge this gap, we introduce a system that combines the strong sequential constraints of SLAM with the flexibility and global optimization of offline SfM, enabling the metric reconstruction of arbitrary, long, uncalibrated videos. This system leverages recent advances in wide-baseline dense image matching, treats temporal ordering as a first-class citizen for reliable loop closure, and augments global optimization with metric monocular depth priors. As a result, thorough evaluations on diverse, challenging datasets that exhibit extreme motion and visual symmetries reveal that our approach is significantly more robust and accurate than both state-of-the-art SLAM and SfM, classical or learned, with given or unknown camera calibration. The code is publicly available at https://github.com/cvg/vidmap.