Learning Scene Coordinate Reconstruction from Unposed Images via Pose Graph Optimization

AI summary

Overview

Research area: Computer vision — learning-based structure-from-motion (SfM), visual localization, and 3D scene reconstruction from unordered, unposed images.

Technical level: Intermediate. The paper blends neural scene coordinate regression with classical pose graph optimization (PGO), so readers benefit from familiarity with camera poses, SE(3), reprojection, and epipolar geometry. The core idea, however, is explainable without the math.

Scope: The paper proposes a hybrid framework that adds uncertainty-aware pose graph optimization on top of ACE-Zero to enforce global multi-view consistency in camera pose estimation from unposed image collections.

What This Paper Is About

ACE-Zero can estimate camera poses and scene coordinates directly from unordered image collections without ground truth supervision, but it refines each pose locally, with no global or multi-view consistency constraint. That makes it prone to pose drift and misalignment in large or ambiguous scenes. The goal of this work is to add a global geometric reasoning stage that corrects this drift, weights unreliable constraints down, and still keeps the pipeline self-supervised.

Key Contributions

1. A hybrid framework that integrates pose graph optimization into ACE-Zero to enforce multi-view and global consistency across all estimated camera poses.
2. A method for constructing pose graphs directly from ACE-Zero outputs — without explicit geometric supervision — by extracting relative pose constraints from geometrically consistent scene coordinate predictions.
3. An uncertainty-aware optimization strategy that assigns confidence to each relative pose constraint using geometric priors (epipolar consistency and optical flow consistency), so unreliable edges are down-weighted during optimization.
4. Reported gains in pose quality (measured indirectly via novel view synthesis PSNR) while requiring on average half the number of iterations compared to ACE-Zero alone, and roughly 30 minutes versus 1 hour on 7-Scenes.

Main Findings

• 7-Scenes (PSNR, dB): The full method averages 21.7 versus 21.2 for ACE-Zero (self-reproduced 21.0), 21.2 for COLMAP (default) and 21.8 for COLMAP (fast). Reported average runtime is 30 min, against 1h for ACE-Zero, 13h for COLMAP (fast) and 38h for COLMAP (default).
• Naive PGO fails: Applying PGO without proper graph construction collapses performance to 12.6 dB on 7-Scenes and 13.8 dB on Mip-NeRF 360, showing that graph construction is essential rather than optional.
• Ablation ladder: Adding graph construction improves accuracy by more than 5 dB; epipolar-based uncertainty adds moderate gains; optical flow further improves robustness. Removing uncertainty altogether gives 18.5 dB on 7-Scenes and 21.0 dB on Mip-NeRF 360.
• Progressive beats one-shot: Running PGO only once at the end underperforms iterative integration (20.5 dB vs. 21.7 dB on 7-Scenes; 23.5 dB vs. 24.3 dB on Mip-NeRF 360).
• Mip-NeRF 360: Full method averages 24.3 dB, close to COLMAP's 24.7 and above ACE-Zero's 22.9 (22.14 self-reproduced). VGGT-SLAM averages 14.3, struggling with large differences between sequential images.
• Tanks and Temples: On the short version, averages are 18.4 (Training), 18.8 (Intermediate) and 16.8 (Advanced) for the full method, versus 17.8, 18.3 and 14.5 for ACE-Zero. On the long version: 19.7, 20.3 and 17.6 versus 19.3, 19.6 and 16.1. COLMAP averages 19.8, 19.9 and 14.5 on the long splits.
• VGGT-SLAM comparison: On the Advanced split of Tanks and Temples, VGGT-SLAM is reported at 13.6 versus ACE-Zero's 14.5 on average. VGGT-SLAM's long-version results are not reported because downsampling to around 4k frames produced fractured scenes.
• Threshold sensitivity: Among fixed geometric consistency thresholds, τ = 0.1 m performs best; τ = 0.2 m admits noisy matches that degrade global consistency. The default pipeline anneals τ from 0.2 m to 0.05 m.
• Symmetric checks: Bidirectional consistency checks give modest gains (under 5%) and can hurt sparser settings by reducing graph connectivity, so one-way checks are used by default.
• Runtime breakdown: The PGO step costs about 30 s per iteration against 150 s for a vanilla ACE-Zero iteration, but total iterations drop by roughly half, yielding a net speedup.

Methodology in Plain English

The starting point is ACE-Zero, which trains a network to predict, for every pixel, the 3D point it corresponds to in a shared scene frame. Those 2D-to-3D correspondences are then turned into camera poses with PnP inside RANSAC. ACE-Zero nudges each pose using only that image's own information, so errors accumulate and different parts of the scene drift out of alignment.

The authors bolt a global stage onto this loop:

• Build the graph. For a pair of images, they take a 3D point predicted in image i, project it into image j using j's current pose, and read off the 3D point predicted at that pixel in j. If the two 3D points agree within a distance threshold τ, the pixel pair counts as a consistent match. These verified matches are fed to RANSAC with PnP to estimate the relative pose between the two cameras. Each image becomes a node; each reliable relative pose becomes an edge.
• Score each edge's reliability. ACE-Zero provides no uncertainty estimates, so the authors derive confidence from two geometric cues. The first is epipolar error: how far the reprojected point sits from the epipolar line implied by the relative pose. The second is optical flow consistency: how far the reprojected point is from the correspondence predicted by RAFT's dense flow. The two errors are combined as a weighted sum with α₁ = 0.4 and α₂ = 0.6, averaged over matches to give one score per edge, and converted into an information matrix of the form 1/(σ² + ε)·I.
• Optimize globally, every iteration. Pose graph optimization then adjusts all camera poses so that the implied relative poses match the measured ones, with Huber loss (scale 1.0) to damp outliers, a tight prior on the first pose (standard deviation 10⁻⁶), between-factors with standard deviation 0.1, and loose priors (standard deviation 10) on unconnected nodes. Levenberg–Marquardt solves it.
• Evaluate indirectly. Because real ground truth poses are hard to obtain, they treat COLMAP poses as pseudo ground truth and judge pose quality by training a Nerfacto model and measuring PSNR on synthesized test views. Experiments run on a single NVIDIA 3090 GPU with ZoeDepth initialization and default ACE-Zero hyperparameters.

Why This Matters

The paper shows that a cheap, classical geometric layer can repair a failure mode of an otherwise self-supervised neural system, and that it can do so without any ground truth poses, depth, or 3D annotations. It also challenges the assumption that learning-based refinement should replace optimization: here the two are complementary, and the optimization stage pays for itself by cutting the number of training iterations roughly in half. For the research community, it opens a route to making unsupervised SfM pipelines scale to larger and more ambiguous scenes.

Real-world applications:

• 3D reconstruction and asset capture from casually photographed, unordered image collections.
• Autonomous navigation and robotics, where globally consistent pose estimates matter for mapping and loop closure.
• Augmented reality, where drift causes virtual content to swim or misalign.
• Visual localization, placing a query image within a previously reconstructed scene.

Industry relevance: the work comes from dConstruct Robotics together with the National University of Singapore, and its emphasis on reduced iteration counts and runtime (30 min vs. 1h on 7-Scenes; 1h vs. 2.2h on Tanks and Temples long splits against ACE-Zero) points at practical deployment rather than benchmark-only gains.

Future Directions

• Replacing the heuristic confidence metrics with learned or probabilistic uncertainty estimates, since the authors note the current metrics may not generalize to highly dynamic or textureless scenes.
• Removing manual hyperparameter tuning, which the authors list as an explicit limitation (α₁, α₂, τ schedule, prior standard deviations).
• Extending the framework to jointly optimize scene structure rather than poses alone, i.e., moving toward full bundle adjustment, which the paper leaves as future work.
• Improving graph connectivity under strict filtering, since symmetric consistency checks trade edge reliability for fewer edges in sparse settings.

Target Audience

Graduate students and researchers in computer vision and robotics working on structure-from-motion, SLAM, visual localization, or neural scene representations; engineers building reconstruction or AR pipelines who want a practical, low-overhead way to add global consistency to a learned pose estimator; and readers interested in hybrid systems that combine neural inference with classical geometric optimization.

Authors’ abstract

Learning-based structure-from-motion methods such as ACE-Zero have demonstrated strong performance in estimating cameraposes and scene coordinates from unordered image collections without requiring ground truth supervision. However, the lack of global and multi-view consistency constraints in ACE-Zero can lead to pose drift and misalignment, particularly in complex or ambiguous scenes. In this work, we propose a hybrid framework that integrates pose graph optimization (PGO) into ACE-Zero to refine camera poses and suppress incorrect refinements. We construct pose graphs directly from ACE-Zero outputs by extracting relative pose constraints from predicted scene coordinates. Furthermore, we introduce an uncertainty-aware optimization strategy by estimating confidence scores using geometric priors, including epipolar and optical flow consistencies across views. Our approach improves the robustness and accuracy of pose estimation, demonstrating that global geometric reasoning can effectively complement learning-based inference in structure-from-motion.