Scene Coordinate Reconstruction Priors

AI summary

Overview

• Research area: Computer vision — 3D reconstruction, structure-from-motion (SfM), visual relocalization, and scene coordinate regression (SCR).
• Technical level: Advanced. The paper assumes familiarity with SCR pipelines (ACE/ACE0/GLACE), RANSAC + PnP pose estimation, and denoising diffusion models/score matching.
• Scope in one sentence: The paper reformulates scene coordinate regression training as maximum likelihood learning so that high-level "reconstruction priors" — depth-distribution priors and a 3D point cloud diffusion prior — can be added as regularizers to produce more coherent scene geometry, better camera poses, and better downstream novel view synthesis and relocalization.

What This Paper Is About

SCR models learn a scene-specific neural network that maps image patches to 3D scene coordinates, but they are trained only on images of one scene, so when multi-view constraints are weak (texture-poor walls, repetitive structures, reflections), the learned geometry partly degenerates into noisy or dispersed points. The authors reinterpret SCR training probabilistically so that a prior over scene coordinates can be optimized jointly with the reprojection error, then test both hand-crafted priors over depth value distributions and a learned 3D point cloud diffusion prior trained on a corpus of indoor scans. The goal is a more plausible scene layout, higher registration rates in SfM, better pose estimates, and no penalty at query time.

Key Contributions

1. A probabilistic reformulation of SCR training as maximum likelihood learning, where the standard reprojection loss is the log-likelihood term and regularization is written as the negative log-likelihood of scene coordinates, -log p(y), allowing arbitrary reconstruction priors to be optimized jointly rather than as a one-off initialization.
2. Hand-crafted depth distribution priors that fit a Laplace distribution to real depth values — one using the per-point negative log-likelihood (Laplace NLL) and one using a Wasserstein distribution loss between the mini-batch of predicted depths and the target Laplace (Laplace WD).
3. A learned high-level prior: a 3D point cloud diffusion model (PVCNN-based, with diffusion timestep embedding modifications from prior work) trained on roughly 700 indoor scenes, whose noise estimate is used as the score/gradient of the log-likelihood of predicted scene coordinates during SCR mapping.
4. Demonstrated integration of these priors into ACE, ACE0 and GLACE, plus as a byproduct an RGB-D variant in which measured depth maps become a narrow Laplace prior, giving effective RGB-D versions of ACE and ACE0.

Main Findings

• Better SfM on ScanNet: With the diffusion prior, ACE0 registration rate rises from 98.1% to 98.6%, median pose error improves from 19.7 cm / 9.0° to 18.8 cm / 8.9°, and the harder 60/60 Splatfacto split improves from 22.3 dB to 22.4 dB.
• Depth priors help pose quality on ScanNet: Laplace NLL improves ATE/RPE from 26.6 / 4.0 to 25.4 / 3.5 and median error to 17.5 cm / 8.8°; Laplace WD reaches 17.5 cm / 6.8°, but shows "a noticeable drop in PSNR for the 60/60 evaluation split" (22.3 dB to 21.7 dB).
• Largest gains on Indoor6: ACE0 alone registers 57.1% of images at 13.5 dB PSNR; the diffusion prior raises registration to 61.8% (+4.7%) and PSNR to 14.6 dB (+1.1 dB); both depth distribution priors reach 14.1 dB (+0.6 dB) with registration rates of 58.0% (NLL) and 57.7% (WD).
• RGB-D prior is a strong win on ScanNet: ACE0 + Laplace NLL (RGB-D) reaches 98.9% registration, ATE/RPE 18.3 / 3.5 cm, median error 12.8 cm / 4.4°, and 22.9 dB on the 60/60 split — better than the ACE0 + DSAC* Loss RGB-D baseline (96.2%, 29.2 / 6.0, 20.9 cm / 5.9°, 21.9 dB), which the authors say performs worse than default ACE0.
• Relocalization on 7Scenes improves: ACE + Diffusion reaches 97.7% average accuracy versus 97.1% for ACE, and improves the hardest scene, Stairs, by 4.1% (86.2% versus 81.9%). ACE + Laplace NLL averages 97.3% (Stairs 84.2%) and ACE + Laplace WD 97.2% (Stairs 83.2%). GLACE + Diffusion averages 95.9% versus 95.6% for GLACE, with Stairs improving from 71.2% to 73.6%.
• Indoor6 relocalization gains are modest given variance: At batch size 5,120, GLACE goes from 44.2% (±1.8%) to 46.4% (±1.9%) and ACE from 36.2% (±1.5%) to 37.5% (±1.8%); at batch size 51,200, GLACE goes from 69.5% (±1.4%) to 69.6% (±2.0%) and ACE from 57.2% (±1.6%) to 57.9% (±1.1%). Results are means over 5 runs.
• Mapping time cost is modest: Depth distribution priors slightly reduce mapping time (e.g., ACE relocalization mapping 5 min to 4.5 min); the diffusion prior adds about 3 minutes (ACE relocalization 5 min to 8 min; GLACE 6 min to 9 min). Query-time efficiency is unaffected because priors are only applied during training.
• Reference point on pose quality: BundleFusion, an RGB-D SLAM system used for pseudo ground truth, achieves 22.2 dB on the 60/60 Splatfacto split, which the paper notes is lower than some of their results.
• The generative prior is coarse but useful: Generated point clouds lack fine details but represent sensible room layouts, which the authors say is sufficient to regularize SCR training.

Methodology in Plain English

The authors start from the standard ACE recipe: a frozen, scene-agnostic feature extractor feeds a small scene-specific regression head that predicts a 3D point for each image patch, and training minimizes the reprojection error of those points against known camera poses. They show that this reprojection loss is really the log-likelihood term of a Bayesian posterior over scene coordinates, so adding any prior on the point cloud is just adding its negative log-likelihood as a regularizer — trained jointly with reprojection instead of only used to initialize.

Three priors follow from this. The first fits a Laplace distribution to real depth values from held-out scenes (mean 1.73 m, bandwidth 60 cm) and either penalizes each predicted point's depth by its negative log-likelihood or measures the Wasserstein distance between the batch of predicted depths and that distribution. The second replaces the fitted distribution with each pixel's measured depth when RGB-D input is available, using a tolerance of 10 cm. The third trains a point cloud diffusion model offline on the 706 ScanNetV2 training scenes: 5,120 points sampled per iteration, 200 diffusion timesteps, batch size 16, 100,000 iterations on a single V100 GPU, AdamW with a learning rate decaying linearly from 0.0002 to 0. At mapping time the frozen diffusion model's noise prediction is used as the gradient of the log-likelihood of the predicted points, applied only after iteration 5,000, with the diffusion timestep interpolated from T/20 down to 0 and skipped for points whose reprojection error is below 30 pixels (assumed to already be well constrained). Gradient magnitude is normalized as in DiffusioNeRF with a weight of 1,000 and a 1,000-iteration warm-up.

Evaluation uses ACE0 for SfM on the first 20 ScanNetV2 test scenes and the Indoor6 mapping sequences (reporting registration rate, ATE/RPE and median error against BundleFusion pseudo ground truth, and Splatfacto PSNR under both a 1/7 and a harder 60/60 train/test split), and uses ACE and GLACE for relocalization on 7Scenes and Indoor6 (reporting the percentage of images localized within 5 cm / 5°).

Why This Matters

Impact on research: The paper shows that scene-agnostic data can be distilled into SCR training as high-level priors for the first time, rather than only being used in test-time components as in prior work. It also demonstrates that a point cloud diffusion model trained on relatively little data (around 700 scenes) is useful as a regularizer even though it cannot generate high-fidelity scenes, and it reframes existing SCR regularization (such as ACE's initialization loss) as a special case of a much broader maximum-likelihood design space.

Real-world applications:

• Augmented and virtual reality headsets that must relocalize a device from camera images against a previously mapped room.
• Indoor mapping and digital twin creation from casually captured phone or robot imagery, where texture-poor walls and floors are common.
• Robotics and autonomous navigation, where a robot needs a coherent map and reliable pose in an indoor space.
• Novel view synthesis and 3D content capture, since better estimated camera poses directly improve downstream re-rendering quality.

Industry relevance: Visual relocalization and on-device mapping are core capabilities for AR platforms, and the authors are affiliated with Niantic Spatial and the University of Oxford. Because the priors are applied only during training, deployment latency and model size are unchanged, which matters for on-device systems; map sizes in the reported relocalization results are 4 MB for ACE variants and 9 MB for GLACE.

Future Directions

• Extending the approach to outdoor scenes, which the authors say needs better models of depth distribution and more diverse data for diffusion training to learn a prior over larger areas (results on a few outdoor scenes are in the supplement).
• Designing a more expressive point cloud encoding network for higher-fidelity generation while keeping it efficient enough to be practical during mapping.
• Adding a conditional signal to the diffusion model as an additional research direction.
• Addressing the observed variance in Indoor6 relocalization results to draw firmer conclusions about the priors on large multi-room scenes.

Target Audience

Researchers and practitioners in 3D computer vision, visual relocalization, and SfM who already know how SCR pipelines such as ACE, ACE0 and GLACE work, and who are interested in how generative models or probabilistic formulations can act as training-time regularizers. It is also relevant to engineers building AR/VR, robotics, or indoor mapping systems who want better pose and geometry quality without increasing query-time cost; readers without a background in SCR, RANSAC/PnP, or diffusion models will find the method sections hard going.

Authors’ abstract

Scene coordinate regression (SCR) models have proven to be powerful implicit scene representations for 3D vision, enabling visual relocalization and structure-from-motion. SCR models are trained specifically for one scene. If training images imply insufficient multi-view constraints SCR models degenerate. We present a probabilistic reinterpretation of training SCR models, which allows us to infuse high-level reconstruction priors. We investigate multiple such priors, ranging from simple priors over the distribution of reconstructed depth values to learned priors over plausible scene coordinate configurations. For the latter, we train a 3D point cloud diffusion model on a large corpus of indoor scans. Our priors push predicted 3D scene points towards plausible geometry at each training step to increase their likelihood. On three indoor datasets our priors help learning better scene representations, resulting in more coherent scene point clouds, higher registration rates and better camera poses, with a positive effect on down-stream tasks such as novel view synthesis and camera relocalization.