AffineGS: Unlocking Accurate Geometry in 3DGS via Spatially Varying Affine Rectification

AI summary

Overview

Research area: Computer Vision — 3D reconstruction and novel view synthesis, specifically geometry improvement in 3D Gaussian Splatting (3DGS) using monocular depth priors.

Technical level: Advanced. The paper assumes familiarity with 3DGS rasterization, monocular depth/disparity estimation, guided filtering, gradient-domain losses, and mesh evaluation protocols.

Scope (one sentence): The paper diagnoses a systematic failure of global scale-and-shift alignment between monocular disparity and rendered 3DGS depth, and replaces it with a per-pixel, edge-aware affine rectification that is supervised through gradient, normal, and metric depth losses.

What This Paper Is About

3D Gaussian Splatting produces highly realistic images, but the underlying 3D geometry is weak — Gaussians overfit to training views, producing floating artifacts and multi-layered surfaces that are poor for mesh extraction. A popular fix is to regularize the optimization with monocular depth priors, which almost all existing methods assume can be aligned to metric depth with one global scale-and-shift transform. The authors show this assumption is wrong: modern monocular estimators such as Depth-Anything-V2, MoGe, and Metric3D-V2 produce region-dependent, piecewise-linear distortions, so a single global transform necessarily warps part of the scene while aligning another part. The goal is to recover accurate geometry from those same monocular priors without any sensor depth.

Key Contributions

1. Diagnosis of piecewise-linear monocular distortion. A statistical analysis on MuSHRoom and ScanNet++ shows that a single global affine fit (K=1) leaves over 40% of the scene misaligned, while partitioning into 3–5 local linear regions raises inlier ratios above 84% and 91% and roughly halves RMSE.
2. Spatially varying affine rectification. The relationship between inverse monocular disparity and rendered depth is modeled as 1/d(u,v) = α(u,v)·𝒟(u,v) + β(u,v), with per-pixel scale α and shift β fields estimated from finite-difference spatial gradients and smoothed by an edge-aware guided filter guided by the disparity image.
3. Gradient-domain, normal-aware regularization. Two losses match the target disparity gradients implied by the rectified model — one against the rendered 3DGS depth, one against depth gradients derived analytically from rendered surface normals and pinhole camera intrinsics — using magnitude MSE plus a cosine-similarity term.
4. Sensor-free geometric fidelity. A metric depth re-alignment loss (via a global least-squares solve for two scalars a, b) anchors the locally corrected depth to the scene's global scale, preventing drift. The method needs no physical depth sensor and does not modify the Gaussian representation or the densification logic.

Main Findings

• Global alignment is statistically inadequate: At a 0.02 residual threshold, a global affine fit gives 58.9% inlier ratio / 0.074 RMSE on MuSHRoom and 58.6% / 0.088 on ScanNet++, versus 84.9% / 0.046 (K=3) and 92.7% / 0.030 (K=5) on MuSHRoom, and 84.6% / 0.056 (K=3) and 91.8% / 0.039 (K=5) on ScanNet++.
• Sparse-view robustness (NeRF-LLFF, k ∈ {2,3,4,5}, averaged over ten random seeds): Mean PSNR is 18.79 / 20.95 / 22.37 / 23.32 dB, i.e. +1.48 / +2.20 / +2.31 / +2.54 dB over the best baseline (DRGS at 17.31 / 18.75 / 20.06 / 20.78). Mean SSIM reaches 0.533 / 0.663 / 0.721 / 0.761 and mean LPIPS 0.235 / 0.164 / 0.134 / 0.111. The paper's Oracle, built from dense pseudo-ground-truth depth, scores 19.88 / 21.76 / 22.83 / 23.93 dB PSNR — the method approaches and occasionally surpasses it.
• Baselines degrade sharply under the same protocol: DropGaussian collapses in the 2-view setting because the strict evaluation builds SfM points only from training views; zero-shot DepthSplat is competitive only on Fern and trails by 4.3 to 5.0 dB in mean PSNR at every view count.
• Depth accuracy on MuSHRoom (left / right splits): Abs Rel of 0.0551 / 0.0490, Sq Rel 0.0176 / 0.0176, RMSE 0.1854 / 0.1953, δ<1.25 of 0.9674 / 0.9729. The directly comparable DN-Splatter (Monodepth) baseline collapses to Abs Rel 0.3542 / 0.3669 with δ<1.25 of 0.5682 / 0.5818, using the exact same prior.
• Approaching sensor supervision: Sensor-based DN-Splatter achieves PSNR 24.58 / 21.89 and Abs Rel 0.0228 / 0.0354 on MuSHRoom; the proposed method reaches 24.49 / 22.07 PSNR and Abs Rel 0.0551 / 0.0490 with no sensor depth, while also beating sensor-free Splatfacto (24.44 / 21.33), 2DGS (22.52 / 21.73), and SuGaR (20.52 / 18.18).
• Mesh quality on MuSHRoom: F-score 0.7097, Chamfer-L1 0.0486, accuracy 0.0535, completion 0.0437, normal consistency 0.8662 — versus SuGaR 0.6378 / 0.0620, 2DGS 0.6039 / 0.0687, Splatfacto 0.5835 / 0.0652, and DN-Splatter (Monodepth) collapsing to 0.2765 / 0.2605. Sensor-supervised DN-Splatter reaches 0.9243 (0.9065 self-reproduced) and MonoSDF 0.9211.
• ScanNet++ mesh results: The qualitative claim is that the method greatly improves over DN-Splatter (Monodepth) and obtains the highest normal consistency among sensor-free methods, but SuGaR and 2DGS obtain lower Chamfer distance and higher F-score. On scene b20a261fdf the method renders depth about 5% shorter than sensor depth, which limits F-score at the 5 cm threshold. The full numeric rows of the ScanNet++ table (including this method's own scores and runtimes) are truncated in the supplied content and are therefore not reported here.

Methodology in Plain English

Instead of asking one global scale and shift to convert a monocular disparity map into metric depth, the authors let both parameters vary across the image. At every optimization iteration they render a depth map and surface normals from the current Gaussians, predict disparity with Depth-Anything-V2-Large, and then solve for the local scale by comparing how fast disparity changes spatially against how fast rendered depth changes spatially. With the scale in hand, the local shift follows directly from the depth–disparity relation. Both fields are clamped to their 5th–95th percentiles to reject outliers, then smoothed with a guided filter that uses the disparity image as its guide — this keeps the fields smooth inside a surface but sharp across object boundaries. The filter radius is large (550–650 pixels on MuSHRoom), and because a guided filter can push values outside the clamp range (even non-positive, which would make the corrected depth undefined), the scale is clamped again afterwards.

Three losses then use these fields. The first compares the gradient the model predicts from rendered depth against the measured disparity gradient, combining a magnitude MSE with a cosine term for direction. The second does the same but replaces rendered depth gradients with gradients computed analytically from the rendered normals and camera intrinsics, which ties the physical orientation of each Gaussian to the local slope of the scene rather than only its position. The third reconstructs a metric depth map from the affine fields, solves a two-scalar least-squares problem to align it to the scene's global frame, and applies an L1 penalty. Weights are λ1=0.1, λ2=0.08, λ3=0.4, scaled by 0.5 and 0.25 on the MuSHRoom kokko and sauna scenes respectively, and by 0.25 on ScanNet++ (trained at half resolution, 960×720). Training runs 30k iterations with unmodified Splatfacto densification, so all gains come from the supervision signal. Scenes are initialized from 1M back-projected points using dataset sensor depths, following DN-Splatter; meshes for the proposed method and DN-Splatter (Monodepth) on MuSHRoom are extracted with Open3D TSDF fusion (voxel size 0.01, truncation 0.03).

Why This Matters

The paper reframes monocular depth from a noisy global signal into a structured, locally linear one, which changes how depth priors should be injected into any radiance-field or Gaussian pipeline. It shows that a sensor-free method can approach sensor-supervised geometry, removing a hardware dependency that limits deployment.

Real-world applications:

• Autonomous driving, where dense depth sensors are costly and geometry must remain stable under sparse views.
• Robotic perception and manipulation, which need reliable meshes rather than only convincing renderings.
• AR/VR content capture, where scanned indoor rooms must be turned into clean, watertight surfaces.
• Low-cost 3D asset creation and digital twins from ordinary RGB video.

Industry relevance: The work comes from dConstruct Robotics and the National University of Singapore, and its practical pitch is direct — the same off-the-shelf monocular model becomes substantially more useful once its distortions are corrected, with no change to the Gaussian representation, the densification schedule, or the capture hardware.

Future Directions

• Residual global bias. On ScanNet++ scene b20a261fdf the rendered depth is about 5% shorter than sensor depth, capping F-score at the 5 cm threshold; removing this low-frequency bias is an open problem.
• Generalization beyond indoor scenes. Evaluation covers NeRF-LLFF, MuSHRoom, and ScanNet++, all indoor or forward-facing captures; outdoor, large-scale, and dynamic scenes remain untested.
• Prior-agnostic validation. The analysis and pipeline are built around Depth-Anything-V2-Large, with MoGe and Metric3D-V2 cited only as examples of the distortion phenomenon; whether one set of hyperparameters transfers across depth networks is unresolved.
• Per-scene tuning and cost. Weights had to be rescaled per scene (0.5 on kokko, 0.25 on sauna and ScanNet++), and per-scene settings are deferred to an appendix, suggesting adaptive weighting and runtime characterization as natural next steps. Runtime for the proposed method is not reported in the supplied content.

Target Audience

Researchers and engineers working on 3DGS, neural surface reconstruction, and depth-guided novel view synthesis; practitioners who need accurate meshes from RGB-only captures; and anyone using monocular foundation depth models as supervision who wants to understand — and correct — their region-dependent distortions. Readers should already be comfortable with Gaussian splatting and gradient-based optimization to get full value.

Authors’ abstract

3D Gaussian Splatting (3DGS) has become the leading approach for photorealistic novel view synthesis, yet its geometric accuracy lags behind its visual quality. Existing depth-guided methods attempt to resolve this by regularizing the optimization with monocular depth priors, typically using a single global scale-and-shift alignment. Crucially, we discover a fundamental flaw in this assumption: modern monocular depth networks systematically exhibit region-dependent, piecewise-linear distortions. This discovery invalidates standard global alignment strategies, which warp scene geometry by enforcing a single transform. To address this, we introduce a spatially varying affine rectification model that corrects monocular disparity into consistent metric depth using regionally smooth, edge-aware fields. We further derive a normal-aware regularizer that uses spatial gradients to couple the rectified disparity with 3DGS depth and normals. Across diverse experimental settings, our sensor-free method shows strong photometric quality of Gaussian Splatting while substantially improving geometric accuracy and outperforming state-of-the art geometry-focused methods.