UrbanGS: A Scalable and Efficient Architecture for Geometrically Accurate Large-Scene Reconstruction

AI summary

Overview

Research area: Computer vision and computer graphics — large-scale 3D scene reconstruction and novel view synthesis, specifically extending 3D Gaussian Splatting (3DGS) to city-scale urban environments.

Technical level: Advanced. The paper assumes familiarity with 3DGS, radiance fields, monocular depth and normal priors, SfM (COLMAP), and TSDF-based mesh extraction.

One-sentence scope: The paper proposes UrbanGS, a framework combining depth-consistent normal regularization, spatially adaptive Gaussian pruning, and a boundary-preserving partitioning scheme to reconstruct large urban scenes with better geometry, lower memory, and faster training than prior 3DGS-based city-scale methods.

What This Paper Is About

3D Gaussian Splatting renders bounded scenes in real time, but extending it to city scale breaks down in three ways: surfaces are geometrically inconsistent, memory use explodes, and block-wise parallel training creates boundary artifacts. UrbanGS attacks all three at once, aiming to keep high rendering fidelity while producing meshes accurate enough for real geometry (verified on the GauU-Scene, Mill-19, and UrbanScene3D datasets). The paper's central technical claim is that supervising rendered depth-derived normals — rather than only rendered normals — is what enables Gaussian positions, not just rotations, to be optimized correctly.

Key Contributions

1. Depth-Consistent D-Normal Regularizer. A regularizer that derives "D-Normals" from the spatial gradient of rendered depth maps and supervises them with pseudo-normal priors, enabling holistic optimization of all Gaussian geometric parameters (position and rotation). The paper argues this fixes the limitation of methods that supervise only rendered normals, which can update rotation but struggle to update position.
2. Adaptive confidence weighting. A geometry-aware confidence term built from depth-gradient cosine similarity and normalized inverse depth deviation, which down-weights unreliable depth predictions and strengthens multi-view geometric alignment.
3. Spatially Adaptive Gaussian Pruning (SAGP). A pruning algorithm aware of local geometric complexity that operates inside voxel cells, combining local geometric complexity, ray-intersection frequency, and visibility-aware importance scores. The paper claims this is the first pruning framework specifically designed for city-scale 3D Gaussian Splatting.
4. Unified partitioning and view assignment scheme. Built on CityGS, it prunes redundant Gaussians before partitioning (so they do not attract non-contributing views), retains common Gaussian primitives at sub-block boundaries to avoid fusion artifacts, and enables parallel refinement across blocks.

Main Findings

• Novel view synthesis gains: On Mill-19 (Building, Rubble) and UrbanScene3D (Residence, Sci-Art), UrbanGS reports the best results among compared methods. Building: SSIM 0.802, PSNR 22.82, LPIPS 0.208. Rubble: SSIM 0.791, PSNR 26.25, LPIPS 0.210. Residence: SSIM 0.823, PSNR 22.48, LPIPS 0.205. Sci-Art: SSIM 0.824, PSNR 22.62, LPIPS 0.279. The paper states it attains the highest PSNR and SSIM in building scenes and reduces LPIPS by 0.006 over CityGS in residential scenes (CityGS Residence LPIPS 0.211 versus 0.205).
• Surface reconstruction gains: On GauU-Scene, UrbanGS reports higher F1 than CityGS-X on all three scenes by improving recall while keeping comparable precision. Residence F1 0.493 (versus CityGS-X 0.456, CityGaussianV2 0.467), Russian Building F1 0.546 (versus CityGS-X 0.542, CityGaussianV2 0.544), Modern Building F1 0.503 (versus CityGS-X 0.487, CityGaussianV2 0.492). NeuS produced FAIL (no valid mesh) and Neuralangelo produced NaN or FAIL across these scenes.
• Memory and model size: In the GauU-Scene comparison, UrbanGS reports Residence at 7.78 M Gaussians / 0.37 G size / 13.2 G memory (versus CityGS 8.05 / 0.44 / 31.5 and CityGS-v2 8.07 / 0.44 / 14.2); Russia at 6.56 / 0.35 / 11.4 (versus CityGS 7.00 / 0.38 / 27.4 and CityGS-v2 6.97 / 0.38 / 15.0); Modern at 7.45 / 0.39 / 15.0 (versus CityGS 7.90 / 0.43 / 29.2 and CityGS-v2 7.90 / 0.42 / 16.1).
• Training speed: UrbanGS completes training on Rubble in 2 hours and 10 minutes, which the paper describes as significantly faster than competing methods. In Figure 1 the authors state that VCR-Gaus fails on A5000 GPUs due to out-of-memory, while UrbanGS reconstructs large scenes.
• SAGP ablation (Russian dataset): Baseline (no SAGP, no partitioning) reaches PSNR 22.54, SSIM 0.778, LPIPS 0.231, F1 0.516, 6.43 M Gaussians, time 235, size 1102.23, and runs out of memory. Adding SAGP reaches PSNR 24.66, SSIM 0.813, LPIPS 0.184, F1 0.546, 2.45 M Gaussians, time 122, size 314.24, memory 14.4. The LightGaussian pruning variant (+LP) reaches 24.53 / 0.785 / 0.195 / 0.528 / 3.02 M / 134 / 467.47 / 17.1, which the paper characterizes as weaker geometry preservation at higher size and memory.
• Partitioning ablation: UrbanGS (with its partitioning strategy ST) reaches F1 0.546 and PSNR 24.66 on Russian; STPG, which substitutes CityGaussian's partitioning with the same SAGP, reaches F1 0.536 and PSNR 24.57.
• Regularization ablation (Modern Building): Removing D-Normal drops to PSNR 25.02, SSIM 0.743, LPIPS 0.215, F1 0.463. Removing depth consistency gives 24.59 / 0.792 / 0.201 / 0.453. Removing geometry-aware confidence gives 26.02 / 0.795 / 0.163 / 0.493. The full model reaches 26.44 / 0.805 / 0.157 / 0.503, which the paper summarizes as F1 improving from 0.453 to 0.503 and PSNR from 24.59 to 26.44.
• Qualitative observations: Figure 3 is reported to show reduced floating artifacts; Figure 4 is reported to show more detailed and clearer surface structures than prior methods.
• Not reported: The paper content provided does not give per-scene camera counts, image counts, or explicit scene-resolution figures, does not report inference/FPS numbers, and does not state the absolute wall-clock training time for methods other than the Rubble comparison.

Methodology in Plain English

The pipeline has four ideas layered together.

1. Supervise normals the indirect way. A pretrained monocular network (Dsine) predicts surface normals from images. Supervising the rendered normal maps with those predictions moves Gaussians toward or away from camera rays, but does not reliably slide them onto the true surface. UrbanGS instead renders a depth map, back-projects it into a point cloud, computes horizontal and vertical finite differences between neighboring points, and takes their cross product to get a "D-Normal." Supervising these D-Normals with the same pretrained normal priors creates a constraint tied to depth, which the authors prove (in an appendix) permits updates to Gaussian positions as well as rotations.

2. Anchor depth to a consistent scale. D-Normals depend on rendered depth being accurate, so the method also supervises depth directly. DepthAnything-v2 supplies dense relative depth from training images; sparse COLMAP points supply metric scale, and per-view scale and shift parameters are robustly fitted to align the two. The loss is computed on inverse depth rather than depth, which balances sensitivity between near and far surfaces. A per-pixel confidence weight, computed from cosine similarity of depth gradients (directional agreement) and normalized inverse depth deviation (magnitude agreement), decays exponentially so unreliable regions contribute less.

3. Prune with local context. Instead of a global opacity threshold, the scene is divided into voxel cells whose size scales with overall Gaussian density (with a slightly enlarged cell factor). Inside each cell, per-Gaussian volume is normalized against roughly the 90th percentile volume and passed through a square root to compress the dynamic range. An importance score multiplies three normalized factors — ray-intersection frequency during training, sigmoid-mapped opacity, and the sub-linear volume weight — so a Gaussian survives only if it is visible, frequently observed, and appropriately scaled. Pruning runs three times during block refinement, at 7k, 15k, and 25k of 30k iterations, plus an initial pass on the coarse global model.

4. Partition and refine in parallel. Starting from coarse global Gaussians, the method prunes first, contracts and partitions the scene into blocks, assigns camera views using geometric and SSIM-based criteria, and refines blocks in parallel on separate GPUs before merging. Boundary Gaussians shared across blocks prevent visible seams.

Final surface meshes are extracted with Open3D's volumetric TSDF fusion over rendered depth maps and camera poses, with Marching Cubes at the zero-level isosurface.

Why This Matters

Impact on research. The paper's central argument — that normal-only supervision cannot properly move Gaussians while depth-derived normal supervision can — is backed by formal propositions (Propositions 1.1, 1.2, 2.1) in the appendix. If it holds generally, it reframes how geometric regularizers should be designed for any splatting-based reconstruction, not just urban scenes. It also offers a template for combining monocular foundation-model priors with SfM scale alignment under learned confidence, which is reusable across reconstruction tasks.

Real-world applications:

• Digital twins and city-scale mapping, where accurate building facades and smooth surfaces matter for downstream analysis.
• Autonomous driving simulation and HD map construction, which need geometry that is metrically plausible, not just photorealistic.
• AR/VR and large-area telepresence, where training memory and runtime determine whether a scene is practical to produce on modest hardware.
• Urban planning, construction inspection, and infrastructure monitoring, where extracted meshes are used for measurement rather than viewed as images.

Industry relevance. The efficiency results matter more than the quality deltas for deployment. UrbanGS is trained on eight RTX A5000 GPUs while baselines used RTX A800 GPUs, and it still reports roughly half the memory (for example 11.4 G versus 27.4 G on Russia) and 2.45 M versus 6.43 M Gaussians in the pruning ablation. The authors explicitly note that VCR-Gaus fails on A5000 GPUs due to out-of-memory, which makes the case that memory, not fidelity, is the deployment bottleneck in this domain.

Future Directions

• Whether the position-update argument generalizes. The D-Normal proof covers the authors' formulation; testing whether it holds for other primitive types (2D Gaussian surfels, mesh-bound Gaussians such as SuGaR) is left open.
• Reliance on external priors. The method depends on DepthAnything-v2 for depth and Dsine for normals, plus COLMAP SfM for scale. The paper does not report behavior when these priors degrade, or on scenes where SfM is sparse or fails.
• Pruning schedule sensitivity. The 7k/15k/25k schedule is motivated by analogy with 3DGS densification dynamics and LightGaussian rather than derived. Whether those points are optimal for other scene scales or capture patterns is not tested in the reported content.
• Evaluation breadth. Meshes are evaluated on three GauU-Scene scenes using precision, recall, and F1. Extending geometric evaluation to the Mill-19 and UrbanScene3D scenes used for rendering, and reporting inference speed, would strengthen the scalability claims. The paper states it will release code and training scripts upon final acceptance, which would make these checks feasible.

Target Audience

Researchers and graduate students working on 3D Gaussian Splatting, NeRF-based reconstruction, and novel view synthesis who want to understand how depth-normal coupling changes geometric optimization. It is also relevant to practitioners in digital twin construction, autonomous driving simulation, and AR/VR content pipelines who need city-scale reconstruction that fits within a fixed GPU memory budget. Readers should already be comfortable with radiance field rendering equations, SfM scale alignment, and mesh extraction, since the paper assumes all three.

Citation: Changbai Li, Haodong Zhu, Hanlin Chen, Xiuping Liang, Tongfei Chen, Shuwei Shao, Linlin Yang, Huobin Tan, Baochang Zhang. arXiv:2602.02089v2 [cs.CV], 20 Feb 2026. License: CC BY 4.0.

Authors’ abstract

While 3D Gaussian Splatting (3DGS) enables high-quality, real-time rendering for bounded scenes, its extension to large-scale urban environments gives rise to critical challenges in terms of geometric consistency, memory efficiency, and computational scalability. To address these issues, we present UrbanGS, a scalable reconstruction framework that effectively tackles these challenges for city-scale applications. First, we propose a Depth-Consistent D-Normal Regularization module. Unlike existing approaches that rely solely on monocular normal estimators, which can effectively update rotation parameters yet struggle to update position parameters, our method integrates D-Normal constraints with external depth supervision. This allows for comprehensive updates of all geometric parameters. By further incorporating an adaptive confidence weighting mechanism based on gradient consistency and inverse depth deviation, our approach significantly enhances multi-view depth alignment and geometric coherence, which effectively resolves the issue of geometric accuracy in complex large-scale scenes. To improve scalability, we introduce a Spatially Adaptive Gaussian Pruning (SAGP) strategy, which dynamically adjusts Gaussian density based on local geometric complexity and visibility to reduce redundancy. Additionally, a unified partitioning and view assignment scheme is designed to eliminate boundary artifacts and optimize computational load. Extensive experiments on multiple urban datasets demonstrate that UrbanGS achieves superior performance in rendering quality, geometric accuracy, and memory efficiency, providing a systematic solution for high-fidelity large-scale scene reconstruction.