Geo-NVS-w: Geometry-Aware Novel View Synthesis In-the-Wild with an SDF Renderer

AI summary

Overview

Research area: Computer vision, specifically novel view synthesis (NVS) and neural implicit surface reconstruction from unstructured photo collections ("in-the-wild" data).

Technical level: Intermediate. The core ideas are accessible, but the method relies on Signed Distance Function (SDF) volume rendering, eikonal/curvature regularizers, and octree feature grids, which assume some familiarity with NeRF-family rendering.

Scope: The paper introduces Geo-NVS-w, a framework that adds an explicit SDF-based geometric backbone and a new Geometry-Preservation Loss to in-the-wild novel view synthesis, and reports both image-quality metrics and measured training energy consumption.

What This Paper Is About

Novel view synthesis methods trained on casual tourist photos (such as NeRF-W and Ha-NeRF) can already render plausible new views, but because they represent scenes as volumetric density clouds, they tend to produce semi-transparent artifacts and blurry architectural detail. Separately, the transient-object masking networks these methods use to remove pedestrians and cars can mistakenly erase real, high-detail static structures, degrading the underlying geometry. Geo-NVS-w addresses both problems by grounding the renderer in a learned Signed Distance Function and adding a loss that stops the transient mask from eating into geometrically complex regions.

Key Contributions

1. Octree-accelerated feature-based volume. A dual sparse octree representation with two grids: a foreground grid decoded into an SDF and color, and a separate background grid decoded by a conventional NeRF. Features are queried via trilinear interpolation and decoded by small MLPs, concentrating computation on regions that actually contain geometry and pruning empty space.

2. Geometry-Preservation Loss (GPL). A novel loss that penalizes the transient mask for being active on rays passing through geometrically significant regions. The edge indicator combines a per-ray average eikonal error and an absolute curvature estimate through a calibrated sigmoid, with weights λ_GPL = 0.05, λ_grad = 10.0, and λ_curv = 2.0 used in all experiments.

3. Quantified energy efficiency analysis. GPU power logging is instrumented directly into the training pipeline, logging cumulative usage over time to document the energy–quality trade-off alongside rendering quality.

4. Integration of SDF rendering with in-the-wild modules. NeuS-style SDF rendering is combined with a per-image latent appearance code (inspired by NeRF-W) and an unsupervised segmentation-based transient mask from a lightweight CGNet (following CR-NeRF), forming a complete in-the-wild pipeline rather than a reconstruction-only one.

Main Findings

• Average quality is competitive but mixed. Across four IMC-PT scenes, Geo-NVS-w reaches average PSNR 24.33, SSIM 0.871, and LPIPS 0.179, versus NeRF-W at 24.31 / 0.840 / 0.161 and Ha-NeRF at 23.43 / 0.788 / 0.197. The method leads on average PSNR and SSIM but has a higher (worse) average LPIPS than NeRF-W.

• Strongest gains on Brandenburg Gate. Geo-NVS-w obtains PSNR 25.40, SSIM 0.944, LPIPS 0.158, compared with NeRF-W's 23.98 / 0.915 / 0.198 and Ha-NeRF's 23.45 / 0.811 / 0.247.

• Also ahead on Trevi Fountain and Taj Mahal PSNR. Trevi Fountain: 24.50 / 0.831 / 0.203 versus NeRF-W 23.01 / 0.751 / 0.109 and Ha-NeRF 22.15 / 0.695 / 0.117. Taj Mahal: 24.19 / 0.860 / 0.194 versus NeRF-W 25.15 / 0.833 / 0.195 and Ha-NeRF 22.72 / 0.767 / 0.301. Note that on Trevi Fountain the LPIPS is worse than both baselines, and on Taj Mahal NeRF-W still has the higher PSNR.

• One scene underperforms. On Sacré-Cœur, Geo-NVS-w records 23.23 / 0.850 / 0.160, below both Ha-NeRF (25.40 / 0.877 / 0.124) and NeRF-W (25.11 / 0.859 / 0.141) on all three metrics.

• Substantial energy savings. A 300,000-iteration run consumes approximately 2.05 kWh for Geo-NVS-w, versus 9.7 kWh for NeRF-W and 7.71 kWh for Ha-NeRF. The abstract describes this as a 4–5× reduction in energy consumption compared to similar methods, which the authors attribute to an architecture permitting significant downsizing of the MLPs.

• Evaluation used all currently available IMC-PT scenes. Four scenes are used, the same ones used in NeRF-W and Ha-NeRF; the paper states the remaining scenes have been removed due to data inconsistencies. CR-NeRF is treated as complementary, with a controlled comparison listed as future work. The number of input images per scene is not reported.

• Hardware and implementation. Implemented in PyTorch and trained on a single NVIDIA A10G GPU using mixed-precision computation.

Methodology in Plain English

The scene is stored twice in sparse octree feature grids. Points inside a foreground unit sphere are looked up in an SDF grid; points outside it are looked up in a background grid handled by a standard NeRF. Trilinear interpolation pulls features from these grids, and small MLPs decode them into an SDF value and a color.

To turn the SDF into an image, the authors use the NeuS formulation. For each sample along a camera ray, the SDF value is converted into an occupancy value using a sigmoid difference, with a term that corrects for the angle between the ray and the surface normal, plus a learned global deviation produced by a small "Deviation Network." These occupancies become alpha-compositing weights, so rendered color is tightly tied to the actual surface, concentrating color integration into a narrow band around the surface rather than smearing it through a density cloud. Normals and derivatives are computed with finite differences.

Two modules handle in-the-wild variability: a per-image latent appearance code to absorb lighting and camera differences, and a CGNet-based transient mask to suppress moving objects such as pedestrians and cars.

The central new idea is the Geometry-Preservation Loss. Because transient masks are trained with photometric loss alone, they tend to also mask out genuinely static, high-detail edges, which harms the SDF. The loss multiplies the predicted mask value for each ray by an edge indicator built from that ray's average eikonal error and absolute curvature, so rays passing through sharp or high-curvature structure are pushed to keep mask values at zero. Training minimizes a composite objective combining photometric L1 loss, eikonal loss, curvature loss, a regularized transient-mask loss, the Geometry-Preservation Loss, a Lipschitz loss on the color network, an off-surface penalty, and a sphere-initialization loss used during a warm-up phase.

Efficiency is measured by embedding GPU power measurement in the training pipeline and logging cumulative energy over time.

Why This Matters

For research, the paper is a data point that geometry-aware neural rendering remains competitive against newer paradigms such as Gaussian Splatting for in-the-wild view synthesis, and it reframes the transient-mask erosion problem explicitly as a geometry-preservation issue with a concrete, differentiable loss. It also contributes a reproducible style of energy accounting that most NVS papers omit, which matters for comparing methods on real deployment budgets rather than only on image metrics.

Real-world applications the paper motivates:

• Virtual reality. Exploring a 3D scene by rendering photorealistic views from arbitrary viewpoints.
• Digital heritage. Reconstructing and viewing landmarks from archived tourist photography, which is exactly the IMC-Phototourism setting used here.
• Visual effects. Generating camera paths and views through locations that were only ever photographed casually rather than captured with controlled rigs.
• Content creation and mapping pipelines. Turning existing unstructured photo collections into navigable 3D representations with lower compute cost.

Industry relevance is driven by the efficiency result: reaching peak PSNR at roughly 2.05 kWh on a single NVIDIA A10G GPU rather than 9.7 kWh or 7.71 kWh lowers the cost of training and makes on-premises or smaller-scale production workflows more practical.

Future Directions

• Integrating 3D Gaussian Splatting in in-the-wild settings. The authors explicitly propose leveraging its speed advantages to further reduce rendering latency and computational overhead while keeping the surface coherence their SDF method enables.
• A controlled comparison with CR-NeRF. CR-NeRF is currently viewed as complementary, and the paper states a controlled comparison is planned.
• Broadening evaluation beyond four scenes. Only the four currently available IMC-PT scenes are used, and the paper notes the rest were removed for data inconsistencies; the number of images per scene is not reported.
• Explaining and fixing the Sacré-Cœur and LPIPS regressions. Geo-NVS-w falls below both baselines on Sacré-Cœur across all three metrics and has a worse average LPIPS than NeRF-W (0.179 versus 0.161), which the paper does not analyze in depth.

Target Audience

Researchers and graduate students working on neural rendering, novel view synthesis, and implicit surface reconstruction, particularly those focused on unconstrained or in-the-wild photo collections. It is also useful for practitioners evaluating the compute and energy cost of training NeRF-family models on limited hardware, and for applied teams in VR, digital heritage, and visual effects who need geometry-consistent rendering from casual photography. Readers without prior exposure to SDF volume rendering or eikonal regularization will need to consult the cited NeuS and Neuralangelo work first.

Authors’ abstract

We introduce Geo-NVS-w, a geometry-aware framework for high-fidelity novel view synthesis from unstructured, in-the-wild image collections. While existing in-the-wild methods already excel at novel view synthesis, they often lack geometric grounding on complex surfaces, sometimes producing results that contain inconsistencies. Geo-NVS-w addresses this limitation by leveraging an underlying geometric representation based on a Signed Distance Function (SDF) to guide the rendering process. This is complemented by a novel Geometry-Preservation Loss which ensures that fine structural details are preserved. Our framework achieves competitive rendering performance, while demonstrating a 4-5x reduction reduction in energy consumption compared to similar methods. We demonstrate that Geo-NVS-w is a robust method for in-the-wild NVS, yielding photorealistic results with sharp, geometrically coherent details.