Do Flat Minima Improve Sparse Novel View Synthesis?

AI summary

Overview

• Research area: Computer vision — sparse novel view synthesis (3D Gaussian Splatting and Neural Radiance Fields) combined with optimization and generalization theory (loss-landscape sharpness, flat minima, Sharpness-Aware Minimization).
• Technical level: Intermediate. Readers need some familiarity with radiance fields, 3D Gaussian Splatting, and sharpness-aware optimization to follow the loss-landscape arguments, though the core idea is explained intuitively.
• Scope: This paper investigates whether flatter minima always improve generalization in sparse novel view synthesis, finds that the answer depends on local image detail, and proposes an optimization strategy that regularizes loss sharpness differently across high-detail and low-detail regions.

What This Paper Is About

Novel view synthesis methods such as 3D Gaussian Splatting produce high-quality images when many input views are available, but in sparse-view settings they overfit the training views and fail on unseen viewpoints, generating unresolved details and floating artifacts. Most prior work attacks this by adding geometric priors or external models; this paper instead studies the training objective itself, asking whether the well-known deep learning belief that "flatter minima generalize better" holds for novel view synthesis. The authors show it does not hold uniformly, and that the desirable amount of loss sharpness depends on the local structure of the image being reconstructed.

Key Contributions

1. A fundamental investigation linking loss landscape to generalization in novel view synthesis. The authors state this is the first fundamental investigation into this relationship, and they reveal that flat minima across all Gaussian parameters is suboptimal because the optimal loss sharpness varies with the local structure of the input images.
2. A novel optimization strategy: Structure-Aware Sharpness Regularization. Based on the insight above, the method introduces structure-aware sharpness defined within structure-adaptive neighborhoods and adaptively adjusts the sharpness regularization weight, encouraging flatness for generalization while preserving the loss sharpness needed to reconstruct fine details.
3. Demonstrated generality across representations and optimizers. The strategy improves a wide range of 3DGS-based frameworks, other flat-minima optimizers (Random Weight Perturbation), implicit NeRF representations (FreeNeRF), and dynamic scenes, suggesting it is a broadly applicable optimization principle rather than a method-specific fix.
4. Faster variant for practical use. The appendix introduces "Ours-L," which applies the method only during the final 12.5% of total iterations, trading a small amount of performance for a large reduction in training-time overhead.

Main Findings

• Flat minima do not always generalize better in novel view synthesis. In Figure 1, Sharpness-Aware Minimization (SAM), a representative flat-minima optimizer, yields the lowest loss sharpness yet a higher test error than the authors' method — challenging the common belief that flatter minima generalize better.
• The optimal sharpness depends on local image structure. High-detail regions (e.g., edges) inherently require a sharp loss landscape for accurate reconstruction, while low-detail regions benefit from a flat landscape for generalization. Pursuing flat minima everywhere over-penalizes high-detail regions and under-penalizes low-detail regions.
• Consistent gains across all baselines and datasets. Table 1 reports improvement on every metric, dataset, and baseline. On LLFF (3 views) 3DGS improves from PSNR 19.810 ± .339 to 20.783 ± .300, SSIM .6790 ± .0078 to .7197 ± .0032, and LPIPS .2145 ± .0065 to .1965 ± .0034. On MipNeRF-360 (12 views), 3DGS improves from PSNR 18.903 ± .179 to 19.303 ± .185, SSIM .5499 ± .0036 to .5622 ± .0051, and LPIPS .3734 ± .0042 to .3552 ± .0047.
• Improvements hold for state-of-the-art and prior 3DGS variants. CoR-GS improves from 20.185 ± .142 to 20.862 ± .154 PSNR on LLFF and 19.515 ± .243 to 19.805 ± .233 on MipNeRF-360; DropGaussian from 20.461 ± .212 to 20.853 ± .227 and 19.514 ± .199 to 19.625 ± .254; NexusGS from 21.048 ± .049 to 21.348 ± .078 and 18.506 ± .098 to 18.736 ± .103; SE-GS from 20.725 ± .217 to 21.141 ± .223 and 19.931 ± .288 to 20.135 ± .232.
• Naively applying SAM to 3DGS hurts. In the ablation (Table 2), plain 3DGS scores PSNR 19.810 ± .339 / SSIM .6790 ± .0078 / LPIPS .2145 ± .0065, while adding SAM alone gives 20.198 ± .218 / .6958 ± .0034 / .2095 ± .0036 — but visual results become blurry because Gaussians are perturbed uniformly regardless of image structure.
• Both components are needed. SAM + structure-aware regularization weight gives 20.570 ± .161 / .6980 ± .0037 / .2142 ± .0042; SAM + structure-aware sharpness gives 20.560 ± .259 / .7116 ± .0038 / .2023 ± .0032; using both gives the best result of 20.783 ± .300 / .7197 ± .0032 / .1965 ± .0034.
• The method converges to less flat minima than SAM yet generalizes better. On the LLFF fern scene, SAM's local loss range between maximum and minimum is 2.37× smaller and its measured loss sharpness (maximum Hessian eigenvalue λ_max) is 1.49× smaller than 3DGS; SAM's test loss is 0.0129 lower, narrowing the generalization gap from 0.0985 to 0.0809. The authors' method's local loss range is only 1.11× smaller and λ_max only 1.01× smaller than 3DGS, yet its test loss is 0.0141 lower and it narrows the gap further to 0.0805, outperforming SAM.
• Gains grow where fewer training views see the region. Using covisibility maps computed with MASt3R following CoMapGS, the method's improvement over 3DGS on LLFF increases as covisibility falls: covisibility 3 goes from .0483 ± .0093 to .0417 ± .0080 (Δ -.0066 ± .0043), covisibility 2 from .0757 ± .0156 to .0644 ± .0137 (Δ -.0114 ± .0067), and covisibility 1 from .0948 ± .0256 to .0806 ± .0232 (Δ -.0142 ± .0074).
• Works with a different flat-minima optimizer. Replacing SAM with Random Weight Perturbation (RWP), 3DGS + RWP improves from 20.079 ± .278 to 20.650 ± .280 PSNR on LLFF and from 19.090 ± .187 to 19.284 ± .193 on MipNeRF-360, with RWP's blurry outputs becoming sharper.
• Works in dynamic, temporally sparse scenes. Applied to Yun et al. on the Neural 3D Video dataset with the 3DGStream backbone, PSNR rises from 32.542 to 32.622, SSIM from .9486 to .9497, and masked total variation (mTV, a temporal-consistency measure) drops from .1109 to .0989.
• Works on implicit NeRF representations too. Applied to FreeNeRF on LLFF, PSNR improves from 19.523 to 19.584, SSIM from .6063 to .6182, and LPIPS from .3103 to .2983. The authors note this gain is smaller than for 3DGS baselines due to inherent differences in the representation.
• A late-stage variant is much cheaper. The full method averages AVGE .0979 (change of -.0132 versus 3DGS's .1111) but takes 237 seconds versus 3DGS's 85.7 seconds (2.77× slower). Ours-L reaches AVGE .1016 (change of -.0095) in 98.2 seconds, only 1.15× slower than 3DGS.

Methodology in Plain English

The authors build on two existing tools: 3D Gaussian Splatting (which represents a scene as a set of 3D Gaussians with mean, rotation, scale, opacity, and spherical-harmonic color parameters) and Sharpness-Aware Minimization (which improves generalization by perturbing parameters within a radius ρ and minimizing the resulting worst-case loss, equivalent to penalizing loss sharpness).

Their change is to make both the perturbation radius and the regularization strength depend on where a Gaussian lands in the image:

1. Geometric tolerance map. For each training image, they compute an image-gradient threshold to find high-detail pixels (edges, fine textures). For every pixel they record the distance to the nearest high-detail pixel. Pixels in flat regions get large values; pixels near edges get small values. This distance is interpreted as how much geometric movement a primitive can afford before the rendered image visibly changes.
2. Structure-aware sharpness. Instead of computing one gradient over all parameters, they compute the perturbation per Gaussian. Each Gaussian's tolerance is looked up at its projected 2D center, then converted into a 3D displacement by scaling with depth d_i divided by focal length f. The perturbation radius for that Gaussian becomes γ_i · d_i / f · ρ_θ. High-detail Gaussians get small perturbations (avoiding overshoot past the intended local maximum); low-detail Gaussians get large ones (to properly ascend to the local maximum).
3. Structure-aware regularization weighting. The regularization weight is set as γ̄_i = 0.95 · γ_i / γ_max, where γ_max is the maximum distance and 0.95 is determined empirically. When the weight is near 0 the sharpness term vanishes and only the empirical loss is minimized; when near 0.95 the sharpness term dominates. So high-detail regions keep their sharpness, and low-detail regions get strongly flattened.
4. Targeted parameters. This strategy is applied to the Gaussian mean, rotation, and scale — the geometric attributes — with additional hyperparameters needed for scale and rotation in practice. The appendix formalizes why the neighborhood radius should behave this way, assuming an ideal fitted solution and bounding how much the image (and therefore the loss) can change inside the tolerance neighborhood.

Evaluation uses LLFF (3 input views) and MipNeRF-360 (12 input views) at 8× downsampled input resolution, measuring PSNR, SSIM, LPIPS (computed with a VGG network), and Average Error (AVGE), the geometric mean of the three. Because 3DGS is stochastic, they run ten runs on LLFF and five runs on MipNeRF-360.

Why This Matters

Impact on research. The paper challenges a near-universal assumption in optimization research — that flatter minima are always better — by showing a task where the optimal sharpness is spatially varying. It reframes several existing tricks in radiance field training (random noise injection into Gaussian parameters or query locations, worst-case loss objectives such as Sparfels) as implicit, inefficient forms of sharpness regularization, and offers a principled explanation for why they help. It also opens a research direction on the loss landscape of novel view synthesis specifically, rather than generic deep learning.

Real-world applications (sparse novel view synthesis is the enabling technology; the paper does not itself name product applications):

• Capturing 3D scenes from a handful of photos rather than a dense multi-camera rig, lowering the cost of 3D content creation.
• Immersive AR/VR and telepresence, where headset rendering needs to look correct from viewpoints never captured.
• Digitizing real objects and environments for e-commerce, cultural heritage, or digital twins using consumer cameras.
• Robust 3D perception for robotics and autonomous systems operating where only limited sensor viewpoints are available.
• Dynamic video-based reconstruction (the paper's temporal experiments) for volumetric video and free-viewpoint replay.

Industry relevance. The method is applied as a plug-in optimizer with no architectural change and no extra priors, and it is shown to work on top of already-strong published systems (NexusGS, SE-GS, CoR-GS, DropGaussian) as well as on NeRF pipelines. The "Ours-L" variant, which cuts the overhead of computing the loss gradient twice per iteration from 2.77× to 1.15× of baseline training time, addresses the main practical objection to SAM-style methods in a time-sensitive production setting.

Future Directions

• Reducing the remaining computational overhead. The authors show the strategy is most effective when applied to late training iterations but is still influential when applied early, and they attribute this to densification dynamically changing the number of parameters. A better schedule, or applying it selectively during densification, remains open.
• Hyperparameters for scale and rotation. The appendix notes that the analysis is cleanest for the Gaussian mean and applies approximately to scale and rotation, requiring additional hyperparameters in practice — a natural target for a fully parameter-agnostic formulation.
• Moving beyond the SAM and RWP instantiations. The authors position the insight as a general principle and demonstrate it on SAM, RWP, FreeNeRF, and dynamic Gaussians, but the space of sharpness estimators and radiance field representations is far from exhausted.
• Transferring the insight to other under-constrained reconstruction problems. The core claim — that desirable sharpness varies with local detail — is not specific to view synthesis, and the paper explicitly invites the community to explore the loss-sharpness/generalization link further.

Target Audience

Researchers and graduate students working on 3D Gaussian Splatting, NeRF, and sparse-view novel view synthesis; optimization and generalization-theory researchers interested in counterexamples to the flat-minima hypothesis; and practitioners who want a drop-in training modification that improves reconstruction without new priors, new architectures, or external foundation models. Readers already comfortable with 3DGS parameterization, camera projection geometry, and sharpness-aware optimization will get the most out of the technical sections, while the introduction and qualitative results are accessible to a broader computer vision audience.

Authors’ abstract

Despite the success of recent novel view synthesis methods, they tend to struggle in sparse-view settings. This poor generalization to unseen viewpoints is an inherent challenge when training with limited data. To address this, we investigate the relationship between loss sharpness and generalization in novel view synthesis-an underexplored direction. Interestingly, while pursuing flatter minima is widely known to improve generalization in deep learning, reducing loss sharpness is not always beneficial in novel view synthesis. We demonstrate that this difference arises because high-detail regions inherently require a sharp loss landscape for accurate reconstruction, whereas low-detail regions benefit from a flat loss landscape for improving generalization. Based on this insight, we introduce structure-aware sharpness, defined within structure-adaptive neighborhoods, and propose to adaptively adjust the sharpness regularization weight according to the local image structure. This strategy encourages flatter minima for generalization while preserving the loss sharpness necessary to reconstruct fine details. Across various datasets and configurations, our strategy consistently improves a wide range of baselines. Code is available at https://bbangsik13.github.io/FASR.