SplatSuRe: Selective Super-Resolution for Multi-view Consistent 3D Gaussian Splatting

AI summary

Overview

Research area: Computer vision, specifically 3D scene reconstruction and novel view synthesis using 3D Gaussian Splatting (3DGS) combined with image super-resolution (SR).

Technical level: Advanced. The paper assumes familiarity with 3DGS rendering, differentiable rasterization, and single-image super-resolution models.

Scope: The paper proposes a selective, geometry-aware super-resolution training framework for 3DGS that applies generative SR detail only in undersampled regions instead of uniformly across every image.

What This Paper Is About

3D Gaussian Splatting produces blurry, over-smoothed, and aliased renders when it is trained only on low-resolution (LR) images, because the model never sees high-frequency detail. A natural fix is to run a super-resolution model on the LR inputs first, but because single-image SR processes each view independently, it hallucinates view-dependent textures that conflict across viewpoints and degrade the 3D model. The goal of this paper is to apply SR only where it is actually needed — in 3D regions that no nearby camera observes at higher frequency — so that renderings become sharper without breaking multi-view consistency.

Key Contributions

1. A per-Gaussian fidelity score that quantifies how well each Gaussian is resolved across views, leveraging low-resolution geometry to estimate how much high-frequency information is already available for that region.
2. A per-view spatial weight map computed from the Gaussian fidelity score that highlights undersampled areas while suppressing SR where low-resolution supervision is already reliable.
3. A selective SR training framework that jointly optimizes a 3DGS model using LR and SR supervision, injecting generative detail only where needed while preserving multi-view consistency.
4. State-of-the-art results across Tanks & Temples, Deep Blending, and Mip-NeRF 360 without adding any neural components or modifying the underlying 3DGS pipeline.

Main Findings

• Super-resolution is not uniformly beneficial: The paper's central result is that applying SR everywhere injects unnecessary inconsistencies in regions already well-supervised by closer LR views, while skipping SR in undersampled regions leaves them blurry. Selecting where SR is applied improves both fidelity and perceptual quality.

• Tanks & Temples results (4× SR, τ = 1.1): SplatSuRe + StableSR reaches SSIM 0.784, PSNR 23.81, LPIPS 0.272, FID 37.72, CMMD 1.040, DreamSim 0.0413, MUSIQ 58.332, and NIQE 3.928. This beats SRGS + StableSR (SSIM 0.771, PSNR 23.32, LPIPS 0.286, FID 49.11, CMMD 1.048, DreamSim 0.0535, MUSIQ 55.209, NIQE 4.633) and Mip-Splatting (SSIM 0.767, PSNR 23.10, LPIPS 0.303, FID 52.46, CMMD 1.137, DreamSim 0.0597, MUSIQ 46.571, NIQE 5.043). Only the LR-trained 3DGS baseline achieves a better NIQE score (3.412).

• Deep Blending results: SplatSuRe achieves the strongest performance across all reported metrics — SSIM 0.872, PSNR 29.01, LPIPS 0.306, CMMD 0.496, DreamSim 0.0330 — compared to Mip-Splatting (0.865, 28.43, 0.327, 0.690, 0.0398) and SRGS (0.861, 28.23, 0.317, 0.630, 0.0409).

• Mip-NeRF 360 is a harder case for SR: SplatSuRe reaches SSIM 0.740, PSNR 26.34, LPIPS 0.323, CMMD 0.339, DreamSim 0.0179, outperforming SRGS on every metric except LPIPS, but both SR-based approaches are surpassed by Mip-Splatting (SSIM 0.759, PSNR 26.48, LPIPS 0.292, CMMD 0.183, DreamSim 0.0132). The authors attribute this to Mip-NeRF 360's smooth circular camera trajectories (dense multi-view coverage, minimal undersampling) and to its LR images being roughly twice as large as those in the other datasets, leaving little room for SR improvement.

• Gains concentrate in foreground regions: The paper reports that improvements are most significant in localized foreground regions where higher detail is desired, and in areas such as text, high-frequency patterns, and distant objects observed in other views, while also reducing Gaussian artifacts seen in other methods.

• SR model choice matters but the method is agnostic: On Tanks & Temples, SplatSuRe improves over SRGS with both SwinIR (PSNR 24.06 vs. 23.84; CMMD 1.135 vs. 1.137) and StableSR (PSNR 23.81 vs. 23.32; CMMD 1.043 vs. 1.048). Gains are larger with StableSR because its higher perceptual quality comes with more multi-view inconsistency. SwinIR achieves higher PSNR due to its conservative reconstruction, but StableSR was chosen for the main experiments.

• Ratio threshold ablation: Introducing a small amount of SR initially improves both PSNR and LPIPS, but applying SR too aggressively degrades performance. The authors select τ = 1.1 as the best trade-off between sharpness and consistency across scenes.

• 8× super-resolution (Appendix): SplatSuRe achieves the best SSIM, PSNR, LPIPS, FID, and DreamSim scores on Tanks & Temples at 8×; its CMMD score ranks fourth. It achieves the best results across all metrics on Deep Blending at 8× and nearly all metrics on Mip-NeRF 360 at 8×, trailing only Mip-Splatting on SSIM.

• Single-stage training is feasible: A unified pipeline (5K LR iterations followed by 25K SR iterations, 30K total) achieves performance comparable to the two-stage formulation within the same training budget as single-stage baselines.

Methodology in Plain English

The authors start from an observation about how cameras sample a scene: a low-resolution photo taken up close often contains more fine detail than a high-resolution photo of the same surface taken from far away. That means some parts of a scene already get high-frequency supervision from nearby views, and other parts do not. Super-resolution should only be used on the second group.

To find that second group, they first train a regular 3DGS model on the low-resolution images to get stable geometry. For each Gaussian, they measure its screen-space radius (how big it appears in pixels) in every training view. If a Gaussian appears large in at least one view and small in others, the close-up view can supervise the distant ones, so no SR is needed. If a Gaussian appears at roughly the same size in every view, no view provides extra detail, so SR is needed. The ratio between each Gaussian's maximum and minimum radius across views captures this.

They convert that raw ratio into a fidelity score in [0, 1] using a sigmoid with an offset threshold τ and smoothness parameter k = 0.05. A key technical detail: 3DGS artificially dilates Gaussians with a fixed low-pass filter (s = 0.3) to prevent aliasing, and the authors exclude that dilation when computing the radii, since it inflates distant Gaussians. Gaussians visible in fewer than three views get a score of zero because they are poorly constrained.

Next, they turn per-Gaussian scores into per-view image weight maps. For each training view, they identify the Gaussians whose largest screen-space radius occurs in that view. The weight map combines two terms: one that lights up where fidelity scores are low (undersampled regions needing SR), and one that lights up where the current view is the closest observer (so no other view can provide higher resolution information there). The map is then normalized so SR loss magnitude stays consistent across views.

Finally, they train a high-resolution 3DGS model with two losses. The LR loss compares the downsampled HR render against the original LR ground truth using the standard 3DGS L1 and D-SSIM combination (with λ = 0.2). The SR loss compares the HR render against a super-resolved image produced by a frozen SR model, with each pixel's contribution scaled by its weight in the map. The two are combined with γ = 0.4 controlling their relative contribution.

Experiments use 4× super-resolution with StableSR and τ = 1.1. Baselines are 3DGS trained on LR inputs, 3DGS + StableSR, Mip-Splatting, and SRGS. Evaluation spans eight metrics: SSIM, PSNR, and LPIPS for reference-based fidelity; FID, CMMD, and DreamSim for distributional and semantic perceptual quality; and MUSIQ and NIQE for no-reference perceptual quality. The authors note that perceptual and distributional metrics internally downsample or resize images before feature extraction, so fine-scale sharpness improvements may be underrepresented by them — which is why they report a broad suite.

Datasets: Tanks & Temples (19 of 21 scenes used; two excluded because COLMAP fails; images downsampled to 240×135 and upsampled 4× to 960×540; roughly 150–500 images per scene with every eighth used for testing); Mip-NeRF 360 (nine scenes — five outdoor, four indoor — about 250–300 images each at approximately 4K×3K, downsampled 8× to about 500×375 and upsampled 4× to about 2K×1.5K); and two scenes from Deep Blending (about 250 images per scene at roughly 1K×1K, downsampled 4× for training and evaluated at original full resolution).

Why This Matters

Impact on research: This paper reframes super-resolution for 3D reconstruction as a spatial allocation problem rather than a uniform enhancement problem. It shows that an explicit geometric signal — camera pose relative to scene geometry — can substitute for learned neural consistency modules, video priors, or diffusion-based consistency enforcement. The method is orthogonal to those techniques and can be combined with them, which opens a research direction in geometry-aware supervision more broadly. It also provides a sober analysis of where SR helps and where it does not, including the finding that in well-sampled scenes like Mip-NeRF 360, anti-aliasing (Mip-Splatting) can beat SR-based methods.

Real-world applications:

• Cultural heritage and archival capture: Digitally reconstructing historic sites or artifacts where only limited or lower-resolution imaging is available, and rendering close-up detail for virtual museum tours.
• E-commerce and product visualization: Turning ordinary-resolution product photography into high-resolution 3D product views that stay consistent as customers rotate the model.
• Real-estate and architectural walkthroughs: Generating high-resolution interior flythroughs from scans shot with modest-resolution cameras, keeping detail consistent across rooms and viewpoints.
• Robotics and simulation: Producing sharp, view-consistent scene representations from onboard low-resolution cameras for training visual policies or building digital twins.

Industry relevance: Real-time rendering pipelines for AR/VR, gaming, and digital twins depend on fast 3DGS-style representations. Requiring every training image to be high-resolution raises capture cost, storage, and processing time. A method that recovers high-resolution detail from LR inputs during training — without adding neural components or changing the underlying 3DGS pipeline, and without extra training at inference time — is directly attractive for teams that already capture many images at moderate resolution. The paper's single-stage variant, which matches single-stage baseline training budgets, removes the main practical objection to a two-stage pipeline.

Future Directions

1. Recovering detail along high-contrast boundaries. The authors note their conservative suppression strategy may miss useful SR refinements, such as stable detail along high-contrast boundaries, where selectively applying SR could further improve quality.
2. Multi-scale SR formulation. The current framework operates at a single upsampling level; a multi-scale version could give finer control over how much SR is integrated and improve sharpness.
3. Better SR backbones for consistency. SplatSuRe is limited by the multi-view inconsistency of the underlying generative SR outputs. Advances in generative SR that reduce inconsistency could be plugged in directly.
4. Adaptive thresholding across scenes. The ratio threshold τ is a scene-dependent hyper-parameter. The paper shows scenes of different geometry and scale behave differently (some improve then drop sharply, some plateau), suggesting the threshold could be inferred per scene rather than fixed.

Target Audience

Researchers and graduate students in computer vision and graphics working on novel view synthesis, 3D Gaussian Splatting, neural radiance fields, and image super-resolution. It is also relevant to practitioners building real-time rendering or 3D reconstruction systems who work with constrained-resolution capture data, and to anyone weighing the trade-offs between anti-aliasing-based and super-resolution-based approaches to high-resolution rendering from low-resolution inputs. Readers without background in differentiable splatting or SR models will find the method section demanding.

Authors’ abstract

3D Gaussian Splatting (3DGS) enables high-quality novel view synthesis, motivating interest in generating higher-resolution renders than those available during training. A natural strategy is to apply super-resolution (SR) to low-resolution (LR) input views, but independently enhancing each image introduces multi-view inconsistencies, leading to blurry renders. Prior methods attempt to mitigate these inconsistencies through learned neural components, temporally consistent video priors, or joint optimization on LR and SR views, but all uniformly apply SR across every image. In contrast, our key insight is that close-up LR views may contain high-frequency information for regions also captured in more distant views and that we can use the camera pose relative to scene geometry to inform where to add SR content. Building on this insight, we propose SplatSuRe, a method that selectively applies SR content only in undersampled regions lacking high-frequency supervision, yielding sharper and more consistent results. Across Tanks & Temples, Deep Blending, and Mip-NeRF 360, our approach surpasses baselines in both fidelity and perceptual quality. Notably, our gains are most significant in localized foreground regions where higher detail is desired.