IE-SRGS: An Internal-External Knowledge Fusion Framework for High-Fidelity 3D Gaussian Splatting Super-Resolution

AI summary

Overview

Research area: Computer Vision — 3D scene reconstruction and novel view synthesis, specifically super-resolution for 3D Gaussian Splatting (3DGS).

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, Mip-Splatting, image/video super-resolution models, depth estimation, and the loss functions and rendering pipeline used in this literature.

Scope (one sentence): The paper introduces IE-SRGS, a framework that reconstructs high-resolution 3D Gaussian Splatting models from low-resolution multi-view inputs by fusing "external" priors from 2D super-resolution and depth models with "internal" priors from a multi-scale 3DGS model, combined through a mask-guided fusion strategy.

What This Paper Is About

Reconstructing a high-resolution 3D Gaussian Splatting model from low-resolution multi-view images is hard, because low-resolution inputs lack fine textures and accurate geometry. Existing methods typically use pre-trained 2D super-resolution (2DSR) models to generate pseudo high-resolution supervision, but those 2D models process each view independently and were trained on different data than the target scene, producing cross-view inconsistency and domain gaps that make the 3D Gaussians ambiguous. IE-SRGS aims to resolve this by jointly using external 2DSR/depth priors and internal, cross-view-consistent 3DGS features to guide optimization toward a high-fidelity high-resolution 3DGS model.

Key Contributions

1. A new 3DGS super-resolution paradigm, IE-SRGS, that addresses the Gaussian ambiguity problem by integrating complementary internal (multi-scale 3DGS-based) and external (2DSR-based) knowledge.
2. The addition of geometric priors from a depth estimation model into the optimization of Gaussians for 3DGS super-resolution, which the authors describe as introducing a geometry optimization branch into 3DGS super-resolution for the first time.
3. A mask-guided fusion strategy that integrates internal and external guidance to supervise 3D Gaussian optimization, using a per-pixel discrepancy mask to decide where each source is trusted.
4. Extensive experimental validation on synthetic and real-world benchmarks (21 scenes across four public datasets) showing consistent improvements over state-of-the-art methods.

Main Findings

• Best results on NeRF Synthetic (4× SR): IE-SRGS reaches 30.97 PSNR, 0.952 SSIM, and 0.054 LPIPS, compared with SRGS at 30.83 / 0.948 / 0.056, CROC at 30.71 / 0.945 / 0.067, FastSR-NeRF at 30.47 / 0.944 / 0.075, SwinIR-3DGS at 30.38 / 0.945 / 0.059, SuperGaussian at 28.44 / 0.923 / 0.067, GaussianSR at 28.37 / 0.924 / 0.087, NeRF-SR at 28.90 / 0.927 / 0.099, DiSR-NeRF at 26.00 / 0.890 / 0.123, Mip-Splatting at 24.59 / 0.909 / 0.101, and 3DGS at 21.77 / 0.867 / 0.104. The paper's "Upper Bound" (HR 3DGS trained directly on HR inputs) is 33.37 / 0.969 / 0.032.

• Large gains over its own backbone: On NeRF Synthetic, IE-SRGS improves over Mip-Splatting by 25.9% in PSNR, 4.73% in SSIM, and 46.5% in LPIPS.

• Best results on real-world datasets (4× SR): On Mip-NeRF360, IE-SRGS scores 27.15 / 0.779 / 0.278 versus 27.02 / 0.774 / 0.279 for Sequence Matters and 26.88 / 0.767 / 0.286 for SRGS. On Deep Blending it scores 29.63 / 0.899 / 0.271 versus 29.49 / 0.896 / 0.275 for SRGS. On Tanks&Temples it scores 23.52 / 0.810 / 0.274 versus 23.43 / 0.808 / 0.274 for Sequence Matters and 23.41 / 0.807 / 0.278 for SRGS. The corresponding Upper Bound values are 27.23 / 0.797 / 0.254, 29.73 / 0.905 / 0.243, and 23.51 / 0.828 / 0.242.

• Ablation shows every component helps (Mip-NeRF360): The Mip-Splatting baseline scores 26.43 / 0.754 / 0.304. Adding MV-Regulation gives 26.68 / 0.757 / 0.297; external texture guidance 26.69 / 0.762 / 0.300; external geometric guidance 26.72 / 0.763 / 0.299; internal texture guidance 27.00 / 0.775 / 0.283; internal geometric guidance 27.05 / 0.775 / 0.282; and the full mask-guided texture integration 27.15 / 0.779 / 0.278.

• Guidance transfers across backbones: On the bicycle and stump scenes, replacing SwinIR with PSRT changes external results from 25.19 / 0.697 / 0.353 to 25.67 / 0.725 / 0.310, and replacing Mip-Splatting with Analytic-Splatting changes internal results from 24.14 / 0.629 / 0.411 to 25.48 / 0.716 / 0.324.

• Robust to the choice of depth estimator: Using DepthAnythingV2 gives 29.58 / 0.889 / 0.205 with a 48.3s runtime; DepthAnythingV2 SMALL gives 29.53 / 0.887 / 0.206 with a 29.3s runtime; DepthPro gives 29.61 / 0.889 / 0.203 with an 80.1s runtime.

• Threshold sensitivity: PSNR gradually improves as the threshold T increases from 0 to 0.9, and drops at T = 1, which the authors attribute to excessive reliance on external priors introducing artifacts.

• Faster inference despite extra modules: On NeRF Synthetic, IE-SRGS reaches 260 FPS versus 191 FPS for SRGS; on MipNeRF-360, 119 FPS versus 92 FPS. Total training time is reported as 19mins30s versus 13mins 56s on NeRF Synthetic, and 54mins17s versus 45mins 51s on MipNeRF-360; the paper states the total training time increases only slightly, by 7-8 minutes.

• Works under stronger downscaling: In an 8× super-resolution experiment on the bicycle and stump scenes, IE-SRGS scores 25.64 / 0.755 / 0.386 versus SRGS at 25.27 / 0.741 / 0.405 and Mip-Splatting at 25.02 / 0.728 / 0.417.

• Cleaner depth: Appendix comparisons show IE-SRGS produces notably cleaner and more coherent rendered depth structures than state-of-the-art methods, and the predicted discrepancy masks align strongly with masks derived from ground-truth high-resolution images and external guidance.

Methodology in Plain English

The pipeline works in stages.

First, the system takes low-resolution multi-view images and produces external knowledge: a pre-trained 2D super-resolution model (SwinIR) generates high-resolution images that carry fine texture detail, and a depth estimation model (Depth Anything V2) estimates depth maps from those images to supply geometry cues. These external references are rich in detail but not necessarily consistent across views, because the 2D model processes each image separately.

Second, the system builds internal knowledge: a multi-scale 3D Gaussian Splatting model is trained directly from the low-resolution inputs, using Mip-Splatting as the backbone because of its anti-aliasing and scale-consistent behavior. Multi-View Regulation supervises several views jointly to improve cross-view coherence. A "SR-Splatting" step then projects and upsamples the 3D Gaussians to produce internally upscaled images and depth maps. These are consistent across views and adapted to the actual scene, but lack high-frequency detail.

Third, the two sources are fused. For texture, the system computes a per-pixel discrepancy between the internal and external images, normalizes it (with a small constant of 10⁻⁶ for numerical stability), and thresholds it into a binary mask. Pixels where the two disagree strongly are supervised by the internal reference; the rest follow the external reference. For geometry, internal and external depth losses are simply combined as a weighted sum, on the reasoning that geometry is coarse and less sensitive to local variation.

The resulting losses guide the optimization of the final high-resolution 3D Gaussians. Implementation-wise, the method is built on the open-source Mip-Splatting codebase with a modified rasterization module for depth rendering; the internal model is trained for 30,000 iterations using 3 randomly sampled views in MV-Regulation, with λᵢ = 0.001 and λₑ = 0.0001, and a discrepancy threshold of 0.9 for real-world scenes and 0.6 for synthetic scenes. All experiments run on a single NVIDIA RTX 4090 GPU.

Why This Matters

Impact on research: The paper argues that prior 3DGS super-resolution work relies mainly on external 2D priors and largely ignores internal 3D consistency and scene-specific adaptation. IE-SRGS reframes the problem as a fusion of two complementary knowledge sources and adds an explicit geometry optimization branch, demonstrating on 21 scenes across Mip-NeRF360, Deep Blending, Tanks&Temples, and NeRF Synthetic that the fusion consistently beats state-of-the-art methods and approaches the high-resolution upper bound. It also reports backbone-agnostic generalization and 8× scaling robustness without scene-specific fine-tuning.

Real-world applications:

• Novel view synthesis and 3D scene reconstruction from low-resolution captures, where high-resolution multi-view data is expensive or infeasible to acquire.
• Storing and transmitting multi-view data: since the method reconstructs high-resolution 3DGS from low-resolution inputs, lower-resolution capture and transfer become viable.
• Immersive media, virtual and augmented reality content, and interactive 3D experiences that need real-time rendering at high fidelity.
• Large-scale, real-world 3D reconstruction pipelines (indoor and outdoor), where the method's performance on the real-world benchmark datasets is relevant.

Industry relevance: Inference speed matters commercially, and IE-SRGS reports 260 FPS on NeRF Synthetic and 119 FPS on MipNeRF-360 versus 191 FPS and 92 FPS for SRGS, with only a modest increase in total training time. The alignment between the paper's rendering-speed and reconstruction-quality claims is relevant for production pipelines that must render interactively, and the ability to swap in alternative super-resolution and 3DGS backbones suggests the approach can be integrated into existing toolchains rather than replacing them. The authors state that code will be released after review.

Future Directions

• Unified frameworks for 3D low-level tasks: The authors explicitly frame this work as laying the foundation for future research on unified frameworks for 3D low-level tasks and more effective internal–external knowledge integration.
• Better integration strategies: Because the current fusion uses a thresholded binary mask for texture and a simple weighted sum for geometry, a natural open question is whether more adaptive fusion mechanisms would improve results, particularly given the PSNR drop observed at threshold T = 1.
• Extending to more extreme scaling: The paper notes that existing state-of-the-art methods rarely report results under extreme scaling and presents only an 8× experiment on two scenes (bicycle and stump); broader evaluation at large magnification factors is left open.
• Reducing the added runtime: The framework introduces extra internal training, depth estimation, and a longer total training time; making these components cheaper or more efficient is an open direction.

Target Audience

This paper is aimed at computer vision and graphics researchers working on 3D reconstruction, novel view synthesis, 3D Gaussian Splatting, and super-resolution, as well as practitioners building 3D capture, rendering, or immersive media pipelines who need high-fidelity 3D models from limited-resolution input. Readers should already be comfortable with 3DGS rendering, splatting-based rasterization, depth estimation, and perceptual and distortion-based image quality metrics (PSNR, SSIM, LPIPS).

Authors’ abstract

Reconstructing high-resolution (HR) 3D Gaussian Splatting (3DGS) models from low-resolution (LR) inputs remains challenging due to the lack of fine-grained textures and geometry. Existing methods typically rely on pre-trained 2D super-resolution (2DSR) models to enhance textures, but suffer from 3D Gaussian ambiguity arising from cross-view inconsistencies and domain gaps inherent in 2DSR models. We propose IE-SRGS, a novel 3DGS SR paradigm that addresses this issue by jointly leveraging the complementary strengths of external 2DSR priors and internal 3DGS features. Specifically, we use 2DSR and depth estimation models to generate HR images and depth maps as external knowledge, and employ multi-scale 3DGS models to produce cross-view consistent, domain-adaptive counterparts as internal knowledge. A mask-guided fusion strategy is introduced to integrate these two sources and synergistically exploit their complementary strengths, effectively guiding the 3D Gaussian optimization toward high-fidelity reconstruction. Extensive experiments on both synthetic and real-world benchmarks show that IE-SRGS consistently outperforms state-of-the-art methods in both quantitative accuracy and visual fidelity.