EGG-Fusion: Efficient 3D Reconstruction with Geometry-aware Gaussian Surfel on the Fly

AI summary

Overview

Research area: Computer vision / computer graphics — real-time simultaneous localization and mapping (SLAM) and 3D scene reconstruction using differentiable rendering.

Technical level: Advanced. The paper assumes familiarity with SLAM, differentiable rendering, 3D Gaussian Splatting, Gaussian surfels, and Bayesian information filters.

Scope: A real-time RGB-D reconstruction system (EGG-Fusion) that combines Gaussian surfel scene representation with an information-filter-based surface fusion strategy and a sparse-to-dense camera tracking pipeline, evaluated on Replica, TUM-RGBD, ScanNet++, and three captured real-world sequences.

What This Paper Is About

Systems that reconstruct 3D scenes while tracking a camera, using differentiable rendering (NeRF or 3D Gaussian Splatting), struggle with two problems: they are computationally heavy, and they are sensitive to sensor depth noise. That noise and the loose parameterization of 3D Gaussians produce geometric ambiguity, which degrades reconstruction accuracy and, in complex scenes with limited viewpoint coverage, can cause outright failure. EGG-Fusion attacks both issues by representing the scene as 2D Gaussian surfels that are explicitly fused frame-by-frame using a probabilistic information filter, so that geometry is refined continuously rather than left entirely to slow differentiable optimization.

Key Contributions

1. EGG-Fusion system. A Gaussian-surfel-based real-time 3D reconstruction system for high-quality scene modeling in real-world environments; the authors state that the source code is released for reproducibility.
2. Information-filter-based surfel fusion. A method that incrementally updates each surfel's geometric state and covariance matrix from new depth observations, explicitly modeling sensor noise and thereby producing stable high-precision surfaces while accelerating optimization convergence.
3. Sparse-to-dense camera tracking. A tracking strategy that initializes pose from sparse 2D–3D feature correspondences and then refines it with dense geometric and photometric alignment, for robustness against rapid motion and poor initialization.
4. Geometry-aware surfel initialization and differentiable surfel optimization. An adaptive scheme that activates new surfels only in geometrically salient regions (low-opacity zones and positive depth-disparity areas) with scale proportional to depth, plus per-surfel geometric regularization that constrains optimized surfels to stay near the fused surface.

Main Findings

• Tracking accuracy on Replica: EGG-Fusion achieves the best average ATE RMSE of 0.17 cm, compared with RTG-SLAM 0.18 cm, SplaTAM 0.39 cm, Point-SLAM 0.49 cm, Vox-Fusion 0.52 cm, NICE-SLAM 1.06 cm, MASt3R 1.67 cm, and SLAM3R 6.61 cm. It is best in most individual scenes and, per the paper, attains the best trajectory precision in most scenarios.
• Tracking accuracy on TUM-RGBD: EGG-Fusion reaches the best average among real-time systems at 4.47 cm, ahead of RTG-SLAM 5.12 cm, SplaTAM 5.48 cm, ORB-SLAM2 4.68 cm, ElasticFusion 6.91 cm, Point-SLAM 8.92 cm, Vox-Fusion 9.28 cm, and NICE-SLAM 15.87 cm. BAD-SLAM fails on several sequences (marked ✗).
• Offline variants: With a global optimization backend similar to RTG-SLAM's official release, EGG-Fusion reaches 1.98 cm average on TUM-RGBD, versus RTG-SLAM offline 2.02 cm and ORB-SLAM2 offline 2.09 cm.
• Reconstruction accuracy (points, Replica / ScanNet++): EGG-Fusion reports accuracy of 0.60 / 0.67 and accuracy ratio [<3 cm] of 99.99 / 99.98, versus RTG-SLAM 0.80 / 1.06 and 98.52 / 95.34, and SplaTAM 2.87 / 1.71 and 74.27 / 89.15. Point-SLAM is the one exception with higher accuracy in this evaluation, which the authors attribute to its use of ground-truth depth to determine rendering sampling positions.
• Reconstruction accuracy (TSDF, Replica / ScanNet++): EGG-Fusion reports accuracy 0.90 / 1.51 with accuracy ratio 97.60 / 94.20, compared with Point-SLAM 0.76 / 0.67, SplaTAM 1.12 / 1.70, RTG-SLAM 1.30 / 1.86, and NICE-SLAM 2.84 / 4.45.
• Headline surface error: The abstract reports a surface reconstruction error of 0.6 cm on Replica and ScanNet++, described as over 20% improvement in accuracy over state-of-the-art GS-based methods.
• Real-time performance: The system runs at 24 FPS, which the authors describe as making it one of the most accurate differentiable-rendering-based real-time reconstruction systems.
• Rendering quality on ScanNet++: EGG-Fusion gets the best PSNR and SSIM on both novel and training views. Novel view: PSNR 25.70, SSIM 0.907, LPIPS 0.212 (SplaTAM 24.75 / 0.900 / 0.208; RTG-SLAM 24.77 / 0.882 / 0.255; Point-SLAM 17.68 / 0.623 / 0.548). Training view: PSNR 29.06, SSIM 0.944, LPIPS 0.141 (SplaTAM 27.30 / 0.940 / 0.130; RTG-SLAM 27.54 / 0.925 / 0.184; Point-SLAM 24.35 / 0.800 / 0.373). Note that SplaTAM's training-view LPIPS is lower than EGG-Fusion's.
• Confidence-aware surface extraction: Maintaining an information matrix per primitive allows the system to estimate confidence and extract high-confidence scene surfaces, shown in the qualitative comparison of Figure 1.
• Convergence behavior: Because surfels are updated geometrically every frame, the authors state that only minor refinement is needed during map optimization, giving rapid convergence and reduced computational overhead.

Methodology in Plain English

Scene representation. The scene is a set of 2D Gaussian surfels — flat, disk-like primitives with a center position, two axis scales, a rotation quaternion, opacity, and color encoded as spherical harmonic coefficients. Constraining the third axis to be flat makes them easier to attach to actual scene surfaces than fully volumetric 3D Gaussians. They are rendered by projecting to image space, sorting by depth, and alpha compositing to produce color, depth, and normal maps.

Adding new surfels. Instead of uniformly sprinkling new primitives across the image, the system only adds surfels where they are actually needed: in low-opacity regions where rendering is failing, and in regions with positive depth disparity indicating newly seen foreground. Their positions and normals are taken from the current depth and normal maps, and their scale grows with distance so that their image-space footprint stays consistent.

Fusing new observations. This is the core idea. Each surfel's geometric state (position plus normal, six numbers) is tracked as a small probabilistic estimate with a 6×6 covariance matrix. Every time a surfel is observed again in a new frame, the measurements are folded in using an information filter — the information form of a Kalman-style recursive Bayesian update. The filter accumulates confidence over repeated observations, which suppresses the impact of noisy consumer-grade depth sensors. Sensor noise is modeled as growing with the square of depth, matching how these sensors behave. Because the covariance is simplified to a diagonal matrix, the update is cheap enough to run online. Position and normal updates are converted into a surfel rotation using the axis normal to the plane spanned by the old and new normals, resolving an otherwise underconstrained rotation estimate.

Optimization. After fusion, surfels are refined with differentiable rasterization over a local map of the most recent N_batch frames, running m·N_batch iterations so each frame is seen about m times on average. The loss combines color, depth, and normal rendering terms against ground truth, plus a per-surfel geometric regularization that penalizes deviation of a surfel's optimized position and normal from its fused values. The authors note that this regularization is not applicable to 3DGS-based methods, because volumetric 3D Gaussians lack an explicit geometric anchor and their inferred multi-view depths are inconsistent.

Tracking. Camera pose is first estimated by minimizing reprojection error over sparse 2D–3D correspondences using Levenberg–Marquardt with a robust loss for outliers. That initial pose is then refined by joint dense alignment: an ICP-style term matching the current frame's vertex map against the global model's vertices and normals, plus a photometric term comparing rendered color to the observed image, balanced by a weight. The system also checks whether dense alignment converged and can reject its result in degenerate cases.

Why This Matters

The paper's argument is that treating sensor noise as a first-class modeling problem, rather than something that differentiable rendering will average away, is what enables both higher accuracy and real-time speed. Explicit frame-by-frame fusion keeps geometry near its final state, so the slow differentiable optimizer has little work left to do. This is a different design principle from prior 3DGS-based SLAM, which relies mainly on gradient descent over primitive parameters.

Real-world applications:

• Mixed and augmented reality, where a headset must build a usable room mesh while the user moves.
• Robotics navigation, where pose accuracy and a reliable surface map directly affect planning and obstacle avoidance.
• Autonomous driving, where consumer-grade depth sensors produce noisy, depth-dependent measurements of the kind this paper explicitly models.
• Digital content creation from ordinary RGB-D scans, using the confidence-weighted surfaces and photorealistic rendering the system produces.

Industry relevance: The system targets consumer-grade sensors such as Azure Kinect, runs at 24 FPS, and the authors release source code, all of which lower the barrier to adopting this class of reconstruction method in products. The information-matrix confidence estimate is also a practical signal for downstream systems that need to know which parts of a reconstructed surface to trust.

Future Directions

• Generalizing the noise model. The filter currently assumes noise scales with the square of depth and uses a diagonal covariance. Whether richer, sensor-specific or learned noise models improve accuracy on different depth sensors is not resolved here.
• Monocular or sensor-fusion operation. All reported experiments use RGB-D input. Extending the information-filter fusion to monocular or inertial-aided settings, where depth is estimated rather than measured, is an open question the paper does not address.
• Scaling to larger scenes. Experiments cover room-scale Replica and ScanNet++ sequences plus three captured scenes (Bag, Hydrant, Stone). Whether the surfel map and fusion cost remain practical for building-scale or city-scale environments is not reported.
• Handling the reported failure cases. The captured sequences include missing depth from out-of-range measurements and reflective or transmissive surfaces. How the filter and initialization behave when larger regions of depth are invalid is not quantified in the content available.

Target Audience

Researchers and graduate students working on SLAM, 3D reconstruction, and differentiable rendering will get the most from this paper, particularly those already familiar with NeRF-based or 3DGS-based SLAM systems such as Point-SLAM, SplaTAM, and RTG-SLAM. Engineers building AR/VR or robotics perception pipelines on consumer RGB-D hardware will benefit from the practical emphasis on real-time rates and noisy-sensor robustness. Readers without a background in Bayesian filtering, Lie algebra, and Gaussian splatting rasterization will find the methodology sections demanding.

Authors’ abstract

Real-time 3D reconstruction is a fundamental task in computer graphics. Recently, differentiable-rendering-based SLAM system has demonstrated significant potential, enabling photorealistic scene rendering through learnable scene representations such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). Current differentiable rendering methods face dual challenges in real-time computation and sensor noise sensitivity, leading to degraded geometric fidelity in scene reconstruction and limited practicality. To address these challenges, we propose a novel real-time system EGG-Fusion, featuring robust sparse-to-dense camera tracking and a geometry-aware Gaussian surfel mapping module, introducing an information filter-based fusion method that explicitly accounts for sensor noise to achieve high-precision surface reconstruction. The proposed differentiable Gaussian surfel mapping effectively models multi-view consistent surfaces while enabling efficient parameter optimization. Extensive experimental results demonstrate that the proposed system achieves a surface reconstruction error of 0.6\textit{cm} on standardized benchmark datasets including Replica and ScanNet++, representing over 20\% improvement in accuracy compared to state-of-the-art (SOTA) GS-based methods. Notably, the system maintains real-time processing capabilities at 24 FPS, establishing it as one of the most accurate differentiable-rendering-based real-time reconstruction systems. Project Page: https://zju3dv.github.io/eggfusion/