OceanSplat: Object-aware Gaussian Splatting with Trinocular View Consistency for Underwater Scene Reconstruction

AI summary

Overview

• Research area: Computer vision / 3D scene reconstruction — specifically underwater scene reconstruction and novel view synthesis using 3D Gaussian Splatting (3DGS).
• Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, Neural Radiance Fields, alpha-blending, epipolar geometry, and the revised underwater image formation (SeaThru) model.
• Scope: The paper proposes a 3DGS-based method that uses translated virtual camera viewpoints ("trinocular" consistency), self-supervised epipolar depth priors, a depth residual loss, and depth-aware alpha adjustment to reconstruct geometrically accurate underwater scenes without external geometric supervision.

What This Paper Is About

Underwater images are degraded by light attenuation and scattering, which causes existing 3D reconstruction methods (both NeRF-based and 3DGS-based) to absorb the water medium into the 3D representation, producing floating artifacts and inaccurate geometry. OceanSplat's goal is to recover accurate scene structure and appearance from underwater image collections without relying on external geometric supervision, by enforcing consistency across horizontally and vertically translated virtual viewpoints and by using self-supervised depth constraints.

Key Contributions

1. OceanSplat, a 3DGS-based underwater reconstruction method that enforces trinocular view consistency across horizontally and vertically translated virtual camera poses, aligning 3D Gaussians with scene geometry in scattering media.
2. Synthetic epipolar depth priors derived by triangulating correspondences from the translated virtual viewpoints, used as a self-supervised depth regularizer to compensate for limited geometric cues in degraded underwater scenes.
3. Depth-aware alpha adjustment, which modulates the opacity of 3D Gaussians during early training based on their depth along the viewing direction, discouraging the formation of medium-induced primitives.
4. A depth residual loss that constrains the z-component of individual 3D Gaussians toward the alpha-blended rendered depth, reducing floating artifacts. The paper reports that the combined approach surpasses existing state-of-the-art methods in both qualitative and quantitative evaluations on real-world and simulated datasets.

Main Findings

• State-of-the-art on real-world underwater scenes: In Table 1 (SeaThru-NeRF dataset's Curaçao, Panama, JapaneseGardens-Redsea, IUI3-Redsea; In-the-Wild dataset's Coral and Composite), OceanSplat achieves the top PSNR, SSIM, and LPIPS in all six reported columns. Its best PSNR values are 34.56 (Curaçao), 32.74 (Panama), 25.35 (JapaneseGardens-Redsea), 30.17 (IUI3-Redsea), 29.15 (Coral), and 26.39 (Composite).
• Margins over prior methods: On the SeaThru-NeRF dataset, OceanSplat outperforms WaterSplatting and SeaThru-NeRF-NS by average PSNR margins of 1.05 dB and 2.88 dB, respectively. On the In-the-Wild dataset, it reduces average LPIPS by 0.021 compared with 3DGS and surpasses SeaSplat with average gains of 0.46 dB in PSNR and 0.014 in SSIM, plus a 0.024 reduction in LPIPS.
• Simulated scenes, novel view synthesis: On the simulated Fern-based scenes (Table 2), OceanSplat reaches 28.80 PSNR / 0.871 SSIM / 0.085 LPIPS underwater and 29.12 / 0.873 / 0.089 in fog. This corresponds to an average PSNR improvement of 7.39 dB over SeaSplat and 0.67 dB over WaterSplatting.
• Simulated scenes, scene restoration: OceanSplat achieves 17.16 PSNR / 0.768 SSIM / 0.138 LPIPS underwater and 20.17 / 0.791 / 0.114 in fog, corresponding to average SSIM gains of 0.048 over SeaSplat and 0.021 over WaterSplatting. WaterSplatting reports a higher PSNR underwater (18.39 vs. 17.16) in this table.
• Efficiency: OceanSplat trains in 19 minutes, runs at 85.67 FPS, and uses 7.6 GB of memory (Table 3). It is over 2× faster to train than SeaSplat (51 m) while remaining competitive among 3DGS-based methods; WaterSplatting trains in 10 m at 88.19 FPS with 5.8 GB. SeaThru-NeRF takes 18 h 25 m, runs at 0.10 FPS, and uses 37.7 GB.
• Component ablation (Curaçao scene, Table 4): The full model scores 34.56 PSNR. Removing the trinocular loss L_tri causes the largest PSNR drop (to 33.20); removing the epipolar depth loss L_epi reduces PSNR to 33.82 and raises LPIPS to 0.120; removing the depth residual loss L_res gives 34.30; removing depth-aware opacity α_d gives 33.90.
• Stereo configuration ablation (Table 5): Baseline (P_c) scores 32.20 PSNR at 9 m training; binocular (P_c + P_h) reaches 33.62 at 13 m; adding a second horizontal view (P_c + P_h + P_h′) gives only 33.72 at 16 m; adding a vertical view (P_c + P_h + P_v) reaches 33.91 PSNR / 0.958 SSIM / 0.115 LPIPS at 16 m — a 0.29 dB increase over the binocular setup.
• Qualitative behavior: In the Curaçao scene, prior methods show degraded reconstructions from floating artifacts, while OceanSplat shows fewer floating artifacts; in J.G-Redsea, prior methods produce plausible RGB images but depth maps reveal erroneous Gaussians in water regions; in the Coral scene, SeaThru-NeRF, 3DGS, and SeaSplat fail to capture fine-grained coral structure and WaterSplatting shows geometry loss and depth holes.

Methodology in Plain English

Instead of assuming a clean, in-air imaging process, OceanSplat models water explicitly: an MLP predicts attenuation, backscattering, and medium color along each ray, and the rendered image is split into an object component (attenuated by depth) and a medium/backscatter component. This extends 3D Gaussian Splatting so that "water" and "objects" can be separated rather than baked into the same primitives.

The core trick is to look at each scene from three related viewpoints. For every real camera pose, the method creates two virtual cameras by shifting the pose horizontally and vertically by baseline distances. It renders the object component from those virtual views and uses the rendered depth map to compute disparity, then inverse-warps the virtual images back onto the central view so they can be compared pixel-by-pixel. A regularized L1 loss (divided by the central image, with a stop-gradient and a small constant) enforces that the object component is consistent, and a smoothness loss keeps the disparity maps well behaved. The paper argues the vertical baseline adds orthogonal constraints that a purely horizontal stereo pair lacks.

To get geometry supervision without external depth sensors, the method triangulates the projected positions of 3D Gaussians (those with opacity above a threshold and inside all three view frusta) across the two virtual views, solves a least-squares system, and converts the result into a depth prior in the central camera frame. An edge-aware Log-L1 loss then regularizes the rendered depth against this synthetic epipolar prior.

Two further pieces reduce floating artifacts. A depth residual loss pulls each Gaussian's z-component toward the alpha-blended depth at its pixel. A depth-aware alpha adjustment blends each Gaussian's opacity with a network-predicted opacity during early training, based on its depth and viewing direction; after a transition step the blending weight decays to zero, so there is no inference-time overhead. The full objective combines a regularized L1 + SSIM photometric loss (weight 0.2 on SSIM) with weighted terms for trinocular consistency (0.1), epipolar depth (annealed from 0.4 to 0.2), and the depth residual (0.01).

Implementation notes from the paper: the method builds on Nerfstudio with a TCNN-based MLP; training uses 7K/3K densification/finetuning steps on SeaThru-NeRF and simulated datasets and 10K/5K on In-the-Wild; the vertical baseline is sampled from [−0.4, 0.4] and the horizontal baseline is set to 1.5× the vertical one; the opacity threshold for the epipolar prior is 0.8; resolution increases from ×1/4 to ×1/2 at 1/5 of training and from ×1/2 to full at 2/5; the alpha-adjustment weight is 0.5 before the transition step (4K for 10K training, 10K for 15K training); all experiments use a single RTX A6000.

Why This Matters

Underwater vision is hard because the medium both absorbs and scatters light, so geometric cues that in-air reconstruction pipelines depend on are degraded. This work shows that multi-view geometric consistency, extended from the usual binocular setup to an orthogonal trinocular one, can substitute for explicit depth supervision and help disentangle objects from the medium — a recipe that may transfer to other scattering media such as fog, and the paper does report results on simulated foggy scenes.

Real-world applications the paper points to:

• Seafloor mapping for marine robotics.
• Ecological monitoring, including surveying reef and coral structure.
• Subsea infrastructure inspection, e.g., inspecting underwater equipment and installations.
• Enabling vision-based autonomy on remotely operated vehicles (ROVs) and autonomous underwater vehicles (AUVs), which the paper describes as hindered by degraded perceptual cues.

Industry relevance: the reported training time (19 minutes on a single RTX A6000) and inference speed (85.67 FPS) are in a range that makes repeated reconstruction on survey data more practical than NeRF-based alternatives (18 h 25 m, 0.10 FPS), and the method removes the need for external depth sensors or foundation-model pseudo-depth priors during reconstruction.

Future Directions

• Non-rigid object representations: the authors state future work will address non-rigid objects, which would extend the method from static scenes to moving underwater subjects such as fish.
• Eliminating SfM dependence: the current pipeline initializes from Structure-from-Motion camera poses; the authors list removing that dependence as future work, which would broaden applicability.
• Dynamic underwater scenes: both of the above are framed by the authors as broadening applicability to dynamic underwater scenes.
• Orthogonal-constraint exploration: the stereo-configuration ablation shows a second horizontal view adds little (33.72 PSNR) while a vertical view adds more (33.91 PSNR); whether other baseline geomet

Authors’ abstract

We introduce OceanSplat, a novel 3D Gaussian Splatting-based approach for high-fidelity underwater scene reconstruction. To overcome multi-view inconsistencies caused by scattering media, we design a trinocular setup for each camera pose by rendering from horizontally and vertically translated virtual viewpoints, enforcing view consistency to facilitate spatial optimization of 3D Gaussians. Furthermore, we derive synthetic epipolar depth priors from the virtual viewpoints, which serve as self-supervised depth regularizers to compensate for the limited geometric cues in degraded underwater scenes. We also propose a depth-aware alpha adjustment that modulates the opacity of 3D Gaussians during early training based on their depth along the viewing direction, deterring the formation of medium-induced primitives. Our approach promotes the disentanglement of 3D Gaussians from the scattering medium through effective geometric constraints, enabling accurate representation of scene structure and significantly reducing floating artifacts. Experiments on real-world underwater and simulated scenes demonstrate that OceanSplat substantially outperforms existing methods for both scene reconstruction and restoration in scattering media.