Synthetic-to-Real Domain Bridging for Single-View 3D Reconstruction of Ships for Maritime Monitoring

AI summary

Overview

Research area: Computer vision for maritime monitoring — specifically single-view 3D reconstruction, synthetic-to-real domain transfer, and 3D Gaussian-based scene representation.

Technical level: Advanced. The work combines the Splatter Image network, 3D Gaussian representations, YOLOv8-based segmentation, homography-based georeferencing, and AIS metadata integration.

Scope: The paper describes a pipeline that reconstructs real ships in 3D from a single image, trained entirely on synthetic data, and places the results on a georeferenced interactive map.

What This Paper Is About

Reconstructing ships in 3D is useful for maritime monitoring, but most state-of-the-art methods need multi-view imagery, annotated 3D ground truth, or heavy computation — none of which fit real-time maritime deployment. This paper builds an efficient pipeline that reconstructs a real ship from just one image at inference time, using only synthetic data for training, thereby avoiding any need for real-world 3D annotations.

Key Contributions

1. An efficient single-view 3D reconstruction pipeline for real ships that is trained entirely on synthetic data and requires only one view at inference.
2. A synthetic-to-real domain bridging strategy: the Splatter Image network (which represents objects as sparse sets of 3D Gaussians) is first fine-tuned on synthetic ShapeNet vessels, then further refined with a diverse custom dataset of 3D ships.
3. Integration of a YOLOv8-based segmentation module and custom preprocessing to make real imagery compatible with the reconstruction network, plus postprocessing for real-world scaling, centering, and orientation alignment.
4. Georeferenced placement of reconstructions on an interactive web map using AIS metadata and homography-based mapping, yielding a system for interactive 3D ship inspection without real-world 3D annotations.

Main Findings

• Strong reconstruction fidelity on synthetic validation data: The abstract reports that quantitative evaluation on synthetic validation data demonstrates strong fidelity, but it does not provide specific metrics.
• Qualitative transfer to real imagery: Results on real maritime images from the ShipSG dataset are described as qualitative and as confirming the potential for transfer to operational maritime settings — not as a validated quantitative real-world result.
• No real-world 3D annotations required: The final system delivers interactive 3D inspection of real ships while training solely on synthetic data, sidestepping the need for annotated real 3D ground truth.
• Efficiency and scalability: The pipeline is presented as an efficient, scalable solution that avoids the multi-view supervision and computational cost of prior reconstruction methods.
• End-to-end georeferenced output: Reconstruction is combined with real-world scaling, centering, orientation alignment, AIS metadata, and homography-based mapping to place ships on an interactive web map.

Methodology in Plain English

The researchers start from an existing reconstruction network, Splatter Image, which describes an object as a sparse collection of 3D Gaussians — a compact representation that allows fast reconstruction from a single image. Because training on real ships with 3D ground truth is impractical, they train in two stages: first fine-tuning the network on synthetic ShapeNet vessels, then refining it on a custom, more diverse dataset of 3D ships to narrow the gap between synthetic training imagery and real maritime photos.

To handle real photographs, the pipeline first locates the ship using a YOLOv8-based segmentation module, with custom preprocessing to make the crop compatible with the reconstruction network. After reconstruction, postprocessing rescales the model to real-world dimensions, centers it, and aligns its orientation. Finally, AIS metadata and homography-based mapping are used to place the reconstructed ship in a georeferenced position on an interactive web map.

Why This Matters

Impact on research: The work targets the synthetic-to-real gap in 3D reconstruction, showing a path to training 3D models without any real-world 3D annotation — a common bottleneck in domains where collecting ground-truth geometry is expensive or impossible. It also connects reconstruction to downstream geospatial metadata and visualization rather than treating reconstruction as an isolated task.

Real-world applications:

• Maritime monitoring and domain awareness, with improved visualization of vessels observed from a single camera.
• Ship inspection, using interactive 3D models instead of flat imagery.
• Decision support in real-world monitoring environments, where operators need fast spatial understanding.
• Georeferenced situational displays that combine 3D ship models with AIS vessel metadata on a web map.

Industry relevance: The pipeline is explicitly framed around practical deployment constraints — real-time operation, no multi-view capture setup, no 3D annotation labor, and integration with existing vessel-tracking metadata. That combination is relevant to maritime authorities, port and harbor operators, and surveillance or monitoring service providers, and it points toward scalable, low-cost 3D visualization of vessel traffic.

Future Directions

• Moving from demonstrated potential to validated real-world performance: the abstract reports only qualitative results on real imagery, so quantitative evaluation on operational maritime data is a clear next step.
• Progressing toward the real-time 3D ship visualization the abstract describes as a path rather than a delivered result.
• Broadening the synthetic-to-real bridging to more varied vessel types, sea states, and camera conditions than the custom dataset currently covers.
• Assessing how much the pipeline depends on the segmentation and preprocessing stages, since errors there would propagate into the reconstructed geometry.

Target Audience

Researchers and engineers working on 3D reconstruction, sim-to-real domain transfer, and maritime or remote-sensing computer vision; practitioners building operational maritime monitoring and visualization systems; and readers interested in training 3D models without real-world 3D ground truth.

Authors’ abstract

Three-dimensional (3D) reconstruction of ships is an important part of maritime monitoring, allowing improved visualization, inspection, and decision-making in real-world monitoring environments. However, most state-ofthe-art 3D reconstruction methods require multi-view supervision, annotated 3D ground truth, or are computationally intensive, making them impractical for real-time maritime deployment. In this work, we present an efficient pipeline for single-view 3D reconstruction of real ships by training entirely on synthetic data and requiring only a single view at inference. Our approach uses the Splatter Image network, which represents objects as sparse sets of 3D Gaussians for rapid and accurate reconstruction from single images. The model is first fine-tuned on synthetic ShapeNet vessels and further refined with a diverse custom dataset of 3D ships, bridging the domain gap between synthetic and real-world imagery. We integrate a state-of-the-art segmentation module based on YOLOv8 and custom preprocessing to ensure compatibility with the reconstruction network. Postprocessing steps include real-world scaling, centering, and orientation alignment, followed by georeferenced placement on an interactive web map using AIS metadata and homography-based mapping. Quantitative evaluation on synthetic validation data demonstrates strong reconstruction fidelity, while qualitative results on real maritime images from the ShipSG dataset confirm the potential for transfer to operational maritime settings. The final system provides interactive 3D inspection of real ships without requiring real-world 3D annotations. This pipeline provides an efficient, scalable solution for maritime monitoring and highlights a path toward real-time 3D ship visualization in practical applications. Interactive demo: https://dlr-mi.github.io/ship3d-demo/.