NAUTILUS: A Large Multimodal Model for Underwater Scene Understanding

AI summary

Overview

Research area: Computer vision and multimodal machine learning, specifically large multimodal models (LMMs) applied to underwater imagery.

Technical level: Intermediate. The paper assumes familiarity with vision-language models (LLaVA, Qwen2.5-VL), instruction tuning, and basic image formation models, but the core ideas are accessible to anyone who has encountered multimodal AI.

One-sentence scope: The paper introduces NautData (a large underwater instruction-tuning dataset), Nautilus (an underwater LMM built on existing baselines), and a physics-inspired Vision Feature Enhancement module that explicitly corrects for water-induced image degradation in feature space.

What This Paper Is About

Underwater imagery is central to marine science, resource exploration, and national security, but it is notoriously hard for AI to interpret: light scattering and absorption distort color and contrast, and existing vision-language datasets are built for in-air scenes. Prior underwater models tend to be task-specific or limited to image-level description, leaving no unified multimodal system that can classify, detect, count, ground, and caption underwater scenes. This paper builds the missing dataset, the missing model, and a physically motivated correction module to close that gap.

Key Contributions

1. NautData dataset — a 1.45M image-text-pair instruction-following dataset spanning 158K images and eight underwater tasks across three granularities (image, region, object), including coarse/fine classification, counting, VQA, detection, grounding, region caption, and image caption. It is the first underwater dataset to cover all eight tasks simultaneously and is fully open-sourced.

2. Nautilus model — the first underwater LMM to perform eight-task, three-granularity scene understanding. Built by integrating the VFE module into LLaVA-1.5 and Qwen2.5-VL, it outperforms both baselines and commercial systems (GPT-4o, Gemini 2.0 Flash, Qwen2.5-VL-72B) on most underwater tasks.

3. Vision Feature Enhancement (VFE) module — a plug-and-play component derived from the physical underwater imaging model. It uses a depth encoder and dark-pixel prior to explicitly remove backscattering and restore light absorption in feature space (not pixel space), avoiding information loss caused by image-level enhancement.

4. Robustness under degraded conditions — systematic evaluation across low-light, green-tinted, blue-tinted, turbid, and clear subsets shows Nautilus improves grounding accuracy by up to 8.1 PR@0.5 over baselines in challenging conditions.

Main Findings

• Domain shift hurts general LMMs: Zero-shot GPT-4o, Gemini 2.0 Flash, and Qwen2.5-VL-72B perform poorly on NautData (e.g., detection mAP@0.5 at 1.4, 7.8, and 14.7 respectively), while fine-tuned open-source models jump to 19–42, confirming that underwater tasks require domain-specific adaptation.

• VFE consistently helps: Adding the VFE module improves LLaVA-1.5 on seven of eight tasks and Qwen2.5-VL on most tasks, with the largest gains in fine-grained classification (+5.6 acc) and grounding (+3.6 mAP@0.5) for the Qwen base.

• Feature-space beating pixel-space enhancement: Applying three SOTA underwater image restoration methods (Reti-Diff, SMDR-IS, CCL-Net) as pre-processing reduces performance on classification, captioning, and detection, while VFE's in-feature correction improves them — the authors attribute this to information loss during pixel-level restoration.

• Depth alone is not enough: Simply adding a depth encoder degrades four tasks due to feature-distribution mismatch; only when depth is used within the VFE formulation (to model light absorption) do gains appear, validating the physics-guided design.

• Generalization to unseen data: Zero-shot grounding on MarineInst20M improves LLaVA-1.5 and Qwen2.5-VL by 0.6 and 0.4 PR@0.5, showing the module transfers across datasets and base models.

• Counting trade-off: Nautilus achieves the best MAE and RMSE on IOCfish5k, but a slight accuracy drop in the multiple-choice counting variant, which the authors attribute to friction between regression and classification objectives in multi-task training.

Methodology in Plain English

The team started with the standard underwater imaging equation, which says a captured image equals direct reflection from the subject plus backscattering from the surrounding water, with the reflection attenuated exponentially by distance-dependent light absorption. Rather than trying to undo these effects on the raw pixels — which loses information — they apply the same math to the features that the image encoder produces.

Two operations are performed. First, they identify the darkest patch in the image (the dark-pixel prior assumes that in a clean scene some pixels should be nearly black, so whatever brightness remains there is contamination from backscattering), isolate its semantic content with cross-attention, and subtract the residual backscatter signal from every feature token. Second, they feed a frozen depth encoder's output through a small MLP to predict an absorption weight per token, then divide the features by the exponential of that weight, mirroring the physical equation.

The enhanced features are concatenated with the original features (both matter — one encodes authentic underwater appearance, the other restored clarity) and passed through the same projector into the language model. They integrated this into LLaVA-1.5 and Qwen2.5-VL using LoRA plus a trainable projector and VFE module, training for one epoch on four A800 GPUs over roughly three days.

The NautData dataset was assembled by combining existing annotated underwater detection, classification, and counting datasets with LMM-generated captions and VQA pairs. Ground-truth labels were converted into templated Q&A pairs for structured tasks; Gemini 2.0 Flash generated free-form content which Qwen2.5-VL-72B then filtered, with a final GPT-4o pass and manual verification on the 7,916-example test set.

Why This Matters

Research impact: This is the first work to unify eight underwater scene understanding tasks under a single multimodal model and a single dataset, replacing a fragmented landscape of task-specific pipelines. The VFE module offers a template for injecting physical priors into LMMs for other degraded-imaging domains (fog, medical ultrasound, satellite haze).

Real-world applications:

• Autonomous underwater vehicles performing survey, inspection, or search-and-rescue in poor visibility without human pilots.
• Marine biology and conservation — automated species census, coral reef health monitoring, and behavioral analysis from ROV footage.
• Fisheries and aquaculture — counting and classifying stock, detecting disease or anomalies in pens.
• Defense and infrastructure — inspecting submerged pipelines, cables, and hulls where turbidity and color cast defeat conventional vision systems.

Industry relevance: Companies deploying underwater drones, offshore energy operators, and marine data platforms all currently rely on manual annotation or narrow single-task models. A general underwater LMM that follows natural-language instructions lowers the integration barrier substantially, and the open dataset gives startups a training foundation that previously only large labs could assemble.

Future Directions

• Open-vocabulary and few-shot generalization: The authors explicitly flag this as a limitation — the sheer diversity of underwater species and environments means any fixed category set will miss novel cases, motivating few-shot or open-vocabulary extensions.

• Extending to video and temporal reasoning: All eight tasks are single-image; underwater robots need to track, track behavior over time, and reason about motion.

• Scaling the physics prior: The current VFE handles backscattering and absorption but not wavelength-dependent effects, artificial lighting, or turbidity gradients; richer imaging models could be plugged into the same framework.

• Balancing multi-task objectives: The counting-accuracy regression on multiple-choice counting suggests more principled multi-task loss design is needed as task counts grow.

Target Audience

Researchers and graduate students in multimodal learning, computer vision, and marine robotics; engineers building underwater perception systems for drones, ROVs, or offshore platforms; and marine scientists who need automated analysis of large underwater image archives. Readers with a basic grasp of vision-language models and instruct-tuning will get the most from the methodology section, while the dataset and benchmark tables are useful to practitioners regardless of background.

Authors’ abstract

Underwater exploration offers critical insights into our planet and attracts increasing attention for its broader applications in resource exploration, national security, etc. We study the underwater scene understanding methods, which aim to achieve automated underwater exploration. The underwater scene understanding task demands multi-task perceptions from multiple granularities. However, the absence of large-scale underwater multi-task instruction-tuning datasets hinders the progress of this research. To bridge this gap, we construct NautData, a dataset containing 1.45 M image-text pairs supporting eight underwater scene understanding tasks. It enables the development and thorough evaluation of the underwater scene understanding models. Underwater image degradation is a widely recognized challenge that interferes with underwater tasks. To improve the robustness of underwater scene understanding, we introduce physical priors derived from underwater imaging models and propose a plug-and-play vision feature enhancement (VFE) module, which explicitly restores clear underwater information. We integrate this module into renowned baselines LLaVA-1.5 and Qwen2.5-VL and build our underwater LMM, NAUTILUS. Experiments conducted on the NautData and public underwater datasets demonstrate the effectiveness of the VFE module, consistently improving the performance of both baselines on the majority of supported tasks, thus ensuring the superiority of NAUTILUS in the underwater scene understanding area. Data and models are available at https://github.com/H-EmbodVis/NAUTILUS.