Nereus: A Large-Scale Underwater Dataset for Fine-Grained Attribute Understanding and Grounded Counting Perception

AI summary

Overview

Research area: Computer vision and multimodal large language models (LMMs) applied to underwater imagery — specifically fine-grained biological attribute recognition and grounded (spatially reasoned) counting.

Technical level: Intermediate. The paper assumes familiarity with vision-language fine-tuning (LoRA), benchmark metrics (MAE, F1, accuracy), and dataset construction pipelines, but its core ideas are accessible without deep marine-biology background.

Scope: The paper introduces Nereus, an 88K-image underwater dataset with 2.68M question-answer pairs that jointly supervises object-level biological attributes and region-, camouflage-, and density-aware counting, plus a benchmark of open-source and proprietary LMMs trained and evaluated on it.

What This Paper Is About

Underwater images carry rich ecological information, but existing underwater datasets label scenes coarsely — broad categories, generic captions, or just a total count of animals. That is not enough for applications that need organism-level traits (species, body shape, condition, pose) or counts that are tied to where animals are and how hard they are to see. Nereus addresses this gap by building one dataset that supports both Fine-Grained Object Attribute Understanding and Grounded Counting Perception, with every image also labeled for underwater degradation (luminance, color cast, turbidity).

Key Contributions

1. A new problem framing — "Underwater Fine-grained Perception." The authors define two core capabilities, Fine-Grained Object Attribute Understanding and Grounded Counting Perception, motivated by four downstream applications: science communication, fishery monitoring, ecological conservation, and oceanographic research.

2. The Nereus dataset itself. 88K images, 2.68M question-answer pairs, 136K annotated marine organism objects, a 136-dimensional attribute set spanning five categories, 2,690 species, 1,515 genera, 65K point annotations, and 67K grounded counting QA pairs, drawn from FathomNet, CoralVQA, iNaturalist, and IOCFish5K.

3. A hybrid annotation pipeline with human verification. Automatic annotation uses a "dual-model discussion with single-model arbitration" scheme (Qwen3-VL-8B and InternVL3-8B discuss, Qwen2.5-VL-72B arbitrates attributes with confidence below 0.6), backed by a biological knowledge graph and expert review of a sampled subset.

4. A comprehensive benchmark. Evaluation of LLaVA-OneVision-7B, InternVL2.5-8B, and Qwen2.5-VL-7B alongside zero-shot baselines including BioCLIP2, GPT-5, Gemini 2.0 Flash, and Gemini 3.1 Pro, across attribute understanding, counting, density evidence, and transfer to an external counting dataset.

Main Findings

• Nereus supervision beats prior underwater data for attribute understanding. Compared with NautData, overall attribute accuracy rises from 11.27% to 51.44% on LLaVA-OneVision-7B, from 15.13% to 51.88% on InternVL2.5-8B, and from 17.53% to 46.17% on Qwen2.5-VL-7B.

• Semantic similarity improves alongside accuracy. Overall similarity scores for Nereus fine-tuning reach 74.80 (LLaVA-OneVision), 76.77 (InternVL2.5), and 73.27 (Qwen2.5-VL), versus 50.95, 44.36, and 50.28 respectively for NautData adaptation.

• Strong proprietary zero-shot models still lead on strict accuracy. Gemini 3.1 Pro reaches 59.59 overall accuracy and 65.42 similarity without any Nereus training, above the fine-tuned open-source backbones on accuracy (46.17–51.88) though below them on similarity (73.27–76.77).

• Structural morphology is the hardest attribute category. The authors attribute this to subtle visual cues such as fin shape, mouth type, and coral growth form, which are easily degraded underwater; Nereus fine-tuning reaches 38.30, 34.93, and 30.75 on this category across the three backbones.

• Grounded counting supervision sharply reduces counting error. Nereus QAS cuts InternVL2.5-8B's counting MAE from 87.38 to 25.99, outperforming the specialized counter CountGD at 32.33.

• Specialized counters retain an edge in localization. CountGD achieves 67.67 densest-cell accuracy, which the authors cite as evidence of complementary strengths between specialized counting models and Nereus-trained VLMs.

• Spatial and visibility supervision, not just counts, drives gains. Nereus QAS raises Qwen2.5-VL-7B's densest-cell accuracy to 49.16 and Camouflage F1 to 52.85, far above the 6.11 and 7.41 obtained with count-only supervision.

• Density evidence helps reasoning. Adding density QA supervision lifts Qwen2.5-VL-7B's Density F1 from 18.13 to 45.26 and Grounding F1 from 16.04 to 37.19.

• Nereus transfers better than an external counting dataset. In the Nautilus transfer comparison, Nereus QAS achieves MAE 31.98 versus 39.96 for Nautilus counting SFT, and densest-cell accuracy of 49.16% versus 6.11%.

• Structured attribute supervision beats isolated QA pairs and captions. With Qwen2.5-VL-7B, structured attribute sets reach 41.55 overall accuracy and 73.10 similarity, versus 39.08/55.20 for QA pairs and 16.78/61.60 for object-level captions.

• Annotation quality is high but imperfect. The training set averages 9.26 on a 10-point scale (200 sampled images); the test set averages 9.14 with 94.8% scoring 8 or above. Of 934 expert-reviewed attributes, 43 were corrected, a 4.60% correction rate.

Methodology in Plain English

Building the attribute annotations. Each annotated organism is cropped into an object-centered patch, expanded by a 15% margin to retain nearby context, and paired with the original image and its bounding-box or point coordinates so the model can fall back on scene context when the local patch is degraded. For iNaturalist images lacking boxes, GroundingDINO generates them and 20% are manually checked. A knowledge graph of trait descriptions covering 2,803 species or genera — built from FishBase, WoRMS, Wikipedia, and the Coral Trait Database — is injected into the annotation prompt, with explicit instructions to trust visible image evidence over the retrieved prior.

Two models discuss, a third arbitrates. Qwen3-VL-8B and InternVL3-8B annotate independently, then go through a three-turn discussion where each reviews and updates the other, producing a consensus value plus a confidence score per attribute. Low-confidence attributes (below 0.6) are escalated to Qwen2.5-VL-72B for an independent final call. The full run took 7 days on four RTX 4090 GPUs.

Turning points into reasoning data. For counting, fish point annotations from IOCFish5K are converted into three task types. Qwen3-VL-8B-Instruct labels coarse semantic regions (open water, coral shelter, reef or rock, seafloor, vegetation, occluded dark areas), SAM refines them into masks, and points are assigned to regions and to 3×3 grid cells. Regions are also labeled clear, ambiguous, or highly camouflaged. Points are converted into adaptive Gaussian density maps to locate hotspots. Qwen3.5-9B writes natural-language QA pairs, while rule-based templates preserve exact numbers.

Benchmarking. The test set is built to include species absent from training, so it probes cross-species generalization. Models are fine-tuned with LoRA on 4 RTX 4090 GPUs (12 days total), keeping visual backbones frozen. Attribute answers are scored by normalized exact match plus an LLM-as-a-judge semantic similarity; counting is scored by MAE, exact count accuracy, tolerance-based accuracy at ±5% and ±10%, densest-cell accuracy, Camouflage F1, Density F1, and Grounding F1.

Why This Matters

Impact on research. Nereus reframes underwater vision from scene captioning to organism-level, evidence-grounded perception. It supplies structured supervision (a 136-dimensional attribute set) and converts point counts into region-, camouflage-, and density-aware instruction data — a supervision format that existing underwater datasets do not jointly provide. The degradation labels (three types, three severity levels, based on the underwater image quality index) also give the field a way to study robustness to color cast, turbidity, and low contrast rather than ignoring them.

Real-world applications:

• Fishery monitoring: counts tied to regions and visibility conditions are more useful for stock assessment than raw totals.
• Ecological conservation: species-level traits and camouflage states support reef and habitat health assessment.
• Science communication and education: automatically generated, evidence-linked descriptions of marine organisms.
• Oceanographic research: large-scale, machine-readable records of what was seen, where, and under what water conditions.

Industry relevance. Any operator processing underwater video at scale — aquaculture, environmental consulting, offshore inspection, marine tourism — faces the same bottleneck: manual annotation by scarce taxonomic experts. A dataset and benchmark for fine-tuning existing open-weight LMMs lowers that barrier, and the paper's finding that grounded supervision beats count-only supervision gives a concrete recipe for building useful counting systems rather than demos.

Future Directions

• Broader taxonomic and geographic coverage. The authors acknowledge Nereus may underrepresent taxa, regions, depths, and imaging devices, and that robustness to rare species and atypical individuals (including injured organisms) is insufficiently evaluated.

• Finer-grained evaluation of annotation error. Expert review covered only a sampled subset, so remaining annotation errors are unquantified; the authors plan to expand expert verification.

• Moving beyond random splits. Random train/test sampling does not explicitly separate locations, dates, or dives, so performance under genuine domain shift remains to be measured.

• Video and real-world deployment. The paper positions training on static images as a starting point, with evaluation on unfiltered continuous video and broader field deployment left as future work.

Target Audience

Researchers and practitioners in multimodal machine learning and computer vision who work on domain-specific vision-language data, particularly those building datasets or fine-tuning LMMs for scientific, ecological, or environmental domains. It is also directly useful to marine biologists and conservation or fishery analysts who want to understand what automated underwater perception can and cannot currently do, and to engineers building underwater inspection or monitoring systems who need a realistic sense of current model accuracy — including that even after domain-specific training, open models trail a strong proprietary zero-shot model on strict attribute accuracy.

Authors’ abstract

Underwater imagery records rich ecological information and supports applications such as science communication, fishery monitoring, ecological conservation, and oceanographic research. Beyond general scene understanding, these applications require models to recognize fine-grained biological traits and reason over spatially grounded evidence. However, existing underwater datasets often provide coarse category labels, generic captions, or total-count supervision, which limits their use for organism-level analysis and ecological reasoning. We introduce Nereus, a large-scale dataset for Underwater Fine-grained Perception that jointly supports Fine-Grained Object Attribute Understanding and Grounded Counting Perception. Nereus contains 88K images and 2.68M question-answer pairs, including annotations for 136K marine organism objects across 2,690 species and 1,515 genera, as well as 65K point annotations and 67K grounded counting QA pairs. Experiments with representative multimodal models show that Nereus improves downstream adaptation over existing underwater-domain data and provides a foundation for future research on fine-grained marine perception.