WildFin: An In-the-Wild Dataset for Fish Behavioral Recognition

AI summary

Overview

• Research area: Computer vision benchmark for animal behavior recognition, at the intersection of video understanding, vision foundation models, and marine field ecology.
• Technical level: Intermediate. The dataset itself is accessible, but the benchmarking protocol assumes familiarity with vision transformers, self-supervised pretraining, frozen-backbone adaptation, pooling heads, and class-imbalance techniques.
• Scope: The paper introduces and benchmarks WildFin, an expert-annotated in-the-wild underwater video dataset for fish behavior recognition, drawn from two real field-collection paradigms.

What This Paper Is About

Field-deployable underwater camera systems now produce far more video than ecologists can manually analyze, and existing computer vision models frequently fail when applied to complex marine footage. The authors address the shortage of public, expert-annotated underwater behavior video by building WildFin from real coral reef field data — collected in Curaçao — and using it to measure how far current vision foundation models fall short of the demands of ecological behavioral analysis. The goal is to characterize these failures and establish baselines rather than to propose a new model.

Key Contributions

1. A new in-the-wild video benchmark. WildFin comprises 9.2 hours (2,058,892 frames) of expert-annotated video spanning 23 behaviors, derived from ongoing ecological research rather than curated for computer vision. Producing it involved 1,350 hours of fieldwork and 600 hours of expert annotation.
2. Two distinct real-world capture paradigms. The dataset pools CoralCam (stationary, multi-agent tripod recordings of schools) and FishFollow (dynamic diver focal-follow recordings of single fish), reflecting the heterogeneity typical of ecologist-curated data collected under year-to-year protocol variation.
3. Released annotations and pipeline artifacts. This includes object detection annotations in COCO format, a trained YOLOv8 detector, expert-verified tracks (213 in CoralCam), and frame-level behavioral annotations, all available on the project website.
4. Baselines quantifying architectural trade-offs. The authors benchmark DINOv3-B, DINOv3-L, ResNet50, VideoMAE-B, VideoMAE-L, and V-JEPA-2-L across static versus spatiotemporal processing, frozen versus full fine-tuning, two pooling strategies, and two class-imbalance mitigation techniques.

Main Findings

• Spatiotemporal models win overall, and win decisively on dynamic behaviors. VideoMAE-L achieves the highest macro-F1 on both subsets (0.450 on CoralCam overall, 0.387 on FishFollow overall). The gap is starkest for Aggression on CoralCam: image-based models score 0.000 (DINOv3-B, ResNet50) and 0.007 (DINOv3-L), while VideoMAE-L reaches 0.332 and V-JEPA-2-L reaches 0.294.
• Static image models remain competitive on appearance-dominated categories. On CoralCam Biting, DINOv3-L reaches 0.490 versus VideoMAE-L's 0.510. On FishFollow Habitat, image backbones post the highest scores, with DINOv3-L at 0.594.
• Frozen foundation features beat fully fine-tuned CNNs on the harder subset. On CoralCam, fully fine-tuned ResNet50 scores 0.358 overall versus 0.329 for frozen DINOv3-L and 0.286 for frozen ResNet50. On FishFollow the ordering reverses: fully-tuned ResNet50 drops to 0.321 and appears to overfit, while frozen DINOv3-L reaches 0.380.
• Focal loss trades precision for recall. Across most configurations, focal loss improves performance by boosting recall at a modest cost to precision. The exception is VideoMAE on CoralCam, where the authors suspect focal loss caused overfitting to very rare aggression behaviors, resulting in zero test performance. Naive random sampling yields near-zero performance on rare behaviors, motivating balanced sampling.
• Attention pooling generally beats mean pooling, with one notable exception. Attention pooling outperforms mean pooling across nearly all backbones. V-JEPA-2-L is the exception, where mean pooling is substantially better; the authors hypothesize its predictive-embedding pretraining objective distributes information across tokens rather than concentrating it in salient patches.
• Video-native backbones are more parameter-efficient. For equivalent parameter counts, video backbones generally achieve higher F1 than image-based models, and VideoMAE-L reaches peak macro-F1 on both datasets despite a smaller footprint than the DINOv3 variants.
• Detection is strong but not solved. On the WildFin detection subset, RT-DETR-L achieves the best results (AP@0.5 of 0.807, AP@0.75 of 0.525, mAP@[0.5-0.95] of 0.477), with YOLOv11-X performing comparably (0.764, 0.516, 0.455).
• The FishFollow focal-animal localization gap is the principal performance limiter. The paper states that because no bounding-box or segmentation annotation ties a behavior label to the focal individual, models may learn to associate behavior with its presence anywhere in the frame; the absence of localization annotations is described as the principal factor limiting model performance on that subset.
• Annotation boundaries, not behavior identity, drive disagreement. FishFollow inter-annotator agreement averages F1 of 0.78 at a 0.25 s tolerance and rises to 0.93 at 0.5 s. CoralCam inter-annotator F1 on a held-out subset was 0.74.

Methodology in Plain English

The authors first assembled the dataset from footage that field biologists had already captured, then annotated it using protocols matched to each capture style. For CoralCam, direct full-frame annotation was impossible because frames contain tens to hundreds of fish, so they used a three-stage pipeline: ecologists hand-labeled 21,116 bounding boxes for three species of interest, those trained a YOLOv8 detector reaching AP@0.5 of 74.5, the detector was deployed and detections were linked into tracks with BoTSort, and an expert then selected and verified 213 tracks and applied frame-by-frame behavioral labels. Mouth visibility was separately encoded to flag frames where occlusion or a fish facing away made classification impossible. For FishFollow, each video follows one focal fish, so annotators labeled frames directly without a tracking stage, covering 20 behaviors.

On top of this data, the benchmark uses a frozen pretrained backbone plus a lightweight trainable head, following a common foundation-model adaptation recipe. Image models (DINOv3-B, DINOv3-L, ResNet50) process single frames; video models (VideoMAE-B, VideoMAE-L, V-JEPA-2-L) process 16-frame windows sampled every frame. Frames are downscaled to 224 x 224 px. Token embeddings or final feature maps are aggregated either by mean pooling or by a learnable cross-attention pooling head with a single query, then passed to a two-layer MLP classifier. Videos were split roughly 70% train and 30% test with splits drawn from different sites and dates.

Evaluation uses macro-averaged F1, precision, and recall. Because point-based behaviors span short intervals, the primary results use a 7-frame tolerance, roughly plus or minus 0.1 seconds at 60 FPS. The paper also benchmarks balanced sampling and focal loss to handle the long-tailed behavior distribution, and evaluates Faster R-CNN variants, YOLO models, and RT-DETR on a detection benchmark split 70% train, 20% val, 10% test with standard COCO metrics.

Why This Matters

This is a benchmark paper, and its impact lies in exposing a measurement gap rather than in a modeling advance. It shows that leading vision foundation models score far below what ecological analysis requires on real field footage, and it argues that data collected under messy, variable field protocols — rather than curated for computer vision — is the more useful test of deployment readiness. The paper also documents concrete structural limits of its own data, such as the unquantified tracking quality in CoralCam and the missing focal-individual localization in FishFollow, giving future work specific targets.

Real-world applications the work supports:

• Reef monitoring and conservation. Automated behavioral tracking would let ecologists observe group behavior, predator-prey interactions, and site-specific dynamics across many fixed sites simultaneously.
• Reducing annotation cost. CoralCam's pipeline artifact release and the detection baselines are framed as a way to cut the expert annotation burden that currently gates marine ecological monitoring.
• Citizen science. The FishFollow design deliberately mirrors diver and citizen-scientist footage, with the stated aim of enabling future integration of crowd-sourced video.
• Neuroscience and behavioral ecology. The paper positions marine species as pivotal model organisms for neuroscience research, where behavior annotation is a shared bottleneck.

Industry relevance: the paper does not report commercial partnerships, deployment, or industry benchmarks — it is an academic dataset and benchmarking effort. Its practical relevance to industry is indirect, in the same sense as any hard in-the-wild video benchmark: it stresses frozen-backbone adaptation, pooling head design, long-tailed class imbalance, and low-contrast, occlusion-heavy video, all of which recur in real-world video analytics pipelines outside marine science.

Future Directions

1. Localization annotations for FishFollow. The paper identifies the absence of bounding-box or segmentation annotations tying behavior labels to the focal individual as the principal limitation, suggesting focal-animal localization as the highest-value addition.
2. Reliable tracking evaluation. CoralCam currently offers no reliable way to evaluate tracking quality, since exhaustive ground truth is infeasible and region-restricted evaluation does not recover true performance. The authors raise this as an open problem.
3. Imbalance-aware and data-efficient methods. The authors posit that WildFin is a challenging testbed for retrieval or metric learning approaches, and highlight the need for more data-efficient, imbalance-aware methods given that rare behaviors carry high scientific value and manual annotation does not scale.
4. Bridge boundary uncertainty to evaluation. FishFollow disagreement concentrates at behavior onsets and offsets rather than in the behaviors assigned, and frame-exact evaluation penalizes models for the same transitions humans annotate inconsistently.

Target Audience

Researchers working at the intersection of computer vision and animal behavior, particularly those studying video foundation models, long-tailed recognition, or frozen-backbone adaptation on hard real-world footage. It is also aimed at marine and behavioral ecologists seeking automated analysis of coral reef video, and at practitioners in bioacoustics-adjacent or camera-trap-adjacent fields who face the same annotation bottleneck and uncontrolled field conditions. Readers looking for a new model architecture will not find one here; the value is in the dataset, its documented limitations, and the baseline measurements.

Authors’ abstract

Recent advances in field technology have led to a massive influx of in-the-wild video data for ecological science. The primary bottleneck in leveraging this data is the high cost of expert annotation. While computer vision offers a potential solution, current models frequently fail when deployed in complex marine environments. To characterize these failures, we introduce WildFin, a novel benchmark for fish behavior recognition collected and annotated by ecologists.WildFin spans two critical real-world paradigms: stationary cameras monitoring groups of fish and dynamic divers following individual subjects. The dataset represents a massive curation effort, involving 1,350 hours of fieldwork and 600 hours of expert annotation to produce 9 hours of behavioral data with over 2 million frame-by-frame labels. We benchmark modern vision foundation models and quantify tradeoffs between static and spatiotemporal architectures, revealing the substantial gap that remains between current model capabilities and the demands of real-world underwater behavioral analysis. Project website: https://team-wildfin.github.io/.