UEOF: A Benchmark Dataset for Underwater Event-Based Optical Flow

AI summary

Overview

Research area: Underwater computer vision, event-based (neuromorphic) sensing, optical flow estimation, and physically-based rendering.

Technical level: Advanced. The paper assumes familiarity with event cameras, contrast maximization, and optical flow evaluation metrics.

Scope: The paper introduces UEOF, the first synthetic underwater benchmark dataset that pairs ray-traced RGBD underwater rendering with temporally dense pseudo-event streams and dense ground-truth optical flow, depth, and camera ego-velocity, then benchmarks state-of-the-art event-based optical flow methods on it.

What This Paper Is About

Underwater vision is hard because light is absorbed in a wavelength-dependent way, particles scatter light, and turbidity blurs imagery, which means ordinary cameras degrade and accurate ground-truth motion is nearly impossible to capture underwater. Event cameras promise microsecond resolution and high dynamic range, but research on underwater event-based perception has stalled because no dataset combines realistic underwater optics with accurate optical flow labels. The authors build such a dataset from physically-based ray-traced underwater sequences and use it to measure how underwater light transport affects event formation and motion estimation.

Key Contributions

1. A synthetic underwater event dataset: the authors couple physically-based ray tracing (PBRT) rendering with high temporal resolution pseudo-events produced through video-to-event conversion, enabling event-based evaluation under realistic underwater physics.
2. Dense ground truth: the dataset supplies dense optical flow, depth, and pose from PBRT simulation, making it, per the authors, the first underwater dataset suitable for benchmarking event-based optical flow algorithms and the first to provide ground-truth optical flow and camera ego-velocity derived from PBRT simulation.
3. A broad benchmark: learning-based (supervised and unsupervised) and model-based contrast maximization optical flow methods are evaluated, revealing failure modes and domain-gap challenges specific to underwater scenes.
4. An analysis of underwater optics on event formation: the paper examines how attenuation, backscatter, caustics, and turbidity affect event generation and event representations, giving insight into whether event cameras suit underwater perception.

Main Findings

• Dataset scale and composition: UEOF contains 12 minutes and 51 seconds of data across 13,714 RGB frames (the paper separately states 13,704 evaluation intervals), for a total of 4.94 billion events across all scenes. It was built from two ray-traced sources: VAROS (five scenes with lossless RGB, surface normals, metric depth, 6-DoF poses, synchronized IMU, and depth-gauge data, with motion arising purely from camera ego-motion) and LOFUE (scenes with dynamic objects such as fish, vegetation, and floating debris, varied turbidity, and both static and moving cameras).
• Resolutions and timing: the dataset provides event streams at 960 × 540 and 1280 × 720; the 1280 × 720 setting coincides with the Prophesee EVK4 HD with the Sony IMX636ES HD sensor. LOFUE scenes run at 30 FPS RGB with 30 Hz flow and velocity evaluation; VAROS scenes run at 10 FPS RGB (10 Hz evaluation) with camera ego-velocities derived from 200 Hz logs.
• Motion statistics: flow is heavy-tailed, with a mean magnitude of 6.1 px, a median of 3.6 px, a 95th percentile of 21.20 px, and significant high-magnitude flow upwards of 80 px.
• Model-based methods win on shallow water: in the shallow-water results, MultiCM and EINCM achieved the lowest error rates, with EINCM reaching a minimum AEE of 1.01 px in scene4 (shallow water). EINCM's shallow-water AEE values across scene1 through scene5 were 4.52, 2.36, 2.06, 1.01, and 4.12 px, while MultiCM recorded 5.10, 1.37, 2.42, 1.37, and 2.27 px. OPCM is not reported for scene1 and scene5 in that table.
• The supervised baseline struggles most: E-RAFT, using checkpoints pretrained on the DSEC dataset, exhibited the highest errors, peaking at 16.32 px in scene5 (shallow water), with shallow-water AEE values of 8.67, 9.05, 10.52, 8.64, and 16.32 px.
• Deep water is uniformly harder: errors increase across all techniques in the deep-water environment. Deep-water AEE values were 3.94, 2.58, 3.37, 2.36, 2.56 px for EINCM and 4.44, 2.39, 3.32, 2.05, 2.35 px for MultiCM; E-RAFT recorded 5.82, 10.11, 4.88, 7.64, 3.61 px and MotionPriorCMax 6.93, 5.61, 6.24, 7.39, 5.25 px.
• N-pixel error rates confirm the gap: at the A1PE threshold, E-RAFT reached 97.9 in shallow-water scene2 and 98.37 in deep-water scene2, while model-based methods dropped sharply at larger thresholds, for example EINCM reaching 0.01 A20PE in shallow-water scene1 and MultiCM reaching below 0.01 A20PE in the same scene.
• Domain gap explanation: the authors hypothesize that the supervised network's failures come from feature distribution shift, because terrestrial datasets are dominated by high-frequency rigid edges (buildings, vehicles) whereas turbid underwater scenes are defined by soft, low-frequency textures from volumetric scattering.
• Caustics corrupt contrast maximization: refractive caustics create high-contrast light patterns that move across the seafloor independent of physical geometry, generating dense high-frequency events not caused by motion, which can cause contrast maximization objectives to falsely align and produce incorrect motion estimates.
• Dynamic objects are a failure mode: in dynamic sequences where independent objects such as fish traverse in different directions, MultiCM failed to resolve the conflicting motions, while MotionPriorCMax maintained robustness.
• Deep-water degradation sources: the paper attributes higher deep-water errors to haloing from UUV lighting, poor contrast from attenuation, and a low-texture seafloor, whereas shallow-water scenes contain strong, consistent sunlight that creates (noisier) seabed texture for tracking.
• Multimodal assistance is inconsistent: results show that multimodal approaches integrating velocity data, specifically OPCM, vary in how much they improve on dynamic scenes relative to their non-multimodal counterparts.

Methodology in Plain English

The authors did not collect data underwater. Instead, they reused two existing ray-traced underwater RGBD datasets: VAROS, which covers UUV infrastructure inspection and seafloor exploration with camera-ego-motion only, and LOFUE, which includes dynamic objects and both static and moving cameras. VAROS supplies metric depth, poses, IMU, and depth-gauge data but no optical flow; LOFUE supplies dense optical flow produced with the VisionBlender add-on, plus dynamic-object motion.

The RGB frames were rendered with Blender Cycles and then converted to video at 30 FPS (LOFUE) and 10 FPS (VAROS). The v2e toolbox, which models a dynamic vision sensor, turned these videos into event streams as (t, x, y, p) tuples exported through HDF5. Because standard v2e settings assume terrestrial contrast, the authors ran slow-motion interpolation so no pixel translates more than 1 px between adjacent frames, lowered the positive and negative thresholds to get events in low-texture or heavily attenuated regions such as sandy seafloors, and reduced the threshold variation to suppress false triggering from fixed-pattern noise without blurring real edges.

For ground truth, LOFUE flow came precomputed, while VAROS flow was reconstructed geometrically: pixels were back-projected into 3D with the depth map and intrinsics, moved by the rigid-body transform between frames, and re-projected to produce induced flow. Occluded points were removed by requiring the projected point to be inside the image, to satisfy the near-plane constraint (z greater than 0.1 m), and to pass a visibility test against the target depth map with a tolerance of 0.01 m. Velocities were stored as .npz archives with lin_vel, ang_vel, and microsecond timestamps; LOFUE velocities were extracted directly through the Blender Python API at each render step (30 Hz), and VAROS velocities were computed from 200 Hz poses using central finite differences with the standard quaternion-to-angular-velocity relationship.

Evaluation covered all scenes at dt = 1 using average endpoint error (AEE) and A_N_PE, the percentage of pixels with endpoint error below N pixels for N in {1, 2, 3, 5, 10, 20}. Learning-based (LB) methods were E-RAFT (supervised) and MotionPriorCMax (unsupervised); model-based (MB) methods were EINCM, MultiCM, and OPCM, using nonlinear optimization with contrast maximization objectives. The authors extended each method's dataloader for their formats, used each publication's DSEC-pretrained checkpoints for the learning-based methods, used identical accumulated event window sizes per scene for the model-based methods, and otherwise kept default configurations.

Why This Matters

The work gives the underwater event-based vision community its first benchmark that pairs physically realistic underwater rendering with dense optical flow and ego-velocity ground truth, exposing that current state-of-the-art methods do not transfer from terrestrial to underwater domains. Because no real-world underwater event dataset provides dense, accurate optical flow, this simulated benchmark fills a measurement gap that hardware alone cannot close.

Real-world applications include:

• UUV autonomy: localization, mapping, navigation, docking, and manipulation, all listed as tasks requiring reliable perception on unmanned underwater vehicles.
• Inspection and examination: infrastructure inspection and object examination in flooded or offshore environments.
• Marine biology and environmental monitoring: detection and segmentation of transparent marine organisms, as pursued by the Aqua-Eye and UTNet lines of work, plus oyster reef monitoring as in OysterSim.
• Object pose estimation and manipulation: subsea intervention tasks that depend on accurate motion estimation.

Industry relevance spans offshore energy and subsea infrastructure inspection, naval and defense autonomy (the work is supported by the Office of Naval Research under award number N000142512349), marine robotics manufacturers working with event sensors such as the Prophesee EVK4 HD, and simulation/rendering toolchains used to train perception systems where real underwater labels are unavailable.

Future Directions

• Extending UEOF with additional data and additional ground-truth modalities.
• Adding richer environmental diversity to cover more water types and conditions.
• Developing physically-grounded event noise models rather than adapting terrestrial v2e defaults.
• Closing the terrestrial-to-underwater domain gap so that supervised and contrast maximization methods generalize, including handling motion-independent events such as caustics and resolving independently moving objects.

Target Audience

Researchers and engineers in event-based vision, underwater robotics, and SLAM or visual odometry who need labeled data for motion estimation; developers of contrast maximization and learning-based optical flow methods who want an out-of-domain benchmark; and practitioners in subsea inspection, naval autonomy, and marine science evaluating whether event cameras are viable underwater. The paper is also relevant to simulation and rendering researchers interested in video-to-event pipelines built on physically-based underwater imagery.

Authors’ abstract

Underwater imaging is fundamentally challenging due to wavelength-dependent light attenuation, strong scattering from suspended particles, turbidity-induced blur, and non-uniform illumination. These effects impair standard cameras and make ground-truth motion nearly impossible to obtain. On the other hand, event cameras offer microsecond resolution and high dynamic range. Nonetheless, progress on investigating event cameras for underwater environments has been limited due to the lack of datasets that pair realistic underwater optics with accurate optical flow. To address this problem, we introduce the first synthetic underwater benchmark dataset for event-based optical flow derived from physically-based ray-traced RGBD sequences. Using a modern video-to-event pipeline applied to rendered underwater videos, we produce realistic event data streams with dense ground-truth flow, depth, and camera motion. Moreover, we benchmark state-of-the-art learning-based and model-based optical flow prediction methods to understand how underwater light transport affects event formation and motion estimation accuracy. Our dataset establishes a new baseline for future development and evaluation of underwater event-based perception algorithms. The source code and dataset for this project are publicly available at https://robotic-vision-lab.github.io/ueof.