Foundation models on the bridge: Semantic hazard detection and safety maneuvers for maritime autonomy with vision-language models

AI summary

Overview

Research area: Maritime robotics and autonomous surface vessels (ASVs)/maritime autonomous surface ships (MASS), combining vision–language models (VLMs), out-of-distribution (OOD) anomaly handling, human-in-the-loop remote operation, and maritime regulation (the draft IMO MASS Code).

Technical level: Advanced. The paper assumes familiarity with camera projection geometry, maritime segmentation, line-of-sight (LOS) path following, and foundation-model prompting.

Scope (one sentence): The paper introduces and empirically evaluates "Semantic Lookout," a camera-only, candidate-constrained VLM module that selects one short-horizon, human-overridable fallback maneuver during the alert-to-override window on a maritime vessel, tested offline on 40 harbor scenes and in a live harbor field run.

What This Paper Is About

Classical maritime autonomy stacks handle geometry — radar/LiDAR detection, AIS tracking, and rule-based collision avoidance — but cannot interpret meaning, such as a diver-down flag (people in the water) or a nearby fire, which are rare OOD scenes that normally require human judgment. The draft IMO MASS Code requires vessels to detect departure from their Operational Design Domain, enter a predefined fallback state, notify the operator, and remain immediately overridable without altering the voyage plan, which leaves an "alert-to-takeover" gap in which the vessel must stay safe on its own. The paper's goal is to test whether a VLM, constrained to choose among pre-vetted water-valid trajectory candidates drawn on a single camera image (or to hold station), can serve as that IMO MASS Code–aligned fallback maneuver selector.

Key Contributions

1. An IMO MASS-aligned fallback maneuver architecture. The authors formalize the alert → fallback maneuver → override loop as the design constraint: short-horizon, pre-approved actions (or Station-keeping), immediate override, and no overall voyage-plan edits, implemented as three modules (perception and proposal; decision; execution and authority).
2. A camera-only candidate set and gating procedure. From a single image frame, the system computes a water mask and pixel-clearance map using the embedded Water Segmentation and Refinement (eWaSR) model, samples and projects short motion primitives, gates them in pixel space, and reduces them by farthest-point thinning to a world-anchored, numbered candidate set used consistently online and offline.
3. A VLM fallback maneuver selector with low-level execution and human override. A strict-schema VLM decision over the numbered candidates, plus an execution and authority path that publishes world-fixed waypoints to a line-of-sight waypoint follower with a direct joystick override blend, ensuring immediate human authority.
4. Evidence of feasibility. A small-n fast anomaly monitor experiment (Appendix A), offline evaluation on overlays with human-labeled ground truth, short-horizon fire "risk-relief" comparison against geometry-only baselines, and a live alert → fallback maneuver → operator handover experiment with immediate joystick override.

Main Findings

• Compact models can be fast enough. Sub-10 s models retain most of the awareness of slower state-of-the-art models, with the measurements taken on 40 harbor scenes via per-call scene understanding and latency.
• The selector beats geometry-only baselines. The fallback maneuver selector outperforms geometry-only baselines in alignment with aggregated human Accept/Best judgments on the same gated candidate set.
• Fire scenes get more room. On fire hazard scenes the selector increases standoff distance relative to the baselines; the paper reports the direction of the effect rather than a numeric magnitude in the content available.
• End-to-end operation verified in the field. A live harbor field run verifies the full alert → fallback maneuver → operator handover chain with immediate joystick override and ROC-legible presentation.
• Safe defaults are built in. If the VLM call times out, returns an API error, or produces invalid/non-conforming JSON, the system selects Station-keeping (choice_id = 0) and notifies the ROC; the same default applies when gating yields no feasible candidates (K = 0) and when majority voting in the ensemble produces no strict majority.
• Anomaly monitoring transfers to maritime. Appendix A provides details and small-n evidence that the fast embedding-space monitor adapted from Sinha et al. (2024) also functions in the maritime domain.
• Human takeover is slow, motivating the approach. Cited work reports degraded operator performance supervising three vessels versus one, takeover times of more than 20 s even in simplified inland-vessel scenarios, and response/situation-awareness recovery on the order of minutes for conventional merchant ships.

Methodology in Plain English

At the moment an anomaly alert fires, the vessel takes a single calibrated forward camera frame. A maritime segmentation model (eWaSR) produces a binary water mask, and the system computes, for every water pixel, the distance to the nearest non-water pixel. From a fixed anchor at the bow (4.0, 0.0) m, the system samples straight motion primitives whose endpoints lie in an annulus and within a forward half-angle, discretizes each into points, and projects them into the image using the known camera model. A candidate survives only if every visible projected sample lies on water (mask = 1) and clears a pixel margin of d_min = 40 px. Surviving candidates are thinned down to K = 15 spread-out options and drawn as a numbered overlay, where ID 0 means Station-keeping.

The VLM receives only the overlaid image and a prompt, and must return a strict JSON object containing short free-text fields ("see," "implications," "action"), an integer choice_id between 0 and K, and a confidence value. The runtime uses one call per alert (FB-1). For evaluation, the authors make n = 3 independent calls with different seeds and aggregate by strict majority (FB-3), with ties falling back to Station-keeping — a self-consistency / test-time compute-scaling pattern. Candidates are frozen in the world (NED) frame at the alert instant so that perception latency is decoupled from actuation.

Execution uses a line-of-sight follower with an acceptance radius of 7.5 m and a 10 m lookahead at a cautious anomaly-mode speed of 1.0 kn, running on a dynamic positioning system with azimuth-thruster allocation. If the action is Station-keeping (a = 0), a single world point at the current navigation pose is published instead. A shared-control override blend combines the machine command and the human joystick input, with an override blending factor α where 0 means autonomy with immediate manual intervention and 1 means full manual; MANUAL strictly dominates AUTONOMY. Geometry-only, semantics-agnostic heuristics (Keep-station / Keep-course / Keep-starboard / Forward / Clearance) on the same gated set serve as baselines and are described as a camera-only simplified proxy to scenario-based MPC (SB-MPC). Closed-loop experiments were run on the Shore Control Lab workstation from Gusev et al. (2025), and the vessel used is identified as the milliAmpere1 experimental ASV.

Why This Matters

Impact on research. The paper positions semantic, foundation-model-based anomaly handling as a distinct and useful role for VLMs in maritime autonomy, arguing that off-the-shelf zero-shot semantics can be paired with classical domain-specific perception and planning rather than replacing it. The authors claim this is the first use of foundation models in the real maritime domain for anomaly handling specifically and for vision-to-action in general, noting a single near exception (a rudimentary simulated VLM vision-to-action ASV stack with no anomaly detection or handling, by Kim and Choi, 2025). It also contributes a concrete, regulated framing (alert → fallback maneuver → override) that ties safety-module design directly to draft IMO MASS Code obligations.

Real-world applications (bullets).

• Remote operation centers supervising one-to-many vessel fleets, where a legible, short-horizon fallback buys time during the alert-to-override gap.
• Harbor and inland-water ASV operations facing semantic hazards (diver-down flags, keep-out lines, vessels on fire) that are outside detector training distributions.
• Fallback-state implementations for MASS seeking alignment with the draft IMO MASS Code's requirements for predefined fallback, operator notification, and immediate override.
• Human-machine interface design for shore control, since the model's brief rationale text feeds the ROC display used in the formative handover study.

Industry relevance. Maritime autonomy vendors, classification societies, and regulators can read the paper as an early demonstration that foundation models can be constrained — through candidate gating, strict output schemas, safe defaults, and hard manual priority — into a form compatible with human-authority requirements rather than conflicting with them.

Future Directions

• Domain adaptation. The authors' stated motivation is future work on domain-adapted, hybrid autonomy pairing foundation-model semantics with multi-sensor bird's-eye-view perception and short-horizon replanning.
• Stronger range-aware gating. The pixel-space margin (d_min = 40 px) is described as a conservative camera-only proxy; the paper suggests a production system would replace or augment this gating with range-aware perception such as stereo, LiDAR, or radar.
• Toward one-to-many supervision. The work is framed as a step toward robust single-operator, multi-vessel supervision, which the authors state has not yet been demonstrated with acceptable performance in realistic conditions; they evaluate only a single vessel here.
• Open questions the text raises. How missed or late anomaly alerts should be handled is left to the upstream monitor and nominal stack, and the paper's reported evidence is a proof of concept with small-n anomaly-monitor evidence, one field run, and no numerical latency, accuracy, or standoff-distance values reported in the available content.

Target Audience

Robotics and maritime autonomy researchers working on OOD anomaly detection, safety fallback behaviors, and foundation-model integration; practitioners building ASV/MASS autonomy stacks and shore-side remote operation centers; human-factors and HMI researchers studying handover in automated ships; and regulators, class societies, and maritime industry engineers tracking how foundation models can be constrained to fit IMO MASS Code-style requirements for operator authority and predefined fallback states.

Authors’ abstract

The draft IMO MASS Code requires autonomous and remotely supervised maritime vessels to detect departures from their operational design domain, enter a predefined fallback that notifies the operator, permit immediate human override, and avoid changing the voyage plan without approval. Meeting these obligations in the alert-to-takeover gap calls for a short-horizon, human-overridable fallback maneuver. Classical maritime autonomy stacks struggle when the correct action depends on meaning (e.g., diver-down flag means people in the water, fire close by means hazard). We argue (i) that vision-language models (VLMs) provide semantic awareness for such out-of-distribution situations, and (ii) that a fast-slow anomaly pipeline with a short-horizon, human-overridable fallback maneuver makes this practical in the handover window. We introduce Semantic Lookout, a camera-only, candidate-constrained VLM fallback maneuver selector that selects one cautious action (or station-keeping) from water-valid, world-anchored trajectories under continuous human authority. On 40 harbor scenes we measure per-call scene understanding and latency, alignment with human consensus (model majority-of-three voting), short-horizon risk-relief on fire hazard scenes, and an on-water alert->fallback maneuver->operator handover. Sub-10 s models retain most of the awareness of slower state-of-the-art models. The fallback maneuver selector outperforms geometry-only baselines and increases standoff distance on fire scenes. A field run verifies end-to-end operation. These results support VLMs as semantic fallback maneuver selectors compatible with the draft IMO MASS Code, within practical latency budgets, and motivate future work on domain-adapted, hybrid autonomy that pairs foundation-model semantics with multi-sensor bird's-eye-view perception and short-horizon replanning. Website: kimachristensen.github.io/bridge_policy