Realistic Curriculum Reinforcement Learning for Autonomous and Sustainable Marine Vessel Navigation

AI summary

Overview

Research area: Reinforcement learning for autonomous maritime navigation, combining curriculum learning, generative simulation, and machine-learning-based fuel estimation to serve sustainability objectives.

Technical level: Advanced. The paper assumes familiarity with Markov Decision Processes, actor-critic policy optimization (PPO), diffusion models, gradient-boosted regression, and maritime domain concepts such as DCPA/TCPA and COLREGs.

Scope: The paper proposes and simulates a curriculum reinforcement learning framework — trained inside a diffusion-augmented, AIS-derived marine environment and coupled to an XGBoost fuel consumption predictor — for safe, low-emission vessel navigation in a region of the Indian Ocean.

What This Paper Is About

Maritime shipping moves roughly 90% of global trade volume but produces billions of tons of greenhouse gas emissions annually, and the International Maritime Organization has called for shipping emissions to fall by at least 50% by 2050. Navigation today still depends heavily on human experience, which is often neither emission-aware nor immune to the human error implicated in disasters such as the SANCHI tanker collision, which killed 32 crew members and released or burned more than 100,000 tons of petroleum products. The paper's goal is a single learning-based navigation agent that simultaneously reduces fuel use, avoids collisions, keeps to schedule, and reliably reaches its destination.

Key Contributions

1. A curriculum reinforcement learning (CRL) framework for vessel navigation that combines curriculum learning with reinforcement learning to jointly address safety, emissions, timeliness, and goal completion in complex ocean environments.
2. A realistic, data-driven simulation environment built from real-world AIS vessel movement data and enhanced with a diffusion model that generates synthetic, plausible vessel trajectories to make traffic conditions more diverse and dynamic.
3. An integrated fuel consumption prediction module trained on historical operational records from ocean-going vessels, giving the agent realistic emission feedback on which to base decisions.
4. A lightweight, image-input policy agent with a multi-objective reward designed for stable learning in continuous action spaces, validated across three simulated navigation instances in the Indian Ocean.

Main Findings

• XGBoost was the strongest fuel predictor. On an independent test set, XGBoost reached an R² of 86.10% with RMSE 0.3827 and MAE 0.1802, which the authors describe as the highest R² among the compared models. Random Forest was close behind at R² 86.06%, RMSE 0.3832, and MAE 0.1752 (slightly better MAE than XGBoost). Multi-Layer Perceptron reached R² 77.06%, and Support Vector Regression was weakest at R² 45.01%.

• Instance Case 1 (21-hour window, 2024-02-08 00:00:00 to 21:00:00 UTC). The CRL agent recorded AR 154.018, AFC 18.015, and ASS 0.888, the lowest fuel consumption and lowest safety score (lowest risk) in the comparison. CL-ABDDQN had a slightly higher reward (AR 155.527) but higher AFC (20.509) and ASS (1.466); CL-A2C failed the task with AR -82.034; DDPG had AR 150.424, AFC 33.533, ASS 1.84. In this paper a higher ASS indicates greater navigational risk.

• Instance Case 2 (46-hour period, 2024-02-08 00:00:00 to 2024-02-09 22:00:00 UTC). CRL achieved the lowest accumulated fuel consumption of the compared models (AFC 19.963) while maintaining a balanced AR of 294.148 and ASS of 4.754. CL-ABDDQN obtained the highest reward (298.026) but also the highest ASS (5.543), and CL-A2C produced the lowest ASS (3.952), described as the safer trajectory.

• Instance Case 3 (21-hour window, 2024-02-08 00:00:00 to 21:00:00 UTC). CRL achieved the lowest ASS (2.143), indicating the safest behavior, at the cost of higher fuel consumption (AFC 28.618) than the most fuel-efficient model; DDPG was the most fuel-efficient (AFC 9.002) but had the highest ASS (2.710) among the non-failing models. CL-A2C again failed, with AR -100.699, AFC 35.154 and ASS 7.258. The authors state CRL is comparable to CL-ABDDQN here rather than clearly superior.

• Curriculum learning is essential. In the ablation, removing it (CRL w/o CL) produced a negative accumulated reward of -708.765 in Case 2, versus 294.148 for full CRL — a failure to complete the task. In Cases 1 and 3, full CRL outperformed the ablated variant across all metrics except fuel consumption in Case 3 (19.557 for w/o CL versus 28.618 for CRL). Training curves showed CRL converging faster and with more stability, while the no-CL variant showed slower early improvement and significant performance degradation.

• Curriculum design specifics. The shrinkable distance-to-goal threshold starts at ω₀ = 5 nautical miles; the safety buffer multiplier is τ = 4, the maximum time to closest approach is t_max = 15 minutes, and the minimum safety distance is floored at 0.5.

• State and action representation. The self-vessel state is a 9-dimensional vector (with sine/cosine encoding of angles), and the environment is a 64 × 64 × 3 image tensor covering a 10 nm × 10 nm area with channels for occupancy, SOG, and COG. Actions are continuous: a heading increment and Speed Through Water. The fuel prediction model used 86 input features and predicted fuel consumption rate across four fuel types (HFO, LSFO, DO, LSGO).

Methodology in Plain English

The authors built a training "world" for a ship-driving AI rather than testing on a real vessel. First, they modelled how much fuel a ship burns. They trained a gradient-boosted regression model (XGBoost) on two years of real operational records from several hundred international ocean-going vessels, using inputs such as travel distance, latitude and longitude, speed over ground, ship dimensions (length overall, beam, gross tonnage), ship type, and time variables like month, day and hour. The output is a fuel consumption rate in metric tons per hour, aggregated across the main engine, auxiliary engine and auxiliary machinery.

Second, they built the maritime environment. Real AIS trajectories provided the base traffic, and a diffusion model — the same family of generative models used for images — was trained to add new synthetic trajectories that look statistically like real ship movements. The forward process gradually adds Gaussian noise to a trajectory; the reverse process learns to denoise back into a plausible track. Monthly-varying ocean currents based on historical direction and speed distributions were also injected as dynamic disturbances.

Third, they set up the driving problem as a Markov Decision Process and solved it with Proximal Policy Optimization, using an actor network and a critic network. The actor processes two input streams: the vessel's own state vector through dense layers, and a small image of the surrounding traffic through two separable convolutional layers; the features are concatenated and used to produce the action distribution. The critic uses a similar structure and outputs a scalar value estimate. Actions are continuous — turn a bit and pick a speed — and the ship's resulting course and speed over ground are computed by simulating the effect of currents.

Fourth, they designed the reward so the agent balances four things: progress toward the destination, fuel burned (scaled by gross tonnage), collision risk (computed from distance and time to closest point of approach against other vessels, with a safe-distance threshold based on the two ships' lengths and beams), and lateness penalties. Curriculum learning is implemented as a threshold that starts at 5 nautical miles and shrinks over episodes, and a rule that penalizes the agent if it moves away from the destination — making early training easier and progressively tightening the task.

Finally, they tested this in a high-traffic Indian Ocean sea area with hundreds of simulated vessels across three navigation instances. Because open AIS data for the region was limited and operational fuel data is typically confidential, the authors used proprietary data obtained through industry collaboration. Baseline methods (CL-ABDDQN, CL-A2C, DDPG) were trained and evaluated in the same environment and reward structure, with only the underlying reinforcement learning architecture changed. An ablation compared full CRL against the same architecture trained without curriculum learning.

Why This Matters

For research: The paper argues that prior deep reinforcement learning work in maritime navigation typically targets one objective at a time — routing and scheduling for fuel, or collision avoidance for safety — and that multi-objective optimization remains hard for policy learning. It offers a recipe combining curriculum learning, generative simulation, and a learned physical cost model as a way to make multi-objective, continuous-control navigation tractable. It also contributes a coupling between a learned fuel model and a reinforcement learning reward, so emission feedback is data-driven rather than assumed.

Real-world applications:

• Emission-aware voyage planning that proposes speed and heading adjustments which cut fuel burn while still arriving on time, supporting slow steaming and Just-In-Time arrival strategies.
• Collision avoidance assistance for congested waterways, using a risk measure built from closest-point-of-approach distances and times.
• Maritime Autonomous Surface Ship (MASS) decision support, providing continuous maneuvering recommendations to crew or autonomy stacks.
• Training and scenario generation, since the diffusion-augmented simulator can produce diverse synthetic traffic conditions for testing procedures and personnel without needing real congested encounters.

Industry relevance: The paper frames operational measures — routing, speed regulation, Just-In-Time arrival — as cheaper and faster to deploy across the existing global fleet than fuel switching or vessel retrofitting, citing IMO estimates that such improvements could deliver emission reductions of up to 30–40% by 2030. That places the work directly in the interim window the International Maritime Organization has identified, while low- and zero-emission technologies mature.

Future Directions

• Generalization beyond the studied region. Evaluation is confined to one Indian Ocean sea area with proprietary fuel and traffic data; whether the policy transfers to other basins, traffic densities, or vessel classes is not established.
• Closing the simulation-to-reality gap. All results come from simulation. Validation on real vessels, or at least against logged historical voyages, would test whether the reward balance holds under sensor noise, unmodelled disturbances, and real traffic behavior.
• Widening the fuel and emissions model. The predictor covers four fuel types and engine-plus-machinery consumption, but the paper does not report per-pollutant (CO₂, NOₓ, SOₓ) conversion or verification against measured emissions, which decarbonization compliance regimes would need.
• Scaling and hybridizing the policy. The authors describe the agent as lightweight and note that CL-A2C failed outright in two of three instances, while CRL and CL-ABDDQN traded places on different metrics — suggesting room to explore architectures, curriculum schedules, and reward weightings (α, the ω thresholds, τ) more systematically.
• Regulatory alignment. The reward uses distance- and time-based risk measures informed by COLREGs-style domains, but the paper does not report whether the learned behavior was formally checked against COLREGs rules, which would matter for any deployment.

Target Audience

This paper suits reinforcement learning researchers working on multi-objective continuous control and curriculum learning; maritime and transportation engineering researchers interested in autonomous navigation and decarbonization; and industry or regulatory practitioners evaluating AI-based decision support for vessel operations. Readers need comfort with deep reinforcement learning terminology and maritime risk metrics to get the most from it; those looking for real-vessel trials or open benchmark results should note that the study is simulation-based on proprietary data.

Authors’ abstract

Sustainability is becoming increasingly critical in the maritime transport, encompassing both environmental and social impacts, such as Greenhouse Gas (GHG) emissions and navigational safety. Traditional vessel navigation heavily relies on human experience, often lacking autonomy and emission awareness, and is prone to human errors that may compromise safety. In this paper, we propose a Curriculum Reinforcement Learning (CRL) framework integrated with a realistic, data-driven marine simulation environment and a machine learning-based fuel consumption prediction module. The simulation environment is constructed using real-world vessel movement data and enhanced with a Diffusion Model to simulate dynamic maritime conditions. Vessel fuel consumption is estimated using historical operational data and learning-based regression. The surrounding environment is represented as image-based inputs to capture spatial complexity. We design a lightweight, policy-based CRL agent with a comprehensive reward mechanism that considers safety, emissions, timeliness, and goal completion. This framework effectively handles complex tasks progressively while ensuring stable and efficient learning in continuous action spaces. We validate the proposed approach in a sea area of the Indian Ocean, demonstrating its efficacy in enabling sustainable and safe vessel navigation.