Adaptive Training for Nautical Rules of the Road

AI summary

Overview

Research area: Human-Computer Interaction, specifically adaptive simulation-based training for maritime navigation (nautical rules of the road / COLREGs).

Technical level: Intermediate. The paper is readable without specialist background, but it assumes familiarity with basic experimental design terminology (pre-test/post-test, t-test, effect size) and with the concept of a continuously adapted difficulty parameter.

Scope: A controlled user study with 30 university students comparing an adaptive ship-navigation trainer (RAFT) against a non-adaptive version (RoR) that shares the same content, scenario framework, and interface.

What This Paper Is About

Students learning the International Regulations for Preventing Collisions at Sea (COLREGs) often struggle to master them, and practicing on real vessels is costly and potentially dangerous. Conventional computer-based trainers present the same sequence or fixed difficulty level to every student, which can leave some learners bored and others overwhelmed. The paper's goal is to test whether adapting scenario difficulty to each student's measured performance, and delivering immediate context-sensitive feedback, produces better learning outcomes and greater engagement than a non-adaptive trainer with identical content and interface.

Key Contributions

1. An adaptive simulation trainer (RAFT) that adjusts a continuous scenario difficulty value in response to a student's score on an automatically generated collision-risk scenario, where difficulty controls time of day, time to closest point of approach, and number of ships.
2. A controlled comparison of adaptive versus non-adaptive training in which instructional content, scenario-generation framework, and user interface are held constant, isolating the effect of the adaptation and feedback configuration.
3. Objective learning evidence: RAFT students scored significantly higher on the post-test and answered significantly faster, with the authors reporting p < 0.0001 and d > 2 for both comparisons.
4. Subjective evidence from a survey administered after every student had used both trainers, covering perceived effectiveness, overall quality, engagement, feedback, and question difficulty.

Main Findings

• Learning gains: Mean normalized post-test scores were 74.3% for the RAFT condition and 61.4% for the RoR condition, compared with pre-test means of 57.1% and 62.3% respectively. The between-condition difference on the post-test was significant (p < 0.0001, d > 2).
• Within-condition change: RAFT students improved significantly from pre-test to post-test (p < 0.0001). RoR students showed no significant increase; their mean score decreased from 62.3% to 61.4%, which was not statistically significant.
• Response speed: Pre-test mean time per question was 91.9 seconds (RAFT) and 90.0 seconds (RoR). On the post-test, RAFT dropped to 79.5 seconds while RoR rose to 99.4 seconds; the between-condition post-test difference was significant (p < 0.0001, d > 2). The RoR change from pre-test to post-test was not significant.
• Difficulty progression: All 15 students in the RAFT condition moved from the novice difficulty range into the intermediate range (δ ∈ [0.34, 0.67]) by the end of training, with final difficulty values ranging from δ = 0.4 to δ = 0.6. Three students reached the highest observed final value of δ = 0.6.
• Perceived effectiveness: 22 of 30 students (73%) judged RAFT the more effective trainer; 8 students (27%) chose RoR.
• Engagement: 22 students rated RAFT "Very engaging," compared with 9 students who gave RoR that rating.
• Feedback: 21 students selected "Very well" for RAFT's feedback and guidance, compared with 10 students for RoR.
• Difficulty perception: 27 of 30 students (90%) agreed or strongly agreed that RAFT questions varied in difficulty, compared with 22 of 30 (73%) for RoR.
• Overall quality and difficulty satisfaction: Students generally rated both trainers positively; no student selected "Somewhat dissatisfied" or "Very dissatisfied" for either trainer's overall quality, and no student reported dissatisfaction with question difficulty in either system.
• Interpretation caveat: The authors note that the post-test contained four intermediate-level questions and that these harder questions likely contributed to the between-condition differences. The survey does not establish which specific RAFT feature caused the differences in perception.

Methodology in Plain English

The researchers built two versions of the same ship-driving trainer. Both generate collision-risk scenarios in which the student operates an ownship, examines surrounding vessels, and fills out a navy-style Captain's Report identifying the target vessel, its location and motion, closest-point-of-approach measures (RCPA, TCPA, collision risk), the applicable rule, and a recommended course of action. The report contains 16 scored components; the overall scenario score is a weighted sum with weights set by subject matter experts.

The non-adaptive version, RoR, has three fixed difficulty levels (novice, intermediate, advanced) selected by the student or instructor, and gives feedback only at the end of a quiz. The adaptive version, RAFT, keeps difficulty as a continuous value δ between 0 and 1, starting students at δ = 0.2 and updating it after each scenario with δ(t+1) = δ(t) + 0.05·S(t), where S(t) is the normalized score. That difficulty value is then mapped to time of day, number of ships, and time to closest point of approach, with ±10% randomness added so configurations are not identical. RAFT also gives immediate feedback, highlighting correct answers in green and incorrect ones in red and annotating the radar display to point at the relevant evidence.

Thirty university students were recruited and randomly assigned, 15 to each condition. The study lasted approximately 90 minutes. All students received an introduction or review of the rules and a trainer demonstration, then took the same 10-question pre-test. Each group practiced with its assigned trainer — RoR students at a fixed novice level, RAFT students starting at δ = 0.2. All students then took the same 10-question post-test, which the authors describe as containing six novice-level and four intermediate-level scenarios; intermediate questions were weighted twice as heavily as novice questions to account for the difficulty difference. After the post-test, each student used the trainer they had not used, so all 30 could compare both systems in a final survey. Reductions in question-difficulty weightings aside, the key comparisons were between-condition post-test scores and response times, plus the survey responses.

Why This Matters

Impact on research: The study is a clean test of adaptation with content, interface, and scenario framework held constant, which addresses a recurring weakness in adaptive-training research where many reported gains cannot be attributed to adaptation itself. It also reports a case where adaptation improved both an objective measure (post-test score and response time) and subjective measures, in contrast to prior work such as the laparoscopic surgery trainer comparison cited in the paper, where preference improved but measured performance did not.

Real-world applications:

• Maritime education and cadet training programs that need to teach COLREGs efficiently before students go to sea.
• Bridge-team and officer-of-the-deck preparation, since the Captain's Report mirrors what an OOD communicates to a captain not on the bridge.
• Conversion of existing fixed-level simulation trainers into performance-based systems without rebuilding content or interface.
• Other safety-critical simulation domains where scenario difficulty can be parameterized, following the pattern of adaptive X-ray screening and stress-inoculation training cited in the paper.

Industry relevance: The work was supported by grant N00014-22-1-2122 from the Office of Naval Research, and the training task, report format, and terminology are drawn from naval practice. Simulation vendors and maritime training institutions could apply the continuous-difficulty approach to existing courseware, and the finding that adaptive students answered harder questions faster is directly relevant to operational settings where decision speed matters alongside accuracy.

Future Directions

• Extend the adaptation rule so difficulty can decrease as well as increase in response to student performance, rather than only accumulating upward.
• Separate the effects of difficulty adaptation from those of immediate feedback using experimental conditions that vary the two features independently.
• Move beyond a single difficulty value by adapting multiple scenario variables independently, which may let the system target a student's specific weaknesses.
• Investigate whether data from previous students can improve the initial training configuration for new students or cohorts.
• Determine whether the findings generalize beyond the 30 university students studied, to other populations and operational settings.

Target Audience

Researchers and practitioners in adaptive training, educational technology, and human-computer interaction; maritime training developers and instructors who build or select simulation courseware; and naval or commercial maritime training organizations interested in converting fixed-difficulty systems into performance-based ones. The paper is also useful to graduate students looking for a compact, clearly structured example of a controlled between-subjects training study with both objective and subjective outcome measures.

Authors’ abstract

Knowledge of the nautical rules of the road is essential for safe ship navigation and collision avoidance. We evaluated adaptive and non-adaptive versions of a ship-driving simulation trainer designed to assess and improve students' knowledge and application of these rules. We randomly assigned 30 university students to an adaptive or non-adaptive training condition and measured learning using pretest and post-test scores. Students who received adaptive training achieved significantly higher post-test scores than those who received non-adaptive training (p &lt; 0.0001). After the post-test, all students experienced both versions of the trainer and compared them in a survey. Of the 30 students, 73% judged the adaptive trainer more effective, and 22 rated it "very engaging," compared with 9 who gave the non-adaptive trainer the same rating. These findings provide evidence that adapting scenario difficulty and providing immediate, context-sensitive feedback can improve both learning outcomes and student engagement in simulation-based training.