A Collaborative Safety Shield for Safe and Efficient CAV Lane Changes in Congested On-Ramp Merging

AI summary

Overview

Research area: Safe control for Connected and Autonomous Vehicles (CAVs), combining Multi-Agent Reinforcement Learning (MARL) with Control Barrier Functions (CBFs) for cooperative highway on-ramp merging.

Technical level: Advanced. The paper assumes familiarity with CBFs, multi-agent reinforcement learning, kinematic vehicle models, and graph-based interaction topologies.

Scope: The paper proposes and evaluates MARL-MASS, a lane change controller that pairs a graph-based collaborative safety shield (MASS) with a MARL policy and a customised efficiency reward, tested in a congested on-ramp merging simulation against unsafe MARL, MARL-CS, and MARL-HSS baselines.

What This Paper Is About

Lane changing in dense traffic is a safety-critical task in which multiple CAVs make independent and concurrent decisions. Existing controllers tend to do one of two things: guarantee safety by assuming other vehicles behave in the worst case (which becomes inefficient in heavy traffic), or improve traffic efficiency collaboratively but without safety guarantees. This paper builds a safety shield that lets CAVs cooperate with their neighbours, so that they can create and use safe lane change opportunities in congested merging traffic while still strictly respecting a safety constraint.

Key Contributions

1. Multi-Agent Safety Shield (MASS): A novel collaborative safety shield, built on CBF constraints and extending the earlier Hybrid Safety Shield (HSS), that uses the control decisions of other CAVs (not just their worst-case behaviour) to evaluate safe control inputs. It is integrated with MARL to form the MARL-MASS lane change controller, with code available at https://github.com/hkbharath/MARL-MASS.

2. Interaction topology algorithm: An algorithm that constructs a graph representing which surrounding vehicles each CAV depends on (the leading vehicle, the adjacent leading vehicle, and, during a lane change, the adjacent leading vehicle in the target lane), enabling the evaluation of joint safe control actions. The adjacent rear vehicle is deliberately excluded from the topology to avoid deadlocks, though its state still constrains unsafe lane changes via lateral constraints under worst-case assumptions.

3. Customised reward function: A tailored reward for MARL controllers that carry a safety shield, prioritising traffic efficiency (smaller headway, higher speed, faster merging) because the shield guarantees safety. It drops the collision-avoidance term used in the baseline reward and combines headway, speed, and merging terms.

4. Empirical evaluation in dense traffic: Training and evaluation of MARL-MASS in a congested on-ramp merging scenario to show safety compliance, improved MARL training stability, and traffic efficiency compared with baseline controllers.

Main Findings

• Safety constraints are strictly respected: Unsafe MARL and MARL-CS produce a minimum time headway of 0.00 s, representing a collision. MASS and HSS both enforce the 0.5 s time headway constraint. In policy evaluation over 100 episodes, MARL-MASS recorded a minimum headway of 0.59 s (standard error 0.01), MARL-HSS-wCR 0.64 s (0.02), and MARL-HSS 0.72 s (0.03).

• Better road utilisation and efficiency than the safety-shield baseline: MARL-MASS achieved an average speed of 24.71 m/s (0.09) and a merging percentage of 83.36% (0.38), compared with MARL-HSS at 21.95 m/s (0.31) and 52.75% (7.07). The paper reports a 12.57% increase in average speed for MARL-MASS over MARL-HSS.

• The custom reward contributes independently: MARL-HSS trained with the custom reward (MARL-HSS-wCR) raised merging percentage from 52.75% to 69.95%, and MARL-MASS raised it further from 69.95% to 83.36%, which the authors attribute to MASS's collaborative gap-creating behaviour.

• Default reward destabilises MARL-MASS: MARL-MASS trained with the default reward (MARL-MASS-wDR) converged toward a smaller reward with higher variance; its evaluation showed 23.23 m/s (0.40) speed and 69.26% (3.65) merging, below the full MARL-MASS.

• Safety-free baselines are slightly more efficient but unsafe: MARL-CS reached 25.83 m/s (0.84) and 89.08% (2.42) merging, and unsafe MARL reached 26.26 m/s (1.52) and 76.91% (14.96) merging — both with a 0.00 s minimum headway. MARL-MASS has a 6.54% higher merging percentage than unsafe MARL and lags MARL-CS by 5.72%.

• MARL-HSS fails to converge in dense traffic: The paper reports that MARL-HSS struggles to find safe lane change opportunities in congested traffic, while MARL-MASS shows a consistently increasing learning curve with minimal variance.

• Training is computationally light: Training used approximately 1.5 GB of GPU memory and 3 GB of RAM on a shared server with an AMD EPYC 9654 processor, 4 x 40 GB Nvidia-L40 GPUs, and 1.5 TB memory.

Methodology in Plain English

The authors keep the existing layered vehicle control structure: a behavioural layer decides discrete manoeuvres (right, left, follow lane, speed up, slow down), and a motion planning layer turns that decision into a safe control input. Their safety shield sits at the motion planning layer.

Instead of assuming surrounding vehicles will behave in the worst possible way (as HSS does), MASS asks each CAV to consider the actual safe control input of the vehicles it depends on. To know who depends on whom, the authors run a simple algorithm that builds a graph: a vehicle following a lane depends on its immediate leading vehicle; if an adjacent vehicle is changing into its lane, the ego vehicle treats that vehicle as its leading vehicle and gives it priority; a vehicle that is itself changing lanes depends on both the leading vehicle and the adjacent leading vehicle in the target lane. Each CAV then solves a small optimisation problem that finds the minimum correction needed to keep the safety constraint satisfied, given the shared states and control parameters of its parents in the graph. Because every vehicle only needs to guarantee safety relative to its parents, safety holds collectively.

With safety handled by the shield, the authors redefine the reward so the learning policy focuses on efficiency rather than collision avoidance: rewards for maintaining a small headway, driving close to highway speeds, and merging early. They then train the MARL policy (MAPPO, the multi-agent form of Proximal Policy Optimisation) inside an extended version of the highway-env simulator, with 7-11 CAVs crossing a 100 m merging section, a 0.5 s time headway constraint, and a 180 m communication range. Reward weights were set to w_h = 1, w_s = 4, and w_m = 8. Training used a learning rate of 1e-4 across five random seeds for 20,000 episodes, with intermediate evaluation on 20 episodes every 200 training episodes. The best policies per controller were then evaluated on 100 separate episodes.

Why This Matters

Impact on research: The paper shows that a safety shield need not be conservative to be safe. By replacing worst-case assumptions about neighbours with graph-structured collaboration, it demonstrates that safety guarantees and traffic efficiency can coexist, and that a reward function designed specifically for shielded MARL policies improves training stability.

Real-world applications:

• Congested highway on-ramp merging, where limited gaps make merging difficult and where cooperative CAVs could create gaps for each other.
• Traffic flow management on urban highways, where the paper notes congestion costs around $224 billion in the United States in 2023, from 8.5 billion hours of delay and 3.3 billion gallons of wasted fuel.
• Fleet-level coordination for logistics and supply chains, where the paper lists increased supply chain costs among the consequences of congestion.
• Deployment of connected vehicle features using V2X communication, since MASS relies on vehicles sharing state, safe control inputs, and control parameters within a limited communication range rather than on a centralised server.

Industry relevance: The approach is decentralised, avoiding the scalability and single-point-of-failure limits the authors attribute to centralised, server-orchestrated methods such as slot-based, game-theoretic, and priority-based merging. It builds on an existing open-source MARL-CS controller and releases MARL-MASS openly, making it straightforward for automotive and simulation-tooling groups to reproduce or extend.

Future Directions

• Extending MARL-MASS to other congestion-prone scenarios, specifically double merging and traffic intersections.
• Applying the controller to mixed traffic scenarios that include human-driven vehicles alongside CAVs.
• Investigating whether the interaction topology construction and the custom reward transfer to other shield designs and MARL algorithms beyond MAPPO.
• Addressing the residual efficiency gap to unshielded controllers (MARL-CS leads MARL-MASS on merging percentage by 5.72%) without weakening the safety guarantee.

Target Audience

Researchers and graduate students working on safe multi-agent reinforcement learning, control barrier functions, and autonomous driving; CAV and intelligent transportation systems engineers interested in cooperative merging; and practitioners in simulation-based validation of autonomous vehicle controllers who want a decentralised alternative to centralised coordination.

Authors’ abstract

Lane changing in dense traffic is a significant challenge for Connected and Autonomous Vehicles (CAVs). Existing lane change controllers primarily either ensure safety or collaboratively improve traffic efficiency, but do not consider these conflicting objectives together. To address this, we propose the Multi-Agent Safety Shield (MASS), designed using Control Barrier Functions (CBFs) to enable safe and collaborative lane changes. The MASS enables collaboration by capturing multi-agent interactions among CAVs through interaction topologies constructed as a graph using a simple algorithm. Further, a state-of-the-art Multi-Agent Reinforcement Learning (MARL) lane change controller is extended by integrating MASS to ensure safety and defining a customised reward function to prioritise efficiency improvements. As a result, we propose a lane change controller, known as MARL-MASS, and evaluate it in a congested on-ramp merging simulation. The results demonstrate that MASS enables collaborative lane changes with safety guarantees by strictly respecting the safety constraints. Moreover, the proposed custom reward function improves the stability of MARL policies trained with a safety shield. Overall, by encouraging the exploration of a collaborative lane change policy while respecting safety constraints, MARL-MASS effectively balances the trade-off between ensuring safety and improving traffic efficiency in congested traffic. The code for MARL-MASS is available with an open-source licence at https://github.com/hkbharath/MARL-MASS