Zero-Shot MARL Benchmark in the Cyber-Physical Mobility Lab

AI summary

Overview

• Research area: Robotics — specifically multi-agent reinforcement learning (MARL) for motion planning of Connected and Automated Vehicles (CAVs), with a focus on sim-to-real transfer.
• Technical level: Advanced. The paper assumes familiarity with MARL, MAPPO, kinematic bicycle models, model predictive control, and hardware testbeds.
• Scope: The paper presents a reproducible three-tier benchmark (simulation, high-fidelity digital twin, and physical testbed) for zero-shot evaluation of MARL motion-planning policies, demonstrated by deploying one SigmaRL-trained policy across all three domains.

What This Paper Is About

MARL policies for vehicle motion planning usually look strong in simulation but degrade when moved onto real hardware, and most published studies never test that transfer at all. The authors build a reproducible platform based on the Cyber-Physical Mobility Lab (CPM Lab) so that a single trained policy can be run without fine-tuning in simulation, in a digital twin, and on physical 1:18-scale vehicles. The goal is to measure and attribute the resulting performance losses rather than to propose a new algorithm.

Key Contributions

1. A reproducible platform for studying sim-to-real transfer in MARL-based motion planning, integrating a simulation environment, a high-fidelity digital twin, and a physical testbed.
2. A baseline evaluation in which a SigmaRL-trained policy is deployed zero-shot across simulation, digital twin, and physical CPM Lab, using the same initial positions across environments for paired comparison.
3. Released tools, configurations, and metrics for replicating the experiments, with a systematic discussion of sim-to-real effects arising from domain shift and increasing system realism. Code for reproducing the SigmaRL simulation results is available at https://github.com/bassamlab/SigmaRL, and a CPM Lab testing guide is to be published upon acceptance at https://bassamlab.github.io/cpm-lab-ros2/.
4. Two concrete interface transformations that make the RL policy usable in the lab: a mapping from the CPM Lab state vector into SigmaRL's state format, and a conversion of policy outputs into reference trajectories for the lab's trajectory-following mode.

Main Findings

• Collision rate rises monotonically with realism. Agent–agent collision rate (CRA-A) increased from 0.37 ± 1.33 events per 100 m in the SigmaRL simulation to 2.10 ± 2.80 in the digital twin and 4.49 ± 6.26 in the physical lab, with IQM values of 0.47, 1.47, and 2.45 respectively.
• A measurable sim-to-real gap exists between the digital twin and the physical lab. Average speed dropped from 0.771 ± 0.031 m/s in the digital twin to 0.728 ± 0.035 m/s in the physical lab, alongside slightly higher CRA-L and centerline deviation. The authors attribute this to sensor noise, actuation delays, and communication latency.
• An unexpected "architectural degradation" favors the more realistic environments on secondary metrics. The SigmaRL simulation had the lowest CRA-A but the worst lane violations (CRA-L 45.99 ± 6.69), centerline deviation (0.050 ± 0.009 m), and average speed (0.759 ± 0.024 m/s). The digital twin, with a mid-level MPC trajectory follower, achieved CRA-L 38.80 ± 9.57, CD 0.041 ± 0.010 m, and AS 0.771 ± 0.031 m/s; the physical lab gave CRA-L 41.09 ± 7.21, CD 0.044 ± 0.010 m, AS 0.728 ± 0.035 m/s. The paper credits this to the MPC smoothing high-frequency control inputs and improving tracking, not to better learned policies.
• Collision rate is the most robust indicator. Across environments, CRA-A increased consistently, while CRA-L, CD, and AS proved more sensitive to control architecture, evaluation horizon, and trajectory smoothing.
• Policies tolerated domain shift. Training occurred only on the intersection segment, while evaluation covered the whole map including highway loops, on-ramps, and intersections; the authors state the policies generalised to unseen map regions and that agents recovered from collisions and disturbances.
• Agent count change was not treated as a domain-shift factor. Training used 4 agents and evaluation used 3; the authors argue this does not significantly contribute to the shift because the policies are decentralized with shared parameters and each agent observes only a fixed number of nearby agents.

Methodology in Plain English

The policy is trained with the authors' prior SigmaRL framework, which uses Multi-Agent Proximal Policy Optimization (MAPPO) with decentralized actors and a centralized critic, and models each agent with a nonlinear kinematic bicycle model. The observation design is the key element: instead of raw pixels, the policy receives structured, geometrically meaningful features such as distances to road boundaries and lane centerlines, which the authors argue are task-relevant invariances that aid generalisation.

Training used 4 agents on the intersection segment only, with randomly initialised states and reference paths, resetting an agent whenever it collided or left the intersection. The actor and critic were feedforward networks with three hidden layers of size 256 and tanh activations, fed by a 32-dimensional observation vector, giving roughly 140k trainable parameters and a model smaller than 0.5 MB. Training ran for 1000 iterations, collecting a batch of 2^12 samples per iteration (about four million samples total), using minibatches of size 2^9 and 30 epochs per iteration, retaining the best-performing policy overall; the trained actor was then transferred to physical vehicles with no fine-tuning.

For deployment in the CPM Lab, two conversions were needed. First, the lab's state list — position, yaw angle, scalar speed, and a normalised steering angle in [-1, 1] — is mapped to SigmaRL's format by denormalising the steering angle with the µCar's maximum steering angle and converting scalar speed to a 2D velocity vector using the kinematic bicycle model and the slip angle. Second, the policy's speed and steering outputs are turned into a reference trajectory: within a control horizon, policy actions are applied and used with the motion model to predict future states; after the control horizon, velocity is held constant and steering is gradually reduced to zero over the remaining prediction horizon. A mid-level MPC trajectory follower then tracks that reference.

Evaluation covered three environments with different realism: the SigmaRL simulation (kinematic bicycle model, instantaneous noise-free actuation, actions applied directly to the vehicle model); the digital twin (a grey-box vehicle model identified from real-world logs, sensor and actuation imperfections, MPC trajectory follower); and the physical lab (scaled vehicles with motor delays, sensor noise, and overhead-camera localisation with known latency, same MPC follower). Three trained policies were each evaluated in all environments across three initial position configurations with three repetitions — 3 × 3 × 3 = 27 runs per environment — with matching initial positions between simulation/digital twin and the corresponding physical runs. Each run lasted 18 s, corresponding to 180 steps at a 0.1 s timestep, using 3 agents. The authors restricted evaluation to three agents because preliminary physical experiments with higher counts (e.g., five agents) produced frequent collisions and required repeated manual intervention; in the physical runs, agents were manually reset after collisions.

Four metrics were reported, normalised to total distance travelled per 100 m of aggregate agent travel: Agent–Agent Collision Rate (CRA-A, detected via minimum translation vector between agent geometries with a hysteresis filter requiring overlap for at least 3 consecutive steps to start and 5 consecutive non-colliding steps to end), Agent–Lane Collision Rate (CRA-L, polygon intersection between agent footprint and lane boundaries with 10% slack relative to vehicle width), Centerline Deviation (CD, mean lateral distance to the reference path), and Average Speed (AS, mean forward speed as the L2 norm of the velocity vector). Results are reported as mean ± std with interquartile mean (IQM) in parentheses.

Why This Matters

The paper argues that most MARL sim-to-real studies rely on simulation-only evaluation and lack real-world deployment and reproducibility. By providing a shared platform and baseline metrics, it lets different policies be compared under identical conditions rather than under bespoke setups.

Real-world applications:

• Intersection and highway merging coordination for connected and automated vehicles, matching the eight-lane intersection, highway loop, and on- and off-ramps of the CPM Lab map.
• Validating decentralized motion planners before full-scale road testing, using the small-scale lab as a safe, repeatable intermediate step.
• Benchmarking domain-randomization and system-identification techniques against a fixed baseline instead of case-by-case simulated results.
• Testing multi-vehicle software stacks, including localization, synchronization, and trajectory-following control layers, under realistic latency and noise.

Industry relevance: the platform targets CAV motion planning, where deployment risk and cost make full-scale fleet testing impractical. The demonstrated interface work — converting RL policy outputs into reference trajectories trackable by an MPC follower — reflects how learned planners must integrate with existing production-style hierarchical control stacks. The paper also notes that small-scale testbeds are cost-efficient and that, from their prior analysis, such testbeds excel at evaluating software components for motion planning and communication but fall short in representing interactions with the surrounding environment.

Future Directions

• Relaxing the global state assumption: the authors plan to investigate partial observability, onboard perception, and communication-limited coordination.
• Disentangling domain shift from environment shift: the current work deliberately does not experimentally separate these effects, holding the training domain fixed and leaving a detailed ablation of domain shift for future work.
• Integrating the mid-level MPC into training: the paper states that folding the CPM Lab's mid-level model predictive control into the SigmaRL training simulation would require a substantial redesign of the training environment, which is left unaddressed.
• Scaling to more agents: evaluation was limited to three agents because five-agent physical trials caused frequent collisions and manual intervention, raising the open question of how to make higher-density multi-vehicle trials stable and reproducible.

Target Audience

Researchers and practitioners working on multi-agent reinforcement learning, sim-to-real transfer, and connected and automated vehicle motion planning. It is also relevant to engineers building small-scale robotic testbeds and to readers interested in benchmarks and reproducibility practices, though it requires a working understanding of MARL training pipelines and vehicle control stacks to follow the technical details.

Authors’ abstract

We present a reproducible benchmark for evaluating sim-to-real transfer of Multi-Agent Reinforcement Learning (MARL) policies for Connected and Automated Vehicles (CAVs). The platform, based on the Cyber-Physical Mobility Lab (CPM Lab) [1], integrates simulation, a high-fidelity digital twin, and a physical testbed, enabling structured zero-shot evaluation of MARL motion-planning policies. We demonstrate its use by deploying a SigmaRL-trained policy [2] across all three domains, revealing two complementary sources of performance degradation: architectural differences between simulation and hardware control stacks, and the sim-to-real gap induced by increasing environmental realism. The open-source setup enables systematic analysis of sim-to-real challenges in MARL under realistic, reproducible conditions.