CoIRL-AD: Collaborative-Competitive Imitation-Reinforcement Learning in Latent World Models for Autonomous Driving

AI summary

Overview

Research area: End-to-end autonomous driving, specifically the offline integration of imitation learning (IL) and reinforcement learning (RL) inside a learned latent world model.

Technical level: Advanced. The paper assumes familiarity with offline reinforcement learning, policy-gradient methods, world models, and the standard nuScenes planning evaluation protocol.

Scope: The paper proposes CoIRL-AD, a dual-actor framework in which an imitation actor and a reinforcement learning actor share a latent world model and periodically compete to transfer knowledge, evaluated on nuScenes for planning accuracy, collision rate, cross-city generalization, and long-tail scenarios.

What This Paper Is About

End-to-end driving models are usually trained by imitating expert demonstrations, which leaves them fragile in long-tail situations where demonstrations are sparse and small errors compound into unseen states. Reinforcement learning could supply complementary task-level supervision, but real-world driving datasets are offline, dominated by near-optimal expert actions, and lack an interactive simulator. The paper asks how RL can be applied in this setting without an external simulator, and answers with a framework that decouples IL and RL into separate actors while keeping them interacting through competition.

Key Contributions

1. A latent-world-model RL procedure for offline driving. Instead of relying on an external simulator such as CARLA, the method trains a latent world model from offline real-world data and uses it as a reactive simulator to imagine future rollouts, enabling long-horizon reward estimation over sampled action sequences.

2. Group sampling with a dreaming critic. Inspired by GRPO, the RL actor samples multiple action candidates, scores them with rule-based rewards built from annotated maps and bounding boxes, and is optimized with a critic that values the world-model-predicted next state (actor + dreaming critic with group sampling, "ADCGS"). A step-aware sampling variant keeps only one action stochastic per sequence to preserve trajectory smoothness.

3. A competitive dual-policy architecture. IL and RL are assigned to separate actors to avoid gradient conflicts from jointly optimizing behavior cloning and reward maximization. Every k iterations the actors are compared using the difference in accumulated reward (Δr_acc) against two thresholds (λ_min, λ_max); the loser is left unchanged, soft-merged with interpolation parameter p, or fully replaced depending on the size of the gap.

4. Backward planning (inverse causality). The waypoint self-attention is re-conditioned so that the i-th action depends on current and future waypoint features (s_w,j≥i) rather than past ones, motivated by the observation that drivers decide where to go before committing to low-level actions and that only the first action is executed before replanning. Experiments show this benefits the RL actor in particular.

Main Findings

• nuScenes planning accuracy and safety improve over the LAW baseline. Without temporal augmentation, CoIRL-AD reaches an average L2 of 0.63 m and average collision rate of 0.18% versus LAW's 0.66 m and 0.22%, with an L2·Col score of 0.11 versus 0.15. With temporal augmentation, CoIRL-AD reaches 0.45 m average L2, 0.17% average collision rate, and an L2·Col of 0.08, the best combined score among all methods compared.

• The lowest collision rate is achieved even without temporal augmentation. The paper notes that CoIRL-AD without temporal augmentation already attains the lowest collision rate in Table 1, while temporal augmentation mainly further reduces L2 error.

• Cross-city generalization improves substantially. Training on nuScenes-Singapore and testing on nuScenes-Boston, LAW degrades to 0.93 m average L2, 0.69% average collision rate, and 0.64 L2·Col, while CoIRL-AD achieves 0.70 m, 0.22%, and 0.15 L2·Col, a 77% reduction on the combined metric.

• Long-tail gains track demonstration coverage. The authors construct two evaluation subsets based on the baseline's performance, one with high L2 error and one with high collision rate, and report that RL produces large improvements over the baseline on both (quantitative details referenced to Appendix E.3, not included in the provided content). The paper states RL helps most where demonstrations are sparse or missing and less in well-covered scenarios, so average nuScenes metrics dilute the effect.

• The latent world model matters. Removing it ("w/o wm") gives 0.65 m average L2, 0.20% average collision rate, and 0.13 L2·Col, consistently worse than full CoIRL-AD, with the largest gap in 3-second collision rate (0.51% versus 0.37%).

• Only decoupled actors with competition beat the baseline on both metrics. Pure RL averages 6.55 m L2 and 4.93% collision (L2·Col 32.29); two-stage pre-training then fine-tuning averages 4.22 m and 4.32% (18.23); loss merging (0.76 m, 0.23%, 0.17) and IL–RL interval (0.68 m, 0.28%, 0.19) also fail to improve both. Decoupling without competition gives 0.72 m, 0.29%, 0.21, while decoupling with competition gives 0.63 m, 0.18%, 0.11.

• Inverse causal masking helps RL but hurts pure IL. Applying inverse AR to the pure IL baseline worsens it to 0.73 m / 0.25% (L2·Col 0.18) versus 0.66 m / 0.22% (0.15). Inside CoIRL-AD it gives the best result (0.63 m / 0.18% / 0.11). Mismatched masks between actors (IL AR + RL inv. AR) fall back to baseline-level performance (0.66 m / 0.23% / 0.15), and removing competition causes degradation (0.72 m / 0.29% / 0.21).

• The actors swap dominance during training. Before 20k iterations the IL actor wins more often and scores higher; afterwards, as the latent world model encodes driving dynamics, the RL actor dominates. IL gradients continue to benefit shared components such as the perception module even when IL loses.

• Competition suppresses value overestimation. Critic value estimates for world-model-predicted future states grow rapidly without the competitive mechanism and plateau with it, and the L2 distance between the IL and RL actors' modal trajectories grows substantially when RL is unconstrained, indicating the mechanism keeps exploration in a safer region closer to the expert policy.

• No additional inference cost. CoIRL-AD shares the baseline's inference architecture, so it adds no inference latency (training cost and latency analyses are referenced to Appendices E.1 and E.2, not included in the provided content).

• Qualitative cases favor the proposed model. Visualizations show CoIRL-AD avoiding a fast-approaching vehicle in an adjacent lane and yielding to crossing pedestrians, where the baseline LAW fails to react, attributed to imitating trajectories without understanding underlying intent.

Methodology in Plain English

The system starts with a perception module that turns camera observations into latent states, and a waypoint query that cross-attends into those states to produce a short future trajectory of x/y actions. The imitation learning objective supervises these actions with an L1 loss against expert trajectories and, following LAW, adds a mean-squared-error world-model loss weighted by α = 0.2, where the world model predicts the latent state 1.5 seconds into the future.

Planning is reorganized around inverse causality: each action is conditioned on the current and future waypoints rather than earlier ones, implemented with an inverse causal attention mask. A separate stochastic head outputs a standard deviation per action, turning the deterministic planner into a Gaussian policy so RL can optimize it.

For RL, the model samples a group of candidate action sequences and converts each into positions by cumulative summation. Rewards combine an imitation term (an exponential of the negative action distance to the expert) and a collision term (one minus a collision check against static map information and dynamic agents, using non-reactive simulation). The two are multiplied per step. A critic values the current state and the world-model-imagined next state for each sample, and the resulting long-horizon advantage is Z-score normalized within the group and used to update the actor and critic jointly. A small behavior-cloning term with coefficient β regularizes the RL actor, and a reference critic maintains an exponential moving average of the learning critic for stability. Step-aware sampling sets all but one action per sequence to its policy mode to avoid unrealistic trajectories.

Finally, the IL actor and RL actor are optimized separately. Periodically the framework compares their accumulated reward scores: close scores leave both untouched, a moderate gap triggers a weighted parameter interpolation between winner and loser, and a large gap replaces the loser outright.

Why This Matters

The work argues that RL need not wait for a photorealistic interactive simulator to help driving policies, and that the obstacle to offline RL on expert-dominated data is not the absence of rewards but instability from world-model bias. Methodologically, it isolates a specific failure mode, value overestimation under an expert-trained world model, and shows that competition between an IL anchor and an RL explorer contains it. This gives a concrete alternative to two-stage IL-then-RL pipelines, which the paper shows fail badly here (4.22 m average L2) despite succeeding in other domains.

Real-world applications:

• Long-tail safety behavior in deployed driving stacks, such as yielding to crossing pedestrians and handling fast-approaching vehicles, which the qualitative results specifically demonstrate.
• Cross-city or cross-region deployment, where a policy trained in one city must operate in another; the paper reports a 77% reduction on the combined L2·Col metric in its Singapore-to-Boston test.
• Fleets with large logged driving datasets but no simulator budget, since the method trains entirely from offline real-world data and adds no inference latency.
• Broader embodied AI systems that face the same expert-dominated, offline data regime, as the authors state in the impact statement.

Industry relevance: the approach targets the practical constraint that most driving companies hold large scale-logs of human driving but lack a validated interactive simulator, and it preserves the deployed model's inference architecture, meaning improvements need not change production latency characteristics.

Future Directions

• Richer reward signals. The paper notes that on offline datasets such as Navsim, rewards could incorporate comfort-related metrics and time-to-collision, suggesting the current two-term reward leaves room for refinement.
• Closed-loop or online validation. All results are offline, and the collision term relies on non-reactive simulation; whether the competitive mechanism holds under reactive agents or online RL is untested.
• Reducing world-model bias directly. Competition mitigates overestimation rather than eliminating its source, leaving open whether better world models or conservatism could allow a less constrained RL actor to do better.
• Scaling and hyperparameter sensitivity. The competition thresholds (λ_min, λ_max), interval k, interpolation p, group size G, and the behavior-cloning coefficient β are referenced to appendices not included in the provided content, so how sensitive the gains are to these settings remains an open question.

Target Audience

Researchers and engineers working on end-to-end autonomous driving, offline reinforcement learning, and world models, particularly those integrating imitation learning with reward-driven optimization. It is also relevant to practitioners who need to improve long-tail and cross-city robustness on logged driving data without access to an interactive simulator, and to graduate students studying policy-gradient methods with group sampling applied outside language models.

Authors’ abstract

End-to-end autonomous driving models trained with imitation learning (IL) often generalize poorly, particularly in long-tail scenarios where expert demonstrations are sparse. Reinforcement learning (RL) can provide complementary task-level supervision, but applying RL to real-world autonomous driving is challenging in offline settings without interactive simulators, where datasets are dominated by expert actions and provide limited behavioral diversity. We propose CoIRL-AD, a competitive dual-policy framework that integrates IL and RL under a unified offline training regime. CoIRL-AD decouples imitation and reward optimization into separate actors to alleviate objective conflicts, uses imagined future rollouts for long-horizon reward estimation, and introduces a competition mechanism that selectively transfers beneficial behaviors while keeping RL anchored to expert-like driving. Experiments on the nuScenes benchmark show that CoIRL-AD consistently improves robustness over strong IL-based baselines, with especially large gains in cross-city generalization and long-tail scenarios. Code is available at: https://github.com/SEU-zxj/CoIRL-AD.