MindDrive: A Vision-Language-Action Model for Autonomous Driving via Online Reinforcement Learning

AI summary

Overview

Research area: End-to-end autonomous driving, specifically Vision-Language-Action (VLA) models trained with online reinforcement learning (RL).

Technical level: Advanced. The paper assumes familiarity with imitation learning, reinforcement learning algorithms (PPO, GAE), LoRA adapters, vision-language models, and closed-loop driving benchmarks.

Scope: The paper proposes MindDrive, a VLA framework that splits a single large language model into a Decision Expert and an Action Expert, and trains the Decision Expert with online RL in the CARLA simulator by converting continuous trajectory actions into a finite set of discrete linguistic driving decisions.

What This Paper Is About

Current VLA models for autonomous driving are mostly trained with imitation learning (IL), which fits expert driving data but suffers from distribution shift and causal confusion, leading to error accumulation in closed-loop driving. Online reinforcement learning could fix this through trial and error, but applying it directly to continuous trajectory action spaces is inefficient to explore. MindDrive's goal is to make online RL tractable for a VLA driving model by having the model reason and explore over a small set of discrete language-based driving decisions, while a second expert converts those decisions into concrete trajectories and feeds trajectory-level reward back into the reasoning space.

Key Contributions

1. MindDrive, an online RL framework for VLA autonomous driving. It introduces a dynamic language-to-action mapping that reduces the exploration space to discrete linguistic decisions, letting trajectory-level rewards drive optimization of the model's reasoning.
2. A two-expert architecture built from one LLM. A Decision Expert (performing scenario reasoning and meta-action selection) and an Action Expert (mapping meta-actions to trajectories) share the same base LLM and vision encoder and differ only in their LoRA parameters.
3. The first VLA-based autonomous driving model trained with online RL in a simulator. The online RL pipeline is built on CARLA with parallel collectors, cached scene tokens as states, and a PPO-based policy optimization with a KL regularization term.
4. State-of-the-art results on Bench2Drive. MindDrive-L (Qwen2.5-3B) reaches 80.59 Driving Score and 58.26% Success Rate, and MindDrive with a 0.5B LLM also surpasses larger imitation-learning and offline-RL baselines.

Main Findings

• MindDrive-L sets new state of the art on Bench2Drive. With a Qwen2.5-3B LLM, MindDrive-L achieves 80.59 DS and 58.26% SR, outperforming the offline RL method AutoVLA (Qwen2.5-3B) by 1.75 DS and 0.53% SR, and surpassing the IL model PGS by 2.51 DS and 9.62% SR.
• The lightweight 0.5B version is highly competitive. MindDrive (Qwen2-0.5B) reaches 78.04 DS and 55.09% SR, which is 5.15% higher in DS and 9.26% higher in SR than an IL baseline of the same parameter size. It exceeds the IL SOTA model PGS by 6.45% SR, the online RL method Raw2Drive by 6.68 DS and 4.85% SR, and the offline RL method RecogDrive by 6.68 DS and 9.64% SR.
• Online RL improves over the same model trained with IL alone. MindDrive gains 2.19 DS and 5.79% SR over its IL version, while MindDrive-L gains 4.81 DS and 7.55% SR over its IL version.
• Multi-Ability improvements concentrate in decision-related skills. MindDrive improves mean ability by 14.91% over RecogDrive and 5.57% over ORION with the same lightweight LLM, including gains of 55.56% in Overtaking and 30% in Give Way over RecogDrive. It trails the SOTA VLA methods on Emergency Brake and Traffic Sign, but still improves 8.33% and 0.98% respectively over its own IL version.
• Penalty events drive progressive gains. Starting from the IL baseline (ID-1: 75.85 DS, 49.30% SR, 49.44 mean ability), adding a collision penalty (ID-2) gives 1.4% SR and 3.76% mean ability improvement; adding the traffic-light penalty (ID-3) lifts Traffic Sign by 1.52% and Emergency Braking by 8.97%; adding route deviation (ID-4) reaches 76.95 DS and 51.85% SR; adding the stop-sign penalty (ID-5) yields 78.04 DS and 55.09% SR, with 5.26% Merging and 3.24% SR gains over ID-4.
• KL regularization outperforms other PPO regularization. PPO-KL achieves 78.04 DS and 55.09% SR, beating vanilla PPO by 3.31 DS and 8.36% SR and an entropy-based variant by 2.33 DS and 5.85% SR.
• Meta-actions beat navigation commands, and RL beats IL meta-actions. Navigation-command IL scores 68.11 DS and 41.59% SR; meta-action IL scores 75.85 DS and 49.30% SR (a 7.74 DS and 7.71% SR gain); meta-action with online RL reaches 78.04 DS and 55.09% SR (a further 2.19 DS and 5.79% SR gain).
• Two RL epochs are optimal. One epoch underperforms the baseline due to inaccurate value estimates, while two epochs give the reported 2.19 DS and 5.79% SR improvement. Further training degrades performance, with DS dropping from 78.04 to 73.69 and SR from 55.09% to 45.12%, attributed to catastrophic forgetting.
• Open-loop performance is not the strength of the method. MindDrive trails DriveMoE on open-loop metrics; the authors attribute this to differing optimization objectives, since MindDrive optimizes for closed-loop driving through RL rewards rather than directly minimizing trajectory-fitting error.

Methodology in Plain English

The researchers start from a single LLM and give it two different LoRA adapters, producing two roles that share the same base model and vision encoder.

• The Decision Expert looks at navigation instructions and multi-view images and outputs a high-level driving decision, called a meta-action. The decision space is deliberately small: 7 speed meta-actions and 6 path meta-actions.
• The Action Expert takes that meta-action plus the scene and instruction and produces concrete trajectory outputs. Longitudinal control is represented as speed waypoints (6 points covering a 3-second future at 2Hz), and lateral control as path waypoints (20 points covering a 20-meter path at 1-meter intervals), extracted through two special tokens and decoded by a VAE with a GRU-based decoder.

Training happens in two stages. First, imitation learning creates a one-to-one mapping between meta-actions and trajectories, using a cross-entropy loss on reasoning and planning QA pairs, a behavior cloning loss, a VAE loss, and a detection loss. This gives the model human-like candidate trajectories and shrinks the space RL has to explore.

Second, online RL runs in CARLA. The framework launches 24 parallel CARLA collectors over routes the IL model failed, caches scene tokens per frame as compact state representations, and samples meta-actions from the Decision Expert's output logits. A value network shares the LLM weights with only the final layer replaced by an MLP. The reward is sparse: +1 for reaching the destination, -1 for triggering a penalty event (collision, red light, route deviation, or failing to obey a stop sign), and 0 otherwise. After a route, the framework computes TD errors, Generalized Advantage Estimation, and returns, then optimizes the Decision Expert with PPO plus a KL divergence regularization term that constrains the meta-action distribution against a reference policy to prevent catastrophic forgetting.

Training used 32 NVIDIA A800 GPUs (80 GB). The vision encoder is EVA-02-L. The base LLMs are Qwen2-0.5B and Qwen2.5-3B. Chat-B2D supplies the reasoning data. LoRA rank and alpha are both set to 16.

Data split: the official Bench2Drive base set of 1000 clips, with 950 for training and 50 for open-loop validation; closed-loop evaluation uses 220 short routes across 44 interactive scenarios; online RL uses 44 rollout routes.

Why This Matters

The paper shows that online reinforcement learning can be made practical for a VLA driving model by moving exploration out of the continuous trajectory space and into a small discrete language-decision space, while still grounding the reward in actual trajectories. It also shows that a 0.5B-parameter model with this training scheme can outperform much larger LLMs (Vicuna-7B and Paligemma-3B) used by ORION and DriveMoE, which points to training method rather than raw model scale as a key lever.

Real-world applications:

• Urban autonomous driving stacks that need to handle interactive scenarios such as overtaking, merging, and giving way, where imitation learning tends to accumulate errors.
• Simulator-grounded policy refinement for production driving models, which could be continuously improved on targeted failure routes without new human demonstrations.
• Interpretable decision interfaces, since the model exposes its driving decision as a discrete linguistic meta-action before converting it to a trajectory.
• Compute-constrained deployment, where a lightweight 0.5B LLM achieves competitive closed-loop scores.

Industry relevance: the work comes from Huazhong University of Science and Technology and Xiaomi EV, and targets the exact problem automakers face when scaling end-to-end driving models, namely that imitation learning alone does not close the loop.

Future Directions

• Evaluation beyond CARLA. The authors state that their evaluation is limited to CARLA because no real-world interactive simulator for autonomous driving exists.
• Scaling the online RL rollout process. The authors note that the challenge of synchronizing multiple CARLA simulators prevented them from evaluating multiple candidate actions (the sentence is truncated in the provided text).
• Combining closed-loop RL objectives with open-loop trajectory quality. MindDrive trails DriveMoE on open-loop metrics because it optimizes closed-loop reward; negotiating both objectives is an open question.
• Controlling catastrophic forgetting more effectively. Performance degrades sharply beyond two RL epochs (DS 78.04 to 73.69, SR 55.09% to 45.12%), so more stable long-horizon online training remains unresolved.

Target Audience

Researchers and engineers working on end-to-end autonomous driving, Vision-Language-Action models, and reinforcement learning for embodied agents, especially those interested in closing the gap between imitation learning and interactive policy optimization. It is also relevant to practitioners at automotive and robotics companies who want to refine driving policies in simulation without collecting new expert demonstrations, and to readers tracking how lightweight LLMs perform relative to much larger ones on closed-loop driving benchmarks such as Bench2Drive.

Authors’ abstract

Current Vision-Language-Action (VLA) paradigms in autonomous driving primarily rely on Imitation Learning (IL), which introduces inherent challenges such as distribution shift and causal confusion. Online Reinforcement Learning offers a promising pathway to address these issues through trial-and-error learning. However, applying online reinforcement learning to VLA models in autonomous driving is hindered by inefficient exploration in continuous action spaces. To overcome this limitation, we propose MindDrive, a VLA framework comprising a large language model (LLM) with two distinct sets of LoRA parameters. The one LLM serves as a Decision Expert for scenario reasoning and driving decision-making, while the other acts as an Action Expert that dynamically maps linguistic decisions into feasible trajectories. By feeding trajectory-level rewards back into the reasoning space, MindDrive enables trial-and-error learning over a finite set of discrete linguistic driving decisions, instead of operating directly in a continuous action space. This approach effectively balances optimal decision-making in complex scenarios, human-like driving behavior, and efficient exploration in online reinforcement learning. Using the lightweight Qwen-0.5B LLM, MindDrive achieves Driving Score (DS) of 78.04 and Success Rate (SR) of 55.09% on the challenging Bench2Drive benchmark. To the best of our knowledge, this is the first work to demonstrate the effectiveness of online reinforcement learning for the VLA model in autonomous driving.