Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving

AI summary

Overview

• Research area: Autonomous driving, vision-language models (VLMs), multimodal perception, motion planning.
• Technical level: Advanced. The paper assumes familiarity with bird's-eye-view (BEV) perception, camera-based 3D detection, occupancy prediction, flow matching, and policy-gradient reinforcement learning.
• Scope in one sentence: The paper describes a single pretrained vision-language model that is extended with an external BEV perception head and a flow-matching Planning Expert so that one shared model performs 3D perception, driving visual question answering, and ego-trajectory planning without changing the underlying VLM architecture.

What This Paper Is About

Most driving vision-language-action models are built by continuing to train a general-purpose VLM on driving data, mostly in the form of text-based visual question answering. The authors argue this has two problems: text targets never directly constrain 3D layout, depth, or occupancy, so a model can write fluent scene descriptions while remaining geometrically imprecise; and heavy driving-specific adaptation can cause catastrophic forgetting of the general knowledge that is needed for rare and out-of-distribution situations.

The goal of Qwen-Drive-1.0 is therefore a single model that keeps the pretrained VLM architecture intact, adds an explicit and inspectable 3D perception interface, and learns driving competence while retaining general-purpose vision-language ability such as multi-turn dialogue, instruction following, and open-ended visual understanding.

Key Contributions

1. A unified driving foundation model. Qwen-Drive-1.0 is presented as the first vision-language foundation model for autonomous driving, to the authors' knowledge, that integrates 3D perception, driving VQA, and motion planning without changing the pretrained VLM architecture.
2. An external BEV perception head that jointly learns 3D object detection, semantic occupancy prediction, and BEV map segmentation. It acts as a probe of the 3D information accessible in the shared representations and exposes explicit, inspectable perception outputs.
3. A staged training and data recipe that unifies cross-dataset labels, rewrites and filters driving VQA responses for format and factual consistency, and mixes driving supervision with general-purpose vision-language data to mitigate catastrophic forgetting.
4. A Planning Expert tailored to pretrained VLM representations, using flow matching to generate 50-waypoint future ego trajectories, with unified trajectory annotations enabling joint training across multiple public driving datasets and support for open-loop, pseudo-closed-loop, and closed-loop evaluation.

Main Findings

• 3D perception results: The model reaches 43.95 mAP and 60.99 map mIoU on nuScenes, and 43.45 mAP and 71.27 map mIoU on OpenScene, which the authors describe as highly competitive with common vision-based 3D detectors.
• Driving scene understanding: It "significantly surpasses" the general-purpose Qwen3.5-4B on driving scene understanding while largely preserving general vision-language capability. Specific per-benchmark VQA scores are not reported in the available content.
• Open-loop planning: It achieves a Predictive Driver Model Score (PDMS) of 90.7 on NAVSIM.
• Pseudo-closed-loop planning: It attains a Rater Feedback Score of 7.91 on the test split of the Waymo Open Dataset end-to-end benchmark (WOD-E2E).
• Closed-loop planning: The paper reports "promising potential for closed-loop driving in AlpaSim" without a numeric score in the available content.
• Head-only pretraining is insufficient: Training only the BEV perception head with frozen vision encoder and VLM yields limited perception performance, indicating that the pretrained representations do not directly expose enough 3D structure; joint training of the head, vision encoder, and VLM is required.
• Architecture choice preserves usability: View and frame tags are ordinary vocabulary tokens, requiring no additional special tokens or architectural changes to the base VLM.
• Reward optimization is practical: Because the deterministic Euler sampler has no differentiable transition probabilities, stochasticity is injected only over the final three of ten integration steps, restricted to a six-mode low-frequency cosine subspace so that perturbations produce coherent trajectory shifts rather than pointwise jitter (average per-waypoint standard deviation σ√(M/N) ≈ 0.010, roughly 1.7 m longitudinally and 0.26 m laterally per stochastic step).

Methodology in Plain English

Base model. The system builds on the natively multimodal Qwen3.5-4B VLM and attaches two external modules: a BEV perception head and a Planning Expert.

Handling many cameras and frames. Images from eight canonical view directions are labeled with view tags (<FRONT VIEW>, <FRONT RIGHT VIEW>, and so on) and frame tags (frame: k). Question answering examples are serialized frame-major (all views at one timestep, then the next), while planning examples are serialized view-major (all timesteps of one view, then the next view), which exposes temporal variation within each camera. Single-view or single-frame inputs simply omit redundant tags.

3D perception probe. The perception head reads two feature streams per view: low-level features from the vision encoder and contextual features from after the full VLM. A lightweight depth network predicts a per-pixel categorical depth distribution without depth supervision, which lifts encoder features into a 3D volume. A query-based BEV transformer then aggregates multiscale VLM features onto the BEV plane, with queries initialized from the height-collapsed volume as a geometric prior. Three branches share the resulting BEV feature: a DETR-style deformable-attention decoder for 3D detection, a 3D UNet for semantic occupancy, and a UNet-style head for map segmentation. Losses are focal plus L1 for detection (weighted 2 and 0.75 per decoder layer), 100 × focal + geo + sem + Lovász for occupancy, and 100 × focal + Lovász for maps.

Planning. A 32-layer diffusion transformer with hidden dimension 1024 (roughly 1.1B parameters) generates 50 waypoints spanning 5 s at 10 Hz. It conditions on cached keys and values from the VLM's grouped-query softmax attention layers, with each cache conditioning four Planning Expert layers. Trajectories are represented as longitudinal position, lateral position, and heading, normalized by fixed scales of 165 m, 25 m, and π/2 rad. Training uses flow matching with an x-prediction parameterization (predicting the clean trajectory rather than velocity or noise), a Beta(1.5, 1.0) flow-time sampler capped at t = 0.9, and two Huber temporal-difference regularizers that penalize jitter and abrupt acceleration. Inference uses a 10-step Euler solver.

Four-stage training. Stage 1 trains only the new BEV head with the perception loss while the vision encoder and VLM stay frozen. Stage 2 jointly optimizes the head, vision encoder, and VLM on mixed perception and vision-language minibatches, giving the head a learning rate 20 times that of the VLM. Stage 3 freezes the vision-language pathway and trains only the Planning Expert by flow matching (the resulting model is called Qwen-Drive-1.0-SFT). Stage 4 applies reinforcement learning with task-level rewards (Qwen-Drive-1.0-RL), using group-relative advantages over 8 rollouts per scene, a discount of 0.6 across the three stochastic steps, and a likelihood surrogate evaluated in the low-frequency subspace.

Data unification. Perception data come from nuScenes (6 camera views, 28K training keyframes, 6K validation) and OpenScene (8 views, 607K training frames, 9K validation). Label taxonomies are aligned to a shared seven-class detection space, a ten-class occupancy space, and a six-class map schema; missing categories such as driveable for OpenScene and generic_object for nuScenes are completed offline from maps and 3D boxes. Because the two datasets use different occupancy grids (200×200×16 with ±40 m and z ∈ [−1.0, 5.4] m at 0.4 m voxels for nuScenes-OccNet versus ±50 m and z ∈ [−4.0, 4.0] m at 0.5 m voxels for OpenScene) and different LiDAR-to-ego transforms, native grids are preserved and alignment is done through a differentiable trilinear sampling of predicted features. Vision-language data combine 24 open-source driving datasets with self-constructed examples, rewritten by Qwen3.5-Plus into a common conversational schema and filtered for consistency. Planning training uses 15K NAVSIM scenes balanced across navigation commands, 15K PhysicalAI-AV scenes, and 479 WOD-E2E scenarios with rater-preference annotations.

Why This Matters

The work matters because it reframes driving adaptation as something that should not sacrifice general capability. The authors' deployment argument is that production vehicles are moving toward cockpit-driving integration, where the intelligent cockpit and the driving system share one compute platform instead of two domain controllers. A single model that handles both domains lowers hardware and integration cost; a model that trades general ability for driving performance would need a second model and extra compute. Explicit 3D perception outputs also make perception errors inspectable rather than hidden inside fluent text.

Real-world applications:

• Integrated cockpit-driving platforms, where one model serves dialogue, instruction following, and open-ended visual understanding alongside driving perception and planning.
• Camera-based 3D perception stacks, providing detection, occupancy, and BEV map segmentation from the same shared representations used for language tasks.
• Trajectory planning with optional reasoning explanations, allowing a plan to be conditioned on a textual driving rationale where such annotations exist.
• Robustness in rare and out-of-distribution scenarios, where retained pretrained visual and world knowledge is expected to support reasoning that finite driving datasets cannot cover.

Industry relevance: The released model artifacts (Qwen-Drive-1.0-4B on Hugging Face and ModelScope, plus a GitHub repository) put a compact 4B-parameter foundation model within reach of teams building unified cockpit-driving systems, and the reported NAVSIM and WOD-E2E scores provide reference points against which production-oriented planning systems can be compared.

Future Directions

• Closing the loop quantitatively. The paper reports only "promising potential" for AlpaSim closed-loop driving, so full closed-loop metrics, safety disengagements, and long-horizon behavior remain to be characterized.
• Single-frame perception limits. The BEV perception head performs single-frame surround-view 3D perception, leaving open how temporal perception and tracking could be added while keeping the shared representation intact.
• Narrowing the reward gap. The authors note that imitation of a single recorded future can penalize other acceptable safe behaviors and does not directly encode collision avoidance, drivable-area compliance, progress, or human preference; how best to combine multi-benchmark rewards into one comparable signal is an open problem.
• Scaling and data coverage. Since the planning mixture uses 15K NAVSIM scenes, 15K PhysicalAI-AV scenes, and 479 WOD-E2E scenarios, and the VQA mixture relies on 24 public datasets plus self-constructed data, the effect of scaling data, modalities, and model size on the perception–language–planning trade-off is not established.

Target Audience

This paper is most useful for autonomous driving researchers and engineers working on end-to-end or vision-language-action driving systems, for multimodal foundation model researchers interested in attaching geometric prediction heads and action experts to a frozen VLM backbone, and for industry teams evaluating a single shared model for cockpit-plus-driving deployment. Readers will benefit most if they are already comfortable with BEV perception, camera-based 3D detection, occupancy and map segmentation, trajectory prediction, flow matching, and reinforcement learning from relative rewards.

Authors’ abstract

We present Qwen-Drive-1.0, an initial step towards a vision-language foundation model for autonomous driving. Qwen-Drive-1.0 retains the architecture of the pretrained vision-language model (VLM) and integrates 3D perception, visual question answering, and motion planning within a unified framework. An external bird's-eye-view (BEV) perception head jointly performs 3D object detection, semantic occupancy prediction, and BEV map segmentation. It serves as a probe of the 3D information accessible from the shared representations and provides an explicit, inspectable interface to 3D scene structure. A Planning Expert conditions on shared VLM representations to generate future ego trajectories. A staged training recipe combines driving supervision with general-purpose vision-language data to acquire driving-specific competence while helping preserve broad visual understanding and instruction-following capabilities. Experiments demonstrate strong 3D perception and driving scene understanding while largely preserving general vision-language capability. Comprehensive evaluations across open-loop, pseudo-closed-loop, and closed-loop settings further show highly competitive motion-planning performance.