Latent Chain-of-Thought World Modeling for End-to-End Driving

AI summary

Overview

Research area: End-to-end autonomous driving, specifically Vision-Language-Action (VLA) models and inference-time reasoning.

Technical level: Advanced. The paper assumes familiarity with autoregressive sequence models, world models, trajectory tokenization, and reinforcement learning (specifically GRPO).

Scope: The paper introduces LCDrive, an end-to-end driving model that replaces text-based chain-of-thought reasoning with an action-aligned latent reasoning process grounded in a learned latent world model.

What This Paper Is About

Most recent driving VLA models perform inference-time reasoning by generating natural-language chain-of-thought before outputting a trajectory. The authors argue that text is a poor substrate for driving because language struggles to represent spatiotemporal geometry and multi-agent interactions, autoregressive text generation adds latency, and the final action can diverge from the stated rationale.

The paper's goal is to move reasoning into a compact latent space where the reasoning tokens use the same vocabulary as the driving actions themselves, so that reasoning and decision-making are unified rather than merely adjacent.

Key Contributions

1. The authors rethink the representation of reasoning in VLA models for end-to-end driving with LCDrive, which conducts latent chain-of-thought using latent reasoning tokens strongly aligned with driving actions and a latent world model.
2. They introduce a training framework that combines latent chain-of-thought cold-start, world model training, and closed-loop reinforcement learning, which they find especially effective for latent reasoning models.
3. They demonstrate consistent empirical gains on a large, diverse end-to-end driving benchmark: LCDrive delivers faster inference, improved driving quality, and larger improvements under interactive reinforcement learning compared to non-reasoning and text-reasoning baselines.

Main Findings

• Latent reasoning beats text reasoning on driving quality: On the PhysicalAI-AV benchmark, LCDrive (Latent CoT with predicted LWM plus RL) reaches 1.626 ADE versus 1.650 for the Text CoT baseline, with lower risk (OffRoad@2.5 of 1.219 vs. 1.391 and Coll@5.0 of 0.836 vs. 0.905). The authors note the Text CoT baseline was pretrained on a proprietary corpus over 100 times larger than their training set, then fine-tuned on roughly 10 times more Text-CoT-paired data.

• Latent reasoning beats no reasoning at all: LCDrive achieves 1.626 ADE versus 1.762 for the non-reasoning baseline, OffRoad@2.5 of 1.219 versus 1.753, and Coll@5.0 of 0.836 versus 2.207.

• Reinforcement learning only helps when the model reasons: Adding RL to the LWM0-only model produced no ADE gain and worsened OffRoad@5.0, whereas RL on Latent CoT consistently improved both accuracy and safety. The authors interpret this as RL "activating" a useful latent chain-of-thought process.

• Oracle world states show the ceiling of the approach: With ground-truth LWM tokens, Latent CoT improves ADE from 1.393 (LWM0-only) to 1.268, and RL further reduces it to 1.197 while reducing Coll@5.0 from 0.905 to 0.867.

• Predicted world models are robust enough to use: Despite prediction errors, interleaved Latent CoT yields consistent gains over the non-reasoning policy, and RL on top of predicted LWM adds further gains in both accuracy and safety.

• Scenario-level improvements are broad: LCDrive reduces ADE by 7 to 15 percent relative to the non-reasoning model on most complex maneuvers such as Intersection Navigation, Turning Maneuver, and Merging, with the largest relative gains in Traffic Control Compliance, Speed Control, and Nudge Static Obstacle Maneuver. Against Text CoT, LCDrive achieves lower ADE in every scenario, with the largest gaps in Lead Vehicle Following (1.708 vs. 1.455) and Stop for Vehicle (0.942 vs. 0.919).

• Oracle results shine in interaction-heavy scenarios: With perfect LWM plus RL, ADE drops to 1.220 in Cut-In and 1.897 in Lane Change; with perfect LWM alone, ADE is 1.300 in Intersection Navigation and 0.542 in Stop for Vehicle.

• Benchmark design matters: The scenario-balanced PhysicalAI-AV subset contains 39,072 training clips (87 hours) and 23,758 validation clips (53 hours), with 30 percent General Driving clips and the remaining 70 percent evenly distributed across 14 specific scenarios at 5 percent per category. The authors contrast this with nuScenes, which has only 150 validation clips (less than 1 hour) and 73.9 percent straight driving.

• Efficiency claims are qualitative: The paper repeatedly states LCDrive uses a smaller inference budget and faster inference than text-based CoT, but no specific latency or token-count numbers are reported in the available content.

Methodology in Plain English

The model takes multi-view camera frames, ego motion history, and produces a 6.4-second future trajectory at 10 Hz, represented as 64 discrete trajectory tokens. Trajectories are quantized with a 1024-entry codebook built by k-means over ego-frame delta-poses.

Instead of writing sentences before driving, LCDrive reasons by alternating two kinds of latent tokens:

• Action-proposal tokens, drawn from the exact same vocabulary as the final output trajectory, grouped in 1.0-second blocks of 10 tokens.
• World model tokens, produced by a latent world model that predicts how nearby agents will move over the same 1.0-second window given a proposed action block.

Reasoning is seeded by an initial latent world state summarizing the most recent 1.0-second history window (10 frames), which can come from online perception or be predicted by the model itself. The model generates a fixed number of reasoned branches (default 2), each with a fixed reasoning depth (default 5), producing interleaved action and world model traces that act as counterfactual futures. The final trajectory attends to all proposals and their predicted outcomes.

Training proceeds in three stages. First, a non-reasoning VLA is trained by supervised fine-tuning for 100k steps at batch size 128 and learning rate 4e-5 with cosine annealing. Second, a frozen copy of that model samples diverse action proposals, and the latent world model targets are generated by re-centering ground-truth future agent boxes into the ego frame implied by each proposal; the model is then cold-started with teacher forcing for 10k steps, minimizing cross-entropy over proposals and the final plan plus a weighted (lambda = 0.1) squared error on world model embeddings. Third, GRPO reinforcement learning runs for 3k steps with group size 8, effective batch size 32 sampled completions per update, and learning rate 1e-6, using a trajectory-level reward equal to the negative average displacement error and no KL regularization.

The implementation uses a Qwen3-0.5B language-action module with a DINOv2 ViT image encoder following the AR1 architecture, two front-view cameras (wide 120 degrees and telephoto 30 degrees) at 320 by 512 resolution. The latent world model state uses N = 64 nearest agents compressed into M = 2 tokens.

Why This Matters

Impact on research: The paper challenges the default assumption that chain-of-thought must be expressed in natural language. It shows that an action-aligned latent reasoning space can serve as a more efficient and better-aligned reasoning substrate for spatially grounded tasks, and that reinforcement learning interacts very differently with latent reasoning than with non-reasoning policies. This connects the driving VLA literature to recent work on latent reasoning in LLMs and vision-language models.

Real-world applications:

• Autonomous passenger vehicles that must anticipate cut-ins, merges, and vulnerable road user behavior in dense urban traffic.
• Robotaxi or shuttle fleets that need real-time planning under latency constraints where generating long text rationales is impractical.
• Driver-assistance systems that must respond to traffic control compliance and speed control scenarios.
• Closed-loop simulation and policy improvement pipelines that use a learned latent world model to generate counterfactual futures for training.

Industry relevance: The work targets a practical bottleneck in deploying reasoning-based driving models: inference latency. By keeping reasoning in the same token vocabulary as actions, it reduces the engineering gap between reasoning output and control output, and it demonstrates that RL post-training gives larger returns on latent reasoning policies than on non-reasoning ones.

Future Directions

1. Reducing supervision dependence: Training latent CoT currently requires a source of grounding supervision such as ground-truth agent bounding boxes, which the authors note may be difficult to obtain at scale, though they point to recent autolabeling efforts as a path forward.
2. Interpretability: The current model cannot easily recover a human-interpretable representation from a latent CoT token, for example for in-car visualization. The authors call the efficiency-interpretability tradeoff an open area.
3. Adaptive reasoning length: The model does not support flexible reasoning lengths that adjust to task difficulty. Reasoning depth and branch factor are fixed at training and evaluation.
4. Broader evaluation of latent reasoning: Whether the action-aligned latent reasoning formulation transfers to other embodied or spatial-reasoning domains beyond driving remains an open question raised by the paper.

Target Audience

This paper is most valuable to autonomous driving researchers working on end-to-end planning and VLA-based policies, and to machine learning researchers interested in latent reasoning, world models, or reinforcement learning post-training for multimodal models. Engineers building production driving stacks will find the latency and reasoning-alignment arguments practically relevant, while readers looking for detailed quantitative efficiency comparisons should note that specific latency and token-count figures are not reported in the available content.

Authors’ abstract

Recent Vision-Language-Action (VLA) models for autonomous driving explore inference-time reasoning as a way to improve driving performance and safety in challenging scenarios. Most prior work uses natural language to express chain-of-thought (CoT) reasoning before producing driving actions. However, text may not be the most efficient representation for reasoning. In this work, we present Latent-CoT-Drive (LCDrive): a model that expresses CoT in a latent language that captures possible outcomes of the driving actions being considered. Our approach unifies CoT reasoning and decision making by representing both in an action-aligned latent space. Instead of natural language, the model reasons by interleaving (1) action-proposal tokens, which use the same vocabulary as the model's output actions; and (2) world model tokens, which are grounded in a learned latent world model and express future outcomes of these actions. We cold start latent CoT by supervising the model's action proposals and world model tokens based on ground-truth future rollouts of the scene. We then post-train with closed-loop reinforcement learning to strengthen reasoning capabilities. On a large-scale end-to-end driving benchmark, LCDrive achieves faster inference, better trajectory quality, and larger improvements from interactive reinforcement learning compared to both non-reasoning and text-reasoning baselines.