ResWorld: Temporal Residual World Model for End-to-End Autonomous Driving

AI summary

Overview

• Research area: End-to-end autonomous driving, specifically world models used as a replacement for auxiliary perception and prediction tasks in trajectory planning (Computer Vision, arXiv:2602.10884v1).
• Technical level: Advanced.
• One-sentence scope: The paper introduces ResWorld, a framework that models only the changes between consecutive BEV scene representations to predict the future positions of dynamic objects, and then uses those predicted future BEV features to refine the ego vehicle's planned trajectory, evaluated on nuScenes and NAVSIM.

What This Paper Is About

End-to-end driving models increasingly use "world models" — networks that predict what the driving scene will look like in the future — as a proxy task to improve planning accuracy without expensive perception labels. The authors argue that existing world models waste effort re-predicting static things (ground, buildings) that do not move, while under-modeling the dynamic objects (vehicles, pedestrians) that actually matter, and that the predicted future scenes are never used to directly interact with the planned trajectory. ResWorld addresses both issues by predicting only the temporal residuals of the scene and by explicitly refining a prior trajectory against the predicted future BEV features.

Key Contributions

1. Current-frame BEV coordinate system for future prediction: Future BEV representations are expressed in the coordinate system of the current BEV feature B_t, so static objects require no re-modeling — the current fused features already describe where they will remain.
2. Temporal Residual World Model (TR-World): Scene queries from adjacent timestamps are subtracted to obtain temporal residuals, which represent dynamic objects without relying on detection or tracking; TR-World takes only these residuals as input and predicts the future spatial distribution of dynamic objects, which is then mapped back onto the base BEV feature via TokenFuser.
3. Future-Guided Trajectory Refinement (FGTR): Deformable attention between waypoint queries and the predicted future BEV features, using the prior trajectory as reference points, both corrects the trajectory (verifying collisions and drivable-area violations) and provides sparse spatial-temporal supervision that prevents world model collapse.
4. State-of-the-art results on two benchmarks: nuScenes open-loop planning and the NAVSIM navtest closed-loop split.

Main Findings

• nuScenes L2 and collision rate: ResWorld without auxiliary tasks and without ego status reaches an average L2 error of 0.65 m and average collision rate of 0.23%, versus 0.74 m and 0.31% for SSR. With ego status in the planner, ResWorld reaches 0.59 m and 0.17%.
• Under the VAD-style metric: ResWorld (no ego status) achieves 0.35 average L2 with 0.07 average collision rate; with ego status, 0.30 average L2 with 0.06 average collision rate. For comparison, BEV-Planner++ reports 0.35 average L2 and 0.34 average collision rate, and DiffusionDrive reports 0.57 and 0.08.
• NAVSIM PDMS: ResWorld with detection and BEV map segmentation tasks reaches 88.3 PDMS, and the version exploiting historical frames for temporal residuals reaches 89.0 PDMS (NC 98.9, DAC 96.5, TTC 95.6, Comf. 100, EP 83.1). The perception-free version using agent queries instead of temporal residuals reaches 87.3 PDMS. Prior world-model methods reported are LAW at 84.6 and World4Drive at 85.1; Hydra-MDP-W-EP reports 86.5 and DiffusionDrive 88.1.
• Component ablation (nuScenes): Combining TR-World and FGTR without ego status cuts the baseline's average L2 error by 8.4% and its average collision rate by 25.8%; with ego status in the planner, the reductions are 9.2% and 39.3%.
• TR-World beats a normal world model: TR-World with temporal residuals produces more accurate future BEV features and higher planning accuracy than a world model trained on the whole scene (average L2 0.59 versus 0.61, average collision rate 0.17 versus 0.21, in the no-supervision setting shown in the paper's table).
• Future supervision hurts TR-World: Using real data at time t+1 to supervise the predicted future BEV features lowers planning performance for TR-World, because it restricts the representation to a single future timestamp instead of a future time period. The same supervision has little effect on a normal world model, which spends most of its capacity on redundant static modeling.
• FGTR prevents world model collapse: Visualizations show that without FGTR the predicted future BEV features look nearly identical across different driving scenes and lack complete spatial information; with FGTR they show more diverse spatial distributions.
• Prior trajectories also improve: Even though the prior-trajectory branch uses the same architecture as the baseline, its average L2 drops from 0.65 m to 0.61 m and its average collision rate from 0.28% to 0.18%, which the authors attribute to the BEV features being improved by TR-World and FGTR. The final trajectory reaches 0.59 m and 0.17%.

Methodology in Plain English

The method builds on GeoBEV to turn multi-view camera images at each timestamp into a bird's-eye-view (BEV) feature map. Features from the current frame and the previous two frames (k = 2) are aligned into the current frame's coordinate system and fused.

From the fused feature, a TokenLearner module extracts a small set of sparse "scene queries" that summarize the scene; a set of waypoint queries attends to these and is decoded by an MLP into a prior trajectory (one 2D coordinate per future timestamp).

To isolate dynamic objects, the same spatial attention mask derived from the fused feature is applied to each individual timestamp's BEV feature, giving a scene query per timestamp. Subtracting the scene query of one timestamp from the next yields temporal residuals, which capture what changed at the same location — that is, moving objects — without any detection or tracking module. Each residual passes through self-attention and the results are summed into a single future dynamic representation.

A TokenFuser module then expands that representation back onto the fused BEV feature to produce a predicted future BEV feature B_future, still in the current frame's coordinate system so static content is inherited rather than re-predicted.

Finally, the FGTR module runs deformable attention where the waypoint queries attend to B_future using the prior trajectory coordinates as reference points, and an MLP decodes the refined final trajectory. Training uses only L1 loss on the prior and final trajectories against ground truth; no future BEV ground truth is used as supervision. For nuScenes, the image backbone is ResNet-50 on 256×704 images, trained 12 epochs on 8 NVIDIA RTX 3090 GPUs with batch size 8, the AdamW optimizer and learning rate 1×10⁻⁴. For NAVSIM, the structure follows TransFuser with two ResNet-34 backbones for images and LiDAR BEV maps, trained 100 epochs on the same GPU type with batch size 512 and learning rate 6×10⁻⁴.

Why This Matters

• Research impact: The paper reframes world modeling for driving as a problem of change detection rather than full-scene prediction, showing that a model can be trained with trajectory loss alone and still produce useful future scene representations. It also introduces the idea that a planner's interaction with the world model doubles as weak supervision keeping that model from collapsing.
• Label efficiency: ResWorld achieves its nuScenes numbers with no auxiliary tasks (no detection, tracking, mapping, motion or occupancy labels), which matters because those annotations are expensive compared with raw driving video.
• Real-world applications:
  • Collision-avoidance planning that checks a proposed route against predicted future positions of other vehicles.
  • Perception-free planning stacks for vehicles where annotation-heavy pipelines are impractical.
  • Training-set-scaling strategies where large amounts of unlabeled driving data are used to train the world model, with a cheaper prior-trajectory head used at inference.
  • Closed-loop simulation evaluation (NAVSIM) of planning policies before deployment.
• Industry relevance: The work targets camera-only, end-to-end driving with open-source code released at https://github.com/mengtan00/ResWorld.git, and reports results under both UniAD-style and VAD-style evaluation protocols plus the NAVSIM PDMS leaderboard metric, making it directly comparable to deployed-style planning baselines.

Future Directions

1. Handling potential dynamic objects: The stated limitation is that TR-World cannot adequately capture occluded or currently static-appearing dynamic objects such as pedestrians and parked cars through temporal residuals; they are instead handled by the prior-trajectory branch alongside static objects. Future work aims to use coarse perception to extract these and model them preventively.
2. Scaling the world model: The authors suggest training larger-scale TR-World and FGTR modules to obtain the best BEV features, then emitting prior trajectories at inference for higher efficiency — an open question is how far this train-large/infer-small asymmetry can be pushed.
3. Richer future supervision: Since supervising on a single future timestamp (t+1) degrades performance, an open question is what form of multi-timestamp or otherwise temporal-aware supervision could help rather than hurt.
4. Extending temporal residuals beyond short windows: With k = 2 historical frames on nuScenes and NAVSIM results depending on historical frames being available (agent queries were substituted otherwise), how the method behaves with longer histories or inconsistent frame availability is not reported.

Target Audience

Researchers and engineers working on end-to-end autonomous driving, BEV perception, and world models; graduate students familiar with transformers, BEV representations, and trajectory planning metrics; and practitioners evaluating perception-free or label-light planning architectures for nuScenes and NAVSIM. Readers without background in BEV feature alignment or deformable attention will find the method section demanding.

Authors’ abstract

The comprehensive understanding capabilities of world models for driving scenarios have significantly improved the planning accuracy of end-to-end autonomous driving frameworks. However, the redundant modeling of static regions and the lack of deep interaction with trajectories hinder world models from exerting their full effectiveness. In this paper, we propose Temporal Residual World Model (TR-World), which focuses on dynamic object modeling. By calculating the temporal residuals of scene representations, the information of dynamic objects can be extracted without relying on detection and tracking. TR-World takes only temporal residuals as input, thus predicting the future spatial distribution of dynamic objects more precisely. By combining the prediction with the static object information contained in the current BEV features, accurate future BEV features can be obtained. Furthermore, we propose Future-Guided Trajectory Refinement (FGTR) module, which conducts interaction between prior trajectories (predicted from the current scene representation) and the future BEV features. This module can not only utilize future road conditions to refine trajectories, but also provides sparse spatial-temporal supervision on future BEV features to prevent world model collapse. Comprehensive experiments conducted on the nuScenes and NAVSIM datasets demonstrate that our method, namely ResWorld, achieves state-of-the-art planning performance. The code is available at https://github.com/mengtan00/ResWorld.git.