SynAD: Enhancing Real-World End-to-End Autonomous Driving Models through Synthetic Data Integration

AI summary

Overview

Research area: Robotics / autonomous driving — specifically end-to-end autonomous driving (E2E AD), synthetic traffic scenario generation, and sensor-free bird's-eye-view (BEV) representation learning.

Technical level: Intermediate to Advanced. The paper assumes familiarity with BEV feature extraction, transformer encoders/decoders, conditional diffusion models, and modular E2E AD pipelines (perception, prediction, planning).

Scope: SynAD is a framework that lets real-world E2E AD models train on synthetic driving scenarios that have no camera or LiDAR inputs, by converting both synthetic and real scenarios into map-based BEV features.

What This Paper Is About

End-to-end autonomous driving models learn from real-world datasets such as nuScenes, but collecting and labeling that data is expensive, so the range of driving scenarios available for training stays narrow. Synthetic scenario generators can produce diverse and controllable traffic situations, but their outputs are only path-level and come with no designated ego vehicle and no sensor inputs, so they have not been usable for training real-world E2E AD models. SynAD closes that gap by selecting the most informative agent as the ego vehicle, projecting scenarios onto maps, and learning a Map-to-BEV Network that turns those maps into BEV features the E2E model can consume.

Key Contributions

1. Ego-centric scenario generation for E2E AD. The paper transforms path-level synthetic scenarios into ego-centric maps by designating the agent that travels the longest distance as the ego vehicle (Eq. 7), cropping a fixed-size map centered on it, and transforming trajectories and bounding boxes into an ego-relative coordinate frame.
2. A Map-to-BEV Network. A transformer encoder with a learnable query produces BEV features directly from ego-centric maps, trained with an L2 loss to match BEV features that a frozen pre-trained BEVFormer extracts from multi-camera images — removing any need for sensor inputs on synthetic data.
3. A training strategy that integrates synthetic and real map data. Real scenarios are also projected onto maps (x_RM) to act as a training bridge that reduces the domain gap, and map-derived BEV features are applied selectively — only to motion forecasting and planning, not occupancy prediction.
4. The first framework to enhance real-world E2E AD models with synthetic data, with ablation studies isolating the effect of each component.

Main Findings

• Scenario generation quality (Table 1). SynAD reports no collision rule 0.033, real 0.045, rel real 0.330 and no offroad rule 0.002, real 0.040, rel real 0.324, compared with CTG++ at 0.036 / 0.040 / 0.332 and 0.004 / 0.038 / 0.328, and BITS at 0.065 / 0.099 / 0.352 and 0.018 / 0.099 / 0.355. The paper states that its method shows strong compliance with traffic constraints, and that the generated trajectories deviate from real-world paths while still capturing diverse driving behaviors — which the authors argue is advantageous for robust E2E AD.
• Planning safety (Table 2). On the nuScenes validation set, SynAD achieves the lowest collision rate of all compared methods: 0.04% at 1s, 0.10% at 2s, 0.20% at 3s, average 0.11%, versus UniAD (0.17% average), VAD (0.22%), Paradrive (0.25%), OccWorld (0.24%), ST-P3 (0.71%) and OCCNet (0.72%). Its L2 errors are 0.52 m, 0.78 m, 1.10 m (average 0.80 m), higher than Paradrive (0.48 m average) and VAD (0.72 m), which the paper attributes to a broader distribution of generated behaviors.
• Prediction accuracy (Table 3). SynAD reaches minADE 0.69, minFDE 1.01 and MR 0.154 for motion forecasting, better than UniAD (0.75 / 1.10 / 0.166) and Paradrive (0.73 / 1.08 / 0.162), and occupancy IoU-n 60.5 / IoU-f 39.6 versus UniAD's 61.9 / 39.7 and Paradrive's 60.0 / 36.4. The authors note this validates that image-only testing still works after synthetic map-based training.
• Selective module integration matters (Table 4). With no map data, the baseline records L2 1.15 and collision rate 0.25%. Adding map-based BEV features to the motion forecasting and planning modules brings these to L2 0.77 and collision 0.14%, and minADE 0.73 / minFDE 1.06 / MR 0.157 with IoU-n 60.2 / IoU-f 39.2. Extending the real map to occupancy prediction degrades occupancy performance (IoU-n 58.4, IoU-f 37.6), so the paper excludes that module from map inputs.
• More synthetic scenes help under equal-epoch training (Table 5). Under the same-epoch protocol, 0, 100, 300 and 500 synthetic scenes give minADE 0.76 / 0.72 / 0.71 / 0.69, minFDE 1.11 / 1.04 / 1.02 / 1.01, MR 0.162 / 0.156 / 0.155 / 0.154, IoU-n 60.1 / 60.3 / 60.3 / 60.5, IoU-f 38.9 / 39.1 / 39.4 / 39.6, L2 1.15 / 0.76 / 0.77 / 0.80 and collision 0.25% / 0.13% / 0.12% / 0.11%. Under the same-step protocol, gains appear at 100 scenes and then show diminishing returns (minADE 0.73 at 100 and 500, 0.72 at 300).
• The Map-to-BEV design beats SwinUNETR (Table 6). With a 224-resolution input, SynAD's network achieves validation map L2 loss 8.96 ×10⁻² and downstream minADE 0.73, minFDE 1.06, MR 0.157, planning L2 0.77, collision 0.14%, versus SwinUNETR at 800-resolution input with 9.55 ×10⁻², 0.75, 1.11, 0.158, 1.08 and 0.26%.
• Guides progressively improve safety (Table 7). Using the agent guide alone gives average collision 0.18%; adding the map guide gives 0.15%; adding the speed guide gives 0.14% (with average L2 moving from 0.75 to 0.76 to 0.77 m).
• Choice of ego vehicle matters (Table 8). The "longest" rule produces average L2 0.77 m and collision 0.14%, versus 0.79 m / 0.15% for "dynamic" and 0.79 m / 0.16% for "random," with average traveled distance 3.70, 3.68 and 3.67 respectively.

Methodology in Plain English

The pipeline has three stages.

1. Generate and reframe scenarios. A conditional diffusion model generates multi-agent trajectories, where each agent's state is 2D position, speed and heading angle. The map layout (road structure only, no vehicles) conditions the reverse diffusion process, and inference-time "guides" — agent collision, map collision and speed-limit terms — are added as gradients to the denoising mean so trajectories satisfy rules. Among the generated agents, the one traveling the longest distance becomes the ego vehicle. A fixed-size map crop is taken around it, and all trajectories and bounding boxes are rotated and translated so the ego's driving direction points along the positive y-axis with the ego at the origin. Because synthetic ego vehicles can vary in size, each instance also carries the ego's width and height.

2. Learn to read BEV from maps. Since synthetic scenarios have no cameras or LiDAR, the authors train a Map-to-BEV Network in which a map encoder supplies keys and values to a transformer encoder and a learnable query produces the BEV feature. Real scenarios are projected onto maps in the same format, so paired (real map, real multi-camera image) data can be used: a frozen pre-trained BEVFormer produces the target BEV feature from images, and the Map-to-BEV Network is trained with an L2 loss to match it. BEVFormer and the Map-to-BEV Network are then frozen while the E2E model is trained.

3. Integrate into the E2E model. The E2E AD model comprises motion forecasting, occupancy prediction and planning. Map-derived BEV features are fed only to motion forecasting and planning, where road geometry and traffic structure help, and are kept out of occupancy prediction, which needs finer spatial detail. Planning uses an imitation loss against the ego trajectory plus a collision loss computed by IoU between the predicted ego box (inflated by a safety margin) and other vehicles' boxes. At test time only real camera images are used, so practical deployment is unchanged.

Experimental setup. All experiments use nuScenes (1,000 scenes, 40 video frames each, 20-second videos at 2 Hz). The future prediction timestamp T_p is set to 6, giving 34 training instances per scene, and instances containing only a single driving agent are excluded. Main results train from scratch for 5 epochs with 500 added synthetic scenes, equivalent to 7.5 epochs on the original nuScenes alone; ablations use 100 synthetic scenes. Training uses 8 NVIDIA RTX 4090 GPUs with batch size 1 per GPU.

Why This Matters

Impact on research. Before this work, synthetic scenario generators could only be validated in simulators or in isolation, because their outputs lacked an ego designation and sensor data. SynAD makes generated scenarios directly trainable in real-world E2E AD pipelines, which gives a route from the active scenario-generation literature (diffusion, LLM-guided, retrieval-based) into production-style driving models.

Real-world applications:

• Expanding the training distribution for autonomous driving

Authors’ abstract

Recent advancements in deep learning and the availability of high-quality real-world driving datasets have propelled end-to-end autonomous driving. Despite this progress, relying solely on real-world data limits the variety of driving scenarios for training. Synthetic scenario generation has emerged as a promising solution to enrich the diversity of training data; however, its application within E2E AD models remains largely unexplored. This is primarily due to the absence of a designated ego vehicle and the associated sensor inputs, such as camera or LiDAR, typically provided in real-world scenarios. To address this gap, we introduce SynAD, the first framework designed to enhance real-world E2E AD models using synthetic data. Our method designates the agent with the most comprehensive driving information as the ego vehicle in a multi-agent synthetic scenario. We further project path-level scenarios onto maps and employ a newly developed Map-to-BEV Network to derive bird's-eye-view features without relying on sensor inputs. Finally, we devise a training strategy that effectively integrates these map-based synthetic data with real driving data. Experimental results demonstrate that SynAD effectively integrates all components and notably enhances safety performance. By bridging synthetic scenario generation and E2E AD, SynAD paves the way for more comprehensive and robust autonomous driving models.