Plug-and-Play Traffic Element Awareness for End-to-End Autonomous Driving

AI summary

Overview

Research area: End-to-end (E2E) autonomous driving, specifically traffic-element (traffic light and road sign) perception and its downstream effect on trajectory planning.

Technical level: Advanced. The paper assumes familiarity with E2E driving paradigms (regression planners, diffusion policies, trajectory scoring, VLA models), BEV representations, and driving benchmarks such as nuScenes, NAVSIM, and Bench2Drive.

Scope: A systematic, plug-and-play study of whether and how traffic-element awareness improves E2E driving across six representative methods and four benchmarks.

What This Paper Is About

End-to-end driving research has focused overwhelmingly on dynamic road participants (vehicles, pedestrians) and dense geometry, while traffic elements such as traffic lights and road signs — which directly constrain what maneuvers are legal — remain largely unquantified. The authors argue the community has never systematically answered how much traffic-element awareness actually matters for planning across modern E2E paradigms. Their goal is therefore not another bespoke planner, but a unified research infrastructure (annotations plus a minimal integration mechanism) and generalizable conclusions valid across diverse model families and datasets.

Key Contributions

1. A systematic, cross-paradigm investigation of traffic element awareness. The paper claims to be the first study that quantifies how traffic elements affect E2E driving across a wide spectrum of architectures, rather than within a single method. Notably, the authors report that in nuScenes, 59.1% of training scenes (16,626 / 28,130) and 58.3% of validation scenes (3,508 / 6,019) contain traffic elements, and NAVSIM-v1 (navtest) contains them in 65.1% of scenes (7,904 / 12,147); Figure 1 states that over 55% of scenes across three benchmarks contain at least one traffic element.

2. Augmentation of multiple public driving benchmarks with structured traffic-element annotations. nuScenes uses 2D traffic-element boxes and topology annotations from OpenLane-V2, with the authors' pipeline deriving 3D centers; Bench2Drive natively provides 3D traffic-element coordinates and topological relationships; NAVSIM-v1 and NAVSIM-v2, which lack traffic-element annotations, are annotated with pseudo-labels produced by a YOLO-based 2D detector trained on OpenLane-V2 (with progressive training strategies such as resampling, reweighting, and pseudo labeling) combined with a depth–LiDAR fusion pipeline.

3. A minimal, plug-and-play integration mechanism. Traffic elements are added as an auxiliary 3D detection objective (L1 loss on center location plus focal loss on category), with the TE loss weight set identically to the other auxiliary losses, plus optional language-guided topology conditioning. The design introduces negligible architectural modification and is applied to six representative methods.

4. Consistent improvements and a new state of the art on NAVSIM-v2. Across all paradigms and datasets the integration improves driving performance; on NAVSIM-v2 the authors report that a scoring-based backbone combined with large-scale simulated co-training data reaches 57.9 EPDMS, which they describe as a new state of the art in their evaluation.

Main Findings

• Traffic element awareness improves both conventional and VLM-based planners on nuScenes. VAD improves from 0.72 to 0.60 average L2 error, with intermediate variants VAD + TE at 0.63 and VAD + Topo at 0.62; Orion improves from 0.34 to 0.26 average L2, with Orion + TE at 0.29 and Orion + Topo at 0.27. The paper summarizes these as reductions of 0.09 and 0.05 in L2 and 0.05% and 0.10% in collision rate for the two baselines. The best results come from combining traffic elements and topology.

• Runtime cost is small. VAD's throughput drops from 5.7 to 5.4 FPS (a 0.3 FPS drop), because topology and traffic-element cues are predicted by lightweight networks. The authors report this delivers more than 10× higher throughput than VLM-based scene understanding methods such as Qwen-2.5-VL (0.8 FPS), ImpromptuVLA (0.9 FPS), and DriveVLM.

• Consistent PDMS gains on NAVSIM-v1 across three paradigms. LTF rises from 84.1 to 85.2 (+1.1); DiffusionDrive from 86.0 to 87.7 (+1.7); DrivoR from 93.1 to 94.4 (+1.3). With SimScale co-training added, LTF + SimScale reaches 87.3 (+3.2) and 87.6 with the method (+3.5); DiffusionDrive + SimScale reaches 88.9 (+2.9) and 89.1 (+3.1); DrivoR + SimScale reaches 94.6 (+1.5) and 95.1 (+2.0). Gains are attributed mainly to higher No Collision (NC) and Drivable Area Compliance (DAC).

• The best NAVSIM-v1 result exceeds both the rule-based and human-driver reference points. DrivoR + SimScale with the method reaches 95.1 PDMS, above PDM-Closed (89.1) and the reported human driver score (94.8).

• NAVSIM-v2 gains are large and consistent. The paper reports roughly +10 EPDMS overall across methods, driven by the compliance and safety terms NC, DAC, DDC, and TLC. Relative improvements are reported as 15% (LTF, 25.1 to 28.9), 11% (DiffusionDrive, 29.4 to 32.7), and 7.2% (DrivoR, 48.3 to 51.8) without SimScale; and 47% (LTF + SimScale, 33.6 to 36.9), 29% (DiffusionDrive + SimScale, 35.8 to 37.9), and 20% (DrivoR + SimScale, 54.6 to 57.9) with SimScale.

• Simulated co-training can hurt comfort, and the method mitigates this. The authors observe that adding SimScale data noticeably reduces Extended Comfort (EC), likely due to distribution shifts from counterfactual simulated scenarios, and that combining SimScale with TE-centric training brings EC back to a higher level.

• Closed-loop improvements on Bench2Drive. VAD improves from 0.91 to 0.75 average L2 and from 42.3 to 56.4 Driving Score (+14.1), with Success Rate rising from 15.00 to 21.30, at a latency change from 278.3 ms to 282.5 ms. DriveTransformer-Large improves from 0.62 to 0.57 L2 and from 63.46 to 68.29 Driving Score (+4.83), Success Rate from 35.01 to 39.61, at 211.7 ms to 216.5 ms.

• Efficiency drops in closed-loop, interpreted as a correction. Efficiency falls from 157.94 to 125.64 for VAD and from 100.64 to 82.45 for DriveTransformer-Large. The authors interpret this as correcting over-aggressive behavior encouraged by efficiency-dominated objectives.

• Explicit 3D representations beat alternatives. On NAVSIM-v2 with LTF, the full method reaches 28.9 EPDMS (+3.8 over the 25.1 baseline), versus 28.1 (+3.0) for 2D front-view traffic elements, 27.7 (+2.6) for full front-view depth, 27.9 (+2.8) for LiDAR-clustering-based traffic elements, 27.8 for a variant using the general-purpose Depth Anything 3 model instead of UniDepthV2, and 26.7 (+1.6) for traffic lights only.

• Signals, not just localization, carry the benefit. A class-agnostic variant that supervises only traffic-element localization with L1 loss and removes semantic classification reaches only 26.2 EPDMS on NAVSIM-v2, versus 28.9 for the full model.

• Integration design choices matter. Decoupling traffic-element prediction into an independent head yields a +1.8 EPDMS boost over treating traffic elements as an additional BEV semantic class with cross-entropy, which even degrades performance (−0.8) because gradients are dominated by dense background. Replacing cross-entropy with focal loss under the independent head adds a further +0.7 EPDMS. Explicitly routing traffic-element constraints to the planner via cross-attention improves the baseline by +0.6, but adaptive max pooling preserves peak activations better than average pooling, and the optimal configuration concatenates max-pooled traffic-element features into BEV memory, outperforming cross-attention by +1.4 EPDMS.

• Robustness to upstream perception noise. Corrupting predicted traffic elements at inference (depth noise, dropped detections, false positives) degrades performance smoothly while staying above the LTF baseline. Exact corruption levels and scores are not reported in the provided content.

• Qualitative evidence on a NAVSIM-v2 intersection. LTF and LTF + SimScale show lateral drift, while the method detects green lights and a straight-ahead sign for the ego lane and continues forward while staying lane-consistent.

• Topology encoding. When lane topology is available, encoding ego-relevant topology into compact language-style embeddings is reported as more effective than GNN-based graph encoding. The per-configuration numbers for the topology ablation appear in a table (Tab. 7) that is truncated in the provided content, so specific values are not reported here.

Methodology in Plain English

The authors start by building the data they need. Many driving datasets lack structured traffic-element labels, so for nuScenes they reuse OpenLane-V2's 2D boxes and topology annotations and then derive 3D positions; for Bench2Drive they use the dataset's native 3D coordinates; and for NAVSIM-v1 and NAVSIM-v2 they train a YOLO-based 2D detector on OpenLane-V2 and combine it with monocular depth estimation and LiDAR geometry to create pseudo-labels.

To get 3D locations from front-view images, they run two branches in parallel: a 2D traffic-element detector producing bounding boxes, and a monocular depth foundation model (UniDepthV2, with frozen weights) producing dense depth. They project the boxes into the LiDAR coordinate system using camera intrinsics and extrinsics, then fuse the box constraints, the estimated depth, and the LiDAR point cloud to determine each element's 3D center (x, y, z).

On top of each baseline planner's existing auxiliary tasks (detection, tracking, occupancy, and so on), they add one extra objective: predict each traffic element's 3D center with an L1 loss and its category with a focal loss, weighted identically to the other auxiliary losses. Nothing about the base architecture needs reworking, which is what makes it plug-and-play.

Optionally, for lane topology, they predict adjacency matrices describing which traffic elements govern which centerlines (LCTE) and how centerlines connect to each other (LCLC). They filter this to the ego vehicle's own lane, convert the resulting graph into a short structured sentence (for example, describing green traffic lights and a no-right-turn sign controlling the ego lane, which connects to one straight centerline), encode it with a frozen BERT-base encoder of roughly 110M parameters, and concatenate the resulting topology queries with the BEV queries before the planning decoder. Topology matrices are predicted with a lightweight TopoMLP.

Evaluation covers four benchmarks: nuScenes, NAVSIM-v1 (navtest), NAVSIM-v2 (navhard), and the closed-loop Bench2Drive. nuScenes and NAVSIM use perception ranges of lateral [−15, 15] m and longitudinal [0, 30] m, and lateral [−32, 32] m and longitudinal [0, 32] m, respectively. NAVSIM uses focal loss with α = 2 and β = 4 for the BEV traffic-element heatmap. Because NAVSIM's camera setup differs from nuScenes, no topology predictor is trained there; only traffic elements are used. Bench2Drive closed-loop planning runs at 2 Hz, and traffic elements fed to the planner at inference come from model predictions rather than annotations. Training used 8×A100 with batch size 1 on nuScenes, 4×RTX 3090 with batch size 32 on NAVSIM, and 8×A100 with batch size 1 on Bench2Drive, following each baseline's original schedules and hyperparameters.

Why This Matters

Impact on research. The work reframes traffic elements as a first-class, quantifiable factor in E2E driving rather than an unexamined detail, and supplies both an annotation pipeline and a shared integration recipe that others can reuse. It also offers evidence that broad conclusions can be drawn across paradigms when the integration is deliberately minimal, which is useful methodological guidance for the field.

Real-world applications:

• Urban robotaxi and shuttle deployment, where traffic lights, no-turn signs, and lane-level permissions determine whether a planned maneuver is legal and safe.
• Driver-assistance and highway/urban pilot features, where a lightweight auxiliary supervision head adds traffic-element awareness at a runtime cost reported as roughly a 0.3 FPS drop rather than the large overhead of a VLM-based scene understanding module.
• Data engine and annotation pipelines, since the paper shows a detector trained on OpenLane-V2 plus depth–LiDAR fusion can auto-generate traffic-element pseudo-labels for datasets that lack them, such as NAVSIM.
• Simulation-based training and validation, given the reported interaction between simulated co-training data (SimScale) and comfort metrics, and the finding that TE-centric training mitigates the comfort degradation.

Industry relevance. The work is a collaboration between Tsinghua's Institute for AI Industry Research and Bosch Corporate Research, which signals direct industrial interest in deploying traffic-element aware planners. Its emphasis on negligible architectural modification and near-unchanged latency addresses practical integration constraints, and its demonstrated plug-and-play behavior across regression, diffusion, scoring, perception–planning, and VLM/VLA paradigms means existing production architectures would not need to be redesigned.

Future Directions

• Improving the depth prior. The authors show that swapping UniDepthV2 for a general-purpose foundation model drops NAVSIM-v2 EPDMS to 27.8 from 28.9, indicating the underlying depth estimator bottlenecks performance; better driving-aligned depth is a clear next target.
• Topology integration remains only partially resolved. The topology ablation results are not fully reported in the available content, and the claim that language-style embeddings beat GNN-based graph encoding invites further study — particularly in settings such as NAVSIM, where topology was not trained at all due to camera differences.
• Closing the gap between open-loop and closed-loop behavior. The efficiency decrease on Bench2Drive suggests remaining tension between rule compliance and progress; how to balance these objectives is left open.
• Scaling behavior. The paper reports that stronger backbones and more training data unlock additional gains, and that simulated co-training helps when paired with traffic-element supervision, but the paper does not report how far this trend extends at larger model or data scale.

Target Audience

Researchers and engineers working on end-to-end autonomous driving, BEV perception, and trajectory planning; practitioners evaluating traffic-element or topology-aware modules for production driving stacks; and graduate students looking for an example of a cross-paradigm empirical study that isolates a single, widely under-examined input signal. Readers should already be comfortable with driving benchmarks, planning metrics such as L2 error, PDMS, and EPDMS, and standard perception terminology.

Authors’ abstract

Traffic elements such as traffic lights and road signs play a fundamental role in human driving decisions and should naturally influence end-to-end driving performance. However, existing end-to-end driving research predominantly focuses on dynamic road participants (e.g., vehicles and pedestrians), while the role of traffic elements remains largely unexplored. The community still lacks a systematic study quantifying their impact, largely because public datasets rarely provide structured traffic-element annotations and modern driving systems vary widely in architecture and training paradigm. In this work, we present the first systematic investigation of traffic element awareness for end-to-end autonomous driving. We construct a unified research infrastructure by augmenting multiple public driving datasets with comprehensive traffic-element annotations. To support diverse model families, we adopt a minimal and universal integration design that incorporates traffic-element signals into existing pipelines in a plug-and-play manner with negligible architectural modification. We evaluate this design across modern paradigms, including perception-prediction-planning pipelines, vision-language-action models (VLA), regression-based planners, diffusion-based policies, and trajectory-scoring frameworks, on nuScenes, NAVSIM-v1, NAVSIM-v2, and Bench2Drive. Across all paradigms and datasets, this simple integration consistently improves driving performance, demonstrating that traffic element awareness provides a robust and generalizable signal for end-to-end driving systems. Notably, on the challenging NAVSIM-v2 benchmark, our approach significantly improves state-of-the-art architectures and data pipelines, establishing a new state of the art.