LEAD: Minimizing Learner-Expert Asymmetry in End-to-End Driving

AI summary

Overview

Research area: End-to-end autonomous driving via imitation learning, with closed-loop evaluation in the CARLA simulator and open-loop transfer to real-world driving benchmarks (NAVSIM, Waymo WOD-E2E). Also touches on synthetic data generation for perception pre-training.

Technical level: Advanced. The paper assumes familiarity with imitation learning, privileged "expert" policies, transformer-based sensor fusion, bird's-eye-view representations, and CARLA Leaderboard metrics.

Scope: The paper diagnoses three specific mismatches between privileged driving experts and sensor-based student policies, fixes them in both the expert design and the policy architecture, and reports new state-of-the-art closed-loop driving results.

What This Paper Is About

Simulators like CARLA can generate unlimited driving data, yet imitation-learned driving policies still plateau far below the performance of the experts that supervise them. The authors argue this is not mainly a model-capacity problem but an asymmetry problem: the expert sees things the student cannot (through occlusions), knows things the student cannot (exact velocities with no noise), and receives a dense route while the student gets only a single target point. The goal is to systematically close these gaps so that expert demonstrations become actually learnable, and then to fix the policy's navigation conditioning so it stops over-relying on that single goal point.

Key Contributions

1. Reducing learner–expert asymmetries. The authors systematically analyze and reduce three asymmetries in CARLA by aligning expert supervision with the student's observable state and strengthening the student's navigation intent.
2. Mitigating target point bias. They show that target point bias persists even with existing mitigation strategies, and identify insufficient intent specification plus poorly integrated (late, bottlenecked) intent conditioning as the key contributing factors.
3. A complete training stack. They release LEAD, a large-scale CARLA dataset and training pipeline that enables state-of-the-art closed-loop driving plus measurable sim-to-real gains on NAVSIM and Waymo benchmarks.
4. A new policy, TransFuser v6 (TFv6). A GRU-free transformer decoder that treats the target point as an explicit token alongside BEV tokens, and uses a three-point route representation (previous, current, future targets).

Main Findings

• State-aligned expert data alone gives large gains without any model change. Training the baseline TFv5 on the LEAD dataset instead of the PDM-Lite dataset raised Longest6 v2 Driving Score from 22.51 ± 4.42 to 34.05 ± 1.50 (+11 DS) and Bench2Drive DS from 83.56 ± 0.32 to 84.94 ± 0.50 (+1.37), while the expert's own performance stayed the same.

• Removing the GRU bottleneck is a bigger win than expected. Replacing the Gated Recurrent Unit refinement stage with direct token conditioning improved Longest6 v2 DS from 34.05 ± 1.50 to 40.70 ± 2.86 (+6 DS) and Bench2Drive DS from 84.94 ± 0.50 to 87.26 ± 0.47 (+2 DS). The GRU is only a single recurrent layer with 64 hidden units sitting after a six-layer, 256-dimensional transformer decoder, and it decoupled steering (target-point conditioned) from speed prediction (not conditioned).

• Denser intent conditioning helps further. Moving from one target point to three (previous, current, future) raised Longest6 v2 DS from 40.70 ± 2.86 to 42.13 ± 0.75 and Bench2Drive DS from 87.26 ± 0.47 to 89.29 ± 0.45.

• New state of the art on CARLA closed-loop benchmarks. TFv6 reaches 95.0 ± 0.7 DS on Bench2Drive and 54 ± 5 DS on Longest6 v2 with a 360° camera plus LiDAR and radar; the best configuration (RegNetY-032 backbone, 140° FOV, LiDAR + radar) reaches 95.2 ± 0.3 DS and 86.8 ± 0.7 SR on Bench2Drive, and 62 ± 1 DS / 91 ± 1 RC on Longest6 v2. This is an 8 DS improvement over the previously established best on Bench2Drive, and gains of +39 DS and +21 RC over TFv5 and SimLingo on Longest6 v2.

• Success Rate reveals hidden weakness. On Bench2Drive, TFv6 comes within 2 DS of the LEAD expert but has a 10-point Success Rate gap. Because DS discounts late infractions on short routes, SR (all-or-nothing) exposes that the policy is less robust than DS suggests.

• Town13 generalization gap is severe. TFv6 achieves 14.65 NDS on Town13 Train but only 4.04 NDS on Town13 Validation, where no data from that town is allowed. Both beat TFv5 (4.94 NDS Train, 2.12 NDS Val), and the privileged PDM-Lite expert reaches 58.50 NDS on Val.

• Aligned supervision does not sacrifice expert competence. LEAD matches PDM-Lite on Bench2Drive (96.8 vs 97.0 DS) and Longest6 v2 (73 DS each).

• Wide FOV and sensor fusion help. A 140° FOV camera setup with both LiDAR and radar gives the best Longest6 v2 results, and the larger RegNetY-032 backbone outperforms ResNet-34.

• Synthetic data transfers to real-world benchmarks. On NAVSIM v1 PDMS, LTFv6 improves from 83.8 (LTF) to 85.4, and to 86.4 with LEAD pre-training; on NAVSIM v2 EPDMS from 23.1 to 28.3, and to 31.4 with LEAD; on WOD-E2E RFS from 7.51 to 7.76. Expert upper bounds are 94.5, 51.3, and 8.10 respectively.

• Fewer infractions, but more route deviation. Infractions generally decrease as alignment improves, except route deviation increases, because the model no longer aggressively snaps back toward target points after going off route.

Methodology in Plain English

The paper follows the "Learning by Cheating" recipe: first build a privileged expert that drives using ground-truth simulator state, then train a sensor-based student to imitate it. The authors keep the model architecture and dataset scale fixed during their controlled ablations so any performance change can be attributed to the alignment changes, not to extra capacity or data.

Reducing visibility asymmetry. The expert is only allowed to react to dynamic actors inside the student's camera view, taking into account actor extent, weather, and time of day. Traffic-light stopping logic only considers lights inside the camera frustum. Since explicit speed-limit signs are only intermittently visible and are not fed to the model, the expert's target speed is capped at the minimum of the posted limit and the typical flow of nearby vehicles.

Reducing uncertainty asymmetry. The expert brakes not only for predicted collision courses but also in the presence of nearby observable hazards, so it does not depend on velocity or acceleration estimates the student cannot reliably infer. Under low-visibility conditions like night or heavy rain, the expert drives more slowly. During unprotected turns at junctions, oncoming actors' bounding boxes are enlarged so safety decisions rely on conservative spatial margins rather than precise motion prediction.

Fixing intent. The student's navigation was previously injected through a small GRU whose hidden state was initialized with a single GNSS target point, inserted after the transformer decoder. The authors remove the GRU and instead feed the target point as an explicit token alongside the BEV tokens (normalized to [-1, 1] using training statistics), and they replace the single target with three targets (previous, current, future) plus a shorter switching threshold so future targets matter earlier.

Final model and data. TFv6 is trained on an expanded LEAD dataset of 73 hours of driving (versus 40 hours used in the ablation section), using 4 L40S GPUs for roughly one week in mixed precision. Four radar units (75 detections each per frame) are pre-processed by a lightweight learned module into object-level features that bypass the sensor-fusion encoder and enter the planning decoder directly as context tokens. For real-world benchmarks where LiDAR and radar are unavailable, only perception labels from CARLA are used, LiDAR is replaced with a positional encoding as in Latent TransFuser, and the resulting model is called LTFv6.

Why This Matters

Impact on research. The paper reframes the bottleneck in simulation-based imitation learning as expert design and goal specification rather than model scale. It provides a concrete, released dataset (LEAD) and pipeline, and shows that in the CARLA closed-loop benchmarks a single DS point is a large margin—the best and fifth-best published methods are separated by roughly 2 DS. It also argues that short-route benchmarks hide failures that only appear on long horizons, where CARLA is currently the only widely used simulator supporting long-form evaluation.

Real-world applications:

• Autonomous driving stacks that need robust lane-change and multi-step maneuver handling, where richer navigation intent (three-point route) directly addresses goal-fixation failures.
• Safety-aware planner design, where conservative, uncertainty-aware braking policies translate into larger safety margins under poor visibility.
• Synthetic data pipelines for perception pre-training, since LEAD pre-training improved NAVSIM and Waymo benchmark scores even with a driving-style mismatch between simulators and human data.
• Radar integration in driving policies, demonstrating object-level radar features as a useful, underused modality in current benchmarks.

Industry relevance. The consistent sim-to-real gains, the open-source release (github.com/kesai-labs/lead), and the fact that improved alignment did not cost expert competence make the approach directly adoptable by teams building simulation-based training loops. The finding that DS alone can mask poor robustness (via the SR gap) is also practically important for teams choosing evaluation metrics.

Future Directions

• Closing the remaining gap to the expert. The authors attribute the residual difference largely to behavior cloning limits such as compounding errors and inability to recover from off-route deviations, and suggest closed-loop training via DAgger or reinforcement learning.
• Planning co-training for sim-to-real. The paper investigates perception co-training only; combining supervision from human driving trajectories and LEAD for planning behavior is left open.
• True closed-loop real-world validation. Current sim-to-real evaluation is limited to open-loop and pseudo-closed-loop benchmarks; demonstrating closed-loop real-world benefits remains future work.
• Generalizing the alignment principles. The study is restricted to a rule-based expert in simulation and requires domain-specific tuning; whether similar principles apply to learned experts, human demonstrations, or real-world driving is unanswered. The authors also note that generative approaches may help overcome the short-form limitation of most current simulation benchmarks.

Target Audience

Researchers and engineers working on end-to-end autonomous driving, imitation learning, and closed-loop simulation benchmarking will get the most from this paper. It is also relevant to practitioners in autonomous vehicle companies interested in sim-to-real data pipelines, radar-based perception, or evaluation metric design, and to academic groups building on CARLA or the TransFuser line of work. Readers unfamiliar with privileged-expert imitation learning or CARLA metrics will find the concepts approachable in the introduction, but the experimental detail is aimed at specialists.

Authors’ abstract

Simulators can generate virtually unlimited driving data, yet imitation learning policies in simulation still struggle to achieve robust closed-loop performance. Motivated by this gap, we empirically study how misalignment between privileged expert demonstrations and sensor-based student observations can limit the effectiveness of imitation learning. More precisely, experts have significantly higher visibility (e.g., ignoring occlusions) and far lower uncertainty (e.g., knowing other vehicles' actions), making them difficult to imitate reliably. Furthermore, navigational intent (i.e., the route to follow) is under-specified in student models at test time via only a single target point. We demonstrate that these asymmetries can measurably limit driving performance in CARLA and offer practical interventions to address them. After careful modifications to narrow the gaps between expert and student, our TransFuser v6 (TFv6) student policy achieves a new state of the art on all major publicly available CARLA closed-loop benchmarks, reaching 95 DS on Bench2Drive and more than doubling prior performances on Longest6~v2 and Town13. Additionally, by integrating perception supervision from our dataset into a shared sim-to-real pipeline, we show consistent gains on the NAVSIM and Waymo Vision-Based End-to-End driving benchmarks. Our code, data, and models are publicly available at https://github.com/autonomousvision/lead.