One Diffusion Model, Two Roles: Guided Trajectory Planning and Safety-Critical Scenario Generation in Closed-Loop Simulation

AI summary

Overview

Research area: Autonomous driving — diffusion-based motion planning and safety-critical scenario generation for closed-loop simulation. Technical level: Advanced (assumes familiarity with diffusion probabilistic models, transformers, and AV planning benchmarks like nuPlan). Scope: The paper shows that a single pretrained diffusion traffic model can be repurposed both as an ego-vehicle planner and as a controllable adversarial scenario generator for stress-testing that same planner.

What This Paper Is About

Autonomous driving planners are usually evaluated on nominal driving data, but they must also handle rare, dangerous interactions that logged data almost never contains. This paper builds a diffusion model that learns the joint future trajectories of the ego vehicle and its neighbors, then reuses it for two jobs at once: planning safe ego trajectories, and generating realistic adversarial scenarios (cut-ins, sudden braking) that expose planner weaknesses in closed-loop simulation.

Key Contributions

1. SSDS-DiT decoder. A Single-Stream Dual-Stream transformer decoder that fuses scene context with trajectory tokens through joint attention (early, symmetric interaction) rather than the late cross-attention used in standard Diffusion Transformer decoders, followed by a single-stream fusion stage.
2. DAPSE guidance. Decoupled Annealing Posterior Sampling with Energy — a training-free sampler that injects arbitrary energy functions (e.g., target speed) at the clean-sample level, avoiding the first-order approximation errors of Diffusion Posterior Sampling and requiring no auxiliary guidance networks.
3. Controllable safety-critical scenario generation in closed loop. The same pretrained diffusion model is repurposed as a behavior generator for a designated adversary in nuPlan, steered by composable energy functions (time-to-collision with temporal lead, lane change, lead-vehicle braking, drivable-area compliance).
4. Dual-role demonstration. Empirical evidence that benchmark superiority (better nominal planning scores) does not guarantee adversarial robustness under generated long-tail scenarios.

Main Findings

• SSDS-DP helps most where interaction matters. Gains are largest on reactive, hard scenarios (Test14-hard Reactive: +6.8 points at 250K training scenarios, +14.4 at 650K), while gains narrow in non-reactive settings.
• Data efficiency advantage at low data. SSDS-DP consistently outperforms the reproduced Diffusion Planner baseline when trained on limited data (250K, 650K), suggesting joint attention helps inter-agent dependency modeling rather than raw trajectory accuracy.
• Baseline catches up at scale. With 1M training scenarios, the baseline DP improves substantially and even outperforms SSDS-DP on some splits — there is no clear winner at the largest scale.
• Adversarial scenarios degrade both planners. Under combined cut-in-and-braking guidance, nuPlan scores drop from 80.49 to 67.29 (DP) and from 84.65 to 53.96 (SSDS-DP); ego-at-fault collisions rise and TTC-in-bound/comfort scores fall.
• The stronger planner degrades more. SSDS-DP, which leads under nominal conditions, loses more performance under adversarial scenarios, showing benchmark superiority does not translate to robustness.
• Planners rely on reactive braking. Across all scenario types, neither planner attempts lateral evasive maneuvers (e.g., overtaking); both rely purely on longitudinal deceleration, which fails in intersection conflicts.
• Generated scenarios remain realistic. Realism deviation (matching speed, jerk, and acceleration distributions) is comparable to SafeSim under the same metric, so the adversarial behaviors stay physically plausible.

Methodology in Plain English

The team trains a diffusion model on nuPlan (1500 hours of human driving) to learn how the ego car and nearby agents jointly move in the future. A scene encoder processes agent histories, lane features, static objects, and the navigation route into a fused context vector. The new SSDS decoder then denoises trajectory and context tokens in two streams that talk to each other through joint attention, before merging them into one stream.

For planning, they introduce DAPSE: at each denoising step they take a noisy trajectory, run standard reverse diffusion to estimate a clean trajectory, then apply a few Langevin updates that pull the sample toward lower energy (e.g., keeping speed in a target range) while a reconstruction term anchors it to the model's prediction. No extra networks, no retraining.

For scenario generation, they reuse the same model but steer a chosen adversary agent with gradients of safety-relevant energy functions — time-to-collision (with a "temporal lead" so the adversary targets the ego's future position, not its current one), lane-centerline deviation for cut-ins, longitudinal progress penalties for braking, and an ESDF-based drivable-area term. Guidance is applied only during the low-noise stage of the reverse process, with 20 denoising steps and a logSNR schedule. The resulting trajectories are tracked by an LQR controller, background vehicles follow the Intelligent Driver Model, and the ego is driven by independent black-box planners in fully closed-loop nuPlan simulation.

Why This Matters

Impact on research: The paper shows that a single learned traffic prior can serve as both a planner and an evaluator, unifying two usually separate lines of work. It also challenges the assumption that higher benchmark scores imply greater robustness — a finding that should reshape how planners are evaluated.

Real-world applications:

• Stress-testing AV planners against rare, high-risk interactions (aggressive cut-ins, lead-vehicle braking) without needing to log them on real roads.
• Regulatory and safety validation, where closed-loop adversarial simulation can substitute for some high-risk physical testing.
• Data-efficient planning pipelines, useful for AV companies that cannot collect millions of expert trajectories.
• Controllable scenario authoring, letting engineers specify desired interaction types via interpretable energy functions rather than hard-coded scripts.

Industry relevance: nuPlan is an industry-standard benchmark, and the paper's finding that state-of-the-art learned planners resort to reactive braking rather than proactive evasion is directly actionable for AV developers. The dual-use framework also fits naturally into existing simulation toolchains.

Future Directions

• Extending DAPSE to other energy functions (e.g., explicit collision-avoidance objectives) and quantitatively evaluating its effect on collision rate and TTC.
• Scaling beyond 1M training scenarios and running multi-seed benchmarks to resolve the mixed SSDS-DP vs. DP results at large data.
• Addressing the selection-bias concern in the 25 adversarial scenes and building automated, criteria-driven scenario mining.
• Improving planners' ability to perform proactive lateral evasive maneuvers, since reactive braking alone fails in intersection conflicts.

Target Audience

AV researchers and engineers working on motion planning, diffusion-based generative models, or closed-loop simulation; safety and validation teams at autonomous-driving companies; and graduate students with background in deep generative models, transformers, and trajectory prediction who want to understand how inference-time guidance can double as both a planning tool and an adversarial test generator.

Authors’ abstract

Diffusion probabilistic models can capture the multi-modal, interaction-rich distribution of joint future trajectories in driving scenes. We show that a single pretrained diffusion traffic model can serve two complementary roles in the autonomous driving development loop: as an ego motion planner, and as a controllable generator of safety-critical scenarios for stress-testing the planners. On the planning side, we introduce a Single-Stream Dual-Stream (SSDS) diffusion-transformer decoder that fuses scene context via joint attention rather than late cross-attention, improving closed-loop performance on nuPlan. We further propose Decoupled Annealing Posterior Sampling with Energy (DAPSE), a training-free guidance scheme that injects arbitrary energy functions at the clean-sample level, avoiding the first-order approximation errors while requiring no auxiliary networks. Beyond planning, we leverage the same diffusion model as a controllable scenario generator to create realistic long-tail driving interactions for closed-loop evaluation. Through inference-time guidance, selected agents are steered toward safety-critical behaviors, including aggressive cut-ins, lead-vehicle braking, and combined longitudinal-lateral interactions, while preserving realistic traffic behaviors. Evaluated in closed-loop nuPlan simulations with independent black-box planners, the generated scenarios expose failure modes that remain hidden under standard benchmarks. Although the SSDS-based planner achieves stronger nominal performance, it experiences larger degradation under these challenging scenarios, demonstrating that benchmark superiority does not necessarily translate to robustness. These results demonstrate that a single learned traffic prior can simultaneously improve motion planning and provide a realistic framework for systematic planner robustness evaluation.