Pseudo-Expert Regularized Offline RL for End-to-End Autonomous Driving in Photorealistic Closed-Loop Environments

AI summary

Overview

Research area: End-to-end autonomous driving (camera-only trajectory prediction), offline reinforcement learning, and closed-loop simulation with neural rendering.

Technical level: Advanced. The paper assumes familiarity with actor-critic RL, offline RL failure modes (Q-value overestimation on out-of-distribution actions), behavior cloning, and bird's-eye-view (BEV) driving architectures.

Scope: The authors propose and evaluate a camera-only end-to-end offline RL framework for driving — trained only on a fixed dataset generated in the NeuroNCAP simulator — that uses pseudo-expert trajectories derived from ground-truth nuScenes logs as a regularization signal.

What This Paper Is About

Most end-to-end driving models are trained by imitation learning (IL), which breaks down when the vehicle encounters states absent from the expert data (covariate shift) or when the model latches onto spurious cues (causal confusion). Online RL could address this, but photorealistic neural rendering plus large end-to-end networks make iterative reward and hyperparameter tuning prohibitively expensive. The paper's goal is to show that a purely offline RL framework — no new simulator interaction during training — can beat IL baselines on safety while remaining stable, provided the policy is regularized toward clean expert-like trajectories rather than the potentially unsafe dataset behavior policy.

Key Contributions

1. A camera-only, end-to-end offline RL framework for autonomous driving that learns from a fixed, pre-collected dataset without any online environment interaction, avoiding the cost of exploration in neural rendering simulators.

2. A pseudo-expert regularization technique. Instead of cloning the dataset's behavior policy (which may include collisions or suboptimal maneuvers), the method builds synthetic expert-like actions by finding the two nearest ground-truth waypoints to the ego-vehicle's position, linearly interpolating their expert trajectories, and selecting the closest discrete action prototype in the vocabulary. This signal is added as a behavior-cloning term in the actor loss.

3. A closed-loop empirical comparison against IL baselines (UniAD, VAD, SparseDrive, VADv2) plus ablations on the regularization weight, reward components, and behavior policy composition, all conducted in the NeuroNCAP simulator built on nuScenes.

4. A detailed empirical analysis of behavior policy composition, showing that the choice and mixture of policies used to generate the offline dataset strongly shapes whether the learned policy ends up "efficiency-focused" or "safety-focused."

Main Findings

• Safety-critical collision rate improves substantially over IL. The main model (VADv2*, trained on a VAD(σ=0.2) + VAD(σ=0.4) mix) reaches a 29.9% collision rate on the Safety-Critical suite, versus 38.2% for the best IL baseline (VADv2 w/ Expert BC) and 83.0% for UniAD. Only the standard-BC offline RL variant scores lower on that single metric (17.9%), but it is far worse elsewhere.

• Best unified trade-off scores. VADv2* attains the highest SRC (37.0) and highest JSR (34.3) of all compared methods, with 51.1% general-driving CR, 52.8% route completion, 0.75 longitudinal jerk and 0.27 lateral jerk.

• Standard behavior cloning in the offline RL loop collapses route completion. The VADv2 (w/ std. BC) variant records only 16.0% RC in general driving — described as catastrophically over-conservative — alongside a 17.9% safety-critical CR and an SRC of 13.1.

• A random-policy dataset produces an unsafe but efficient policy. VADv2† (trained on VAD(σ=0.2) + Random) achieves 72.0% RC but a 74.7% safety-critical CR, with SRC 18.2 and JSR 16.3.

• Removing regularization causes instability. With α = 0.0 the model collapses to 21.6% RC, 87.3% safety-critical CR, SRC 2.7 and jerk 1.67. Very strong regularization (α = 1.0) also hurts (48.8% CR, SRC 23.3). The paper reports optimal performance for α in [0.1, 0.4], with α = 0.1 giving the best SRC (37.0).

• Reward shaping matters in both directions. Dropping the dense imitation reward (w_imitation = 0) drops RC from 46.5% to 26.5%. A weak terminal penalty (C_event = −5) yields 46.2% safety-critical CR; an overly harsh one (C_event = −20) yields 34.5% RC with 37.2% CR. The paper states w_imitation = 0.1 and C_event = −10 give the best trade-off.

• Dataset composition dominates the safety-efficiency trade-off. Isolated behavior policies score poorly on SRC: VAD(σ=0.2) 22.4, VAD(σ=0.4) 21.7, Random 17.3, VAD(σ=0.2)+Random 18.2, VADv2(σ=0.2) 9.5. The VAD(σ=0.2)+VAD(σ=0.4) blend is best by far on SRC (33.5) and safety-critical CR (28.1%).

• The authors attribute these personalities to Q-value shaping. With high-randomness data, "proceed straight" looks clearly superior to implausible alternatives, so Q-values favor a go-focused policy; with low-randomness data, common actions are repeatedly associated with collisions, so Q-values become pessimistic and the policy favors stopping.

• Jerk is worse than IL baselines. The paper states this is expected because the reward function includes no explicit jerk penalty.

• Qualitative behavior differs by dataset. In a scooter cut-in scenario and a blocked-lane scenario, the main model stops and waits or stops behind the obstacle, while the IL baseline and the random-data ablation collide or swerve.

Methodology in Plain English

The team starts from an existing camera-only end-to-end driving model (VADv2) that outputs a probability distribution over a finite vocabulary of 4096 trajectory prototypes, each produced by k-means clustering of expert nuScenes trajectories. The network has an encoder for camera and proprioceptive input, a BEV decoder that converts perspective features into a bird's-eye-view map using deformable cross-attention, an actor head over the discrete action set, and a critic head that outputs Q-values for each action from the same action features.

Instead of letting the car explore, they first use behavior policies — IL-trained VAD and VADv2 models with added Gaussian waypoint noise at σ ∈ {0.1, 0.2, 0.4}, plus a uniform random policy — to drive around the NeuroNCAP simulator and record a fixed dataset containing successes, collisions, and off-route events.

Training then proceeds with an actor-critic objective. The critic minimizes a one-step temporal-difference error against target networks updated by exponential moving average. The actor maximizes advantage-weighted log-probabilities of its own sampled actions, but a crucial extra term pulls the policy toward a pseudo-expert action rather than the recorded one: for each training state, they locate the two ground-truth expert waypoints nearest the ego-vehicle, interpolate their trajectories into a synthetic reference, and pick the discrete prototype closest to that reference. The reward combines a dense imitation term (negative squared distance between the executed action and this pseudo-expert action) with a sparse terminal penalty of −10 for collisions, off-road, and off-route events.

Evaluation is fully closed-loop in the NeuroNCAP simulator, which uses the NeuRAD neural rendering engine and is built from the public nuScenes dataset, across 137 general-driving scenes from the nuScenes validation split and 20 NeuroNCAP safety-critical scenes.

Why This Matters

Impact on research. This is presented as among the first systematic evaluations of a purely offline, camera-only end-to-end RL driving policy in closed-loop form, and the authors state that to their knowledge it is the first application of offline RL to end-to-end autonomous driving. It also raises behavior-policy selection as an open problem in offline RL, showing empirically that dataset composition — not just algorithm choice — determines whether the resulting policy is cautious or aggressive.

Real-world applications:

• Fleet data reuse: offline RL lets a manufacturer squeeze additional policy improvement out of driving logs already collected, without new on-road or simulator data collection.
• Safe policy iteration before deployment: a repaired policy can be validated in closed-loop simulation before any vehicle is put at risk.
• Cost reduction in development cycles: neural rendering simulation and E2E reward/hyperparameter sweeps are expensive; a fixed-dataset pipeline shortens the loop.
• Scenario-specific tuning: the behavior policy analysis offers a practical lever for tuning a policy toward safety-critical performance or route-completion performance depending on the deployment context.

Industry relevance. The work comes from Toyota Motor Corporation's InfoTech group, and the paper points out that online RL in E2E driving (e.g., RAD with 3D Gaussian Splatting, ReCogDrive with NAVSIM and GRPO) demands large in-house data and heavy computation. Offline RL is framed as the pragmatic path for an automaker that already has abundant logs but limited tolerance for exploration cost. Code is released at https://github.com/ToyotaInfoTech/PEBC.

Future Directions

• Principled behavior policy selection. The authors explicitly state they do not yet have comprehensive criteria for choosing which policies and mixtures should generate the offline dataset, and identify this as a critical underexplored question.
• Systematic construction of pseudo-expert trajectories. The current interpolation-from-nearest-GT-waypoints procedure is empirically defined; the paper calls for a more systematic approach.
• Comfort-aware reward design. The learned policies have higher jerk than IL baselines because no jerk penalty was included in the reward; adding one is a natural extension.
• Generalizing beyond the NeuroNCAP setup. All training and evaluation use one simulator built on nuScenes; whether the pseudo-expert regularization transfers to other simulators, other datasets, or real vehicles is not reported.

Target Audience

Researchers and engineers working on offline reinforcement learning, end-to-end autonomous driving, and closed-loop simulation — particularly those interested in safety-critical policy improvement from existing driving logs. It is also relevant to practitioners at automakers or autonomy companies who already hold large driving datasets and want to improve policies without funding large-scale online exploration, and to RL researchers studying behavior regularization and dataset composition. Readers without background in actor-critic methods and BEV architectures will find the training objective and metrics demanding.

Authors’ abstract

End-to-end (E2E) autonomous driving models that take only camera images as input and directly predict a future trajectory are appealing for their computational efficiency and potential for improved generalization via unified optimization; however, persistent failure modes remain due to reliance on imitation learning (IL). While online reinforcement learning (RL) could mitigate IL-induced issues, the computational burden of neural rendering-based simulation and large E2E networks renders iterative reward and hyperparameter tuning costly. We introduce a camera-only E2E offline RL framework that performs no additional exploration and trains solely on a fixed simulator dataset. Offline RL offers strong data efficiency and rapid experimental iteration, yet is susceptible to instability from overestimation on out-of-distribution (OOD) actions. To address this, we construct pseudo ground-truth trajectories from expert driving logs and use them as a behavior regularization signal, suppressing imitation of unsafe or suboptimal behavior while stabilizing value learning. Training and closed-loop evaluation are conducted in a neural rendering environment learned from the public nuScenes dataset. Empirically, the proposed method achieves substantial improvements in collision rate and route completion compared with IL baselines. Our code is available at https://github.com/ToyotaInfoTech/PEBC.