ExpertAD: Enhancing Autonomous Driving Systems with Mixture of Experts

AI summary

Overview

Research area: End-to-end autonomous driving systems (ADSs) — specifically, applying Mixture of Experts (MoE) architectures across the perception and prediction stages of a modular driving pipeline.

Technical level: Intermediate. Readers should be comfortable with terms like BEV (bird's-eye-view) features, transformers, attention, queries, and the standard perception–prediction–planning pipeline.

Scope: The paper proposes ExpertAD, an MoE-based framework layered onto existing end-to-end ADSs, and reports open-loop (nuScenes), closed-loop (Bench2Drive), ablation, generalization, and qualitative results across three baselines.

What This Paper Is About

End-to-end ADSs still struggle with three things: ambiguous or noisy scene semantics that can mislead decisions, interference between multiple driving tasks that can hurt planning, and inference latency that is slow enough to matter for safety. ExpertAD tackles all three at once by inserting a Perception Adapter that amplifies task-critical BEV feature channels, and by replacing the prediction module with a Mixture of Sparse Experts that activates only the driving tasks relevant to the current scene.

Key Contributions

1. Perception Adapter (PA). A module combining a Learned Adapter (which selects a subset of task-specific BEV feature dimensions using a constrained optimization with an entropy regularizer) and an Alignment Layer (which reweights and amplifies those features before they reach the tracking and mapping transformers).
2. Mixture of Sparse Experts (MoSE). A prediction module with eight experts in three families — Environmental (Tracking, Mapping), Ego State (Velocity, Yaw, Acceleration), and Navigation (Reference Point, BEV, Command) — each using a different sparse-attention pattern, plus a Router that activates only the top-k experts for the current driving context.
3. The ExpertAD framework itself, which spans perception and planning rather than being confined to the planning module, and which the authors contrast with prior planning-only MoE work such as ARTEMIS.
4. Large-scale integration and evaluation, applying ExpertAD to three vision-only baselines — UniAD, VAD, and VADv2 (yielding Expert-UniAD, Expert-VAD, Expert-VADv2) — on both open-loop and closed-loop datasets.

Main Findings

• Joint effectiveness and efficiency gains across all three baselines. The abstract states ExpertAD reduces average collision rates by up to 20% and inference latency by 25%. Table 1 gives the specific per-model results: Expert-UniAD drops average collision rate from 0.31 to 0.24, average L2 from 1.03 to 0.89, DS from 44.62 to 55.49, and latency from 534 ± 18 ms to 445 ± 20 ms; Expert-VAD goes from 0.43 to 0.34 collision rate and 225 ± 25 ms to 157 ± 23 ms; Expert-VADv2 goes from 0.12 to 0.10 collision rate and 330 ± 18 ms to 258 ± 22 ms. Latency in Fig. 1 is measured on an NVIDIA GeForce RTX 3090; Table 1 notes performance measured on a single RTX 3090.
• Reported relative improvements. The paper states ExpertAD achieves a 23% collision-rate reduction, 14% L2 reduction, and 1.2× speedup over UniAD; 21% and 17% collision-rate reductions, 9% and 15% L2 reductions, and 1.4× and 1.3× speedups for Expert-VAD and Expert-VADv2. In closed-loop evaluation it reports DS, SR, and RC improvements of 16%, 22%, and 14% averaged over the three ADSs.
• Parameter cost is not zero. Params rise from ~89M to ~125M (UniAD), ~58M to ~90M (VAD), and ~76M to ~105M (VADv2), while GFLOPs fall from ~856 to ~728, ~558 to ~461, and ~660 to ~573 respectively. The authors describe this as "minimal parameter overhead."
• Multi-skill scenario results (Bench2Drive220, Table 2). ExpertAD improves over baselines in Merge, Overtake, and T-sign groups and in Emergency Braking, while results for Giving Way are weaker — Expert-VADv2 scores 40.00 on GiveWay versus VADv2's 50.00. The table caption says performance is "comparable" in EmgBrake and Giveway, while the body text says improvements are only marginal or inconsistent in Overtaking and Giving Way scenarios demanding human-like reasoning. The paper states that these scenarios suggest rule-based fallback systems are still needed.
• PA hyperparameter sweep (Table 3). Performance peaks at τ = 128 selected BEV features (DS 52.53, SR 18.41, RC 76.73), with τ = 32 (DS 40.35) falling below the UniAD baseline of 44.62 and τ = 256 (DS 43.31) degrading relative to τ = 128. Gains at τ = 128 over baseline are reported as 17% DS, 30% SR, 11% RC.
• MoSE expert count (Table 4). Top-8 (all experts) gives DS 46.24, SR 17.27, RC 67.54 versus baseline 44.62/14.09/68.68; Top-4 gives 49.32/18.41/72.03. The paper reports Top-4 improves DS, SR, and RC by 11%, 30%, and 5%.
• Ablation of PA components (Table 5). On UniAD tracking, baseline AMOTA is 0.388 and AMOTP 1.304; MLP-only gives 0.384/1.306; ADD-only gives 0.390/1.298; both together give 0.404/1.277.
• Ablation of MoSE components (Table 6). For Expert-UniAD, adding both Router and Sparse Attention reduces latency by 178 ms (623 ± 35 ms to 445 ± 20 ms) while improving average L2 by 0.09 m and average collision rate by 0.03%. For Expert-VAD, the corresponding figures are 132 ms latency reduction with gains of 0.02 m and 0.02%. Router alone improves planning metrics; Sparse Attention alone mainly cuts latency.
• Cross-city generalization (Table 7). Expert-UniAD reduces collision rates by 17% when trained on Boston and tested on Singapore; trained on Singapore and tested on Boston, it achieves 13% lower L2 errors and 14% fewer collisions. Bolded best values are not printed in the extraction for both directions, so only the textual claims above are reported.
• Qualitative case study (Fig. 3). UniAD misses a traffic officer on the right and plans a path drifting toward them; Expert-UniAD detects the officer, adjusts the trajectory, and completes a lane change safely.
• Statistical significance. Paired t-tests give an average p-value of 0.026 (p < 0.05) across benchmarks and metrics; all results are averaged over five independent runs.

Methodology in Plain English

The researchers start from an existing end-to-end ADS and keep everything except the perception and prediction modules, which they replace.

For perception, they first pool BEV features along time to get a frame-agnostic representation. For each perception task (tracking, mapping), a task-specific learnable vector scores how important each feature channel is. A small optimization — maximize the score-weighted selection while regularizing with entropy, under the constraint that the weights sum to a fixed number τ — produces soft per-channel selection weights. Those weights multiply the BEV features, pass through an MLP, and are added back to the original BEV features. This residual addition preserves spatial information and creates a gradient shortcut. Tracking and mapping transformers then consume the recalibrated features, and their outputs are concatenated with a learnable embedding into a single "ego query."

For prediction, that ego query drives a Router. The Router computes expert logits with a learned gating matrix plus Gaussian noise scaled by a softplus term (noise is added only during training, to prevent overfitting), applies softmax, and takes the top-k experts. Each of the eight experts attends over the ego query using a sparse-attention pattern suited to its role: block-wise local attention for environmental experts, sliding-window attention for ego-state experts, and global TopK attention for navigation experts. Expert embeddings carry modality-specific information — text features for navigation commands, geometric features for environment, time series for vehicle dynamics. The top-k expert outputs are combined by their routing scores into a motion query that goes to the planning module.

Training uses the baseline's own perception, prediction, and planning losses, plus a switch-loss that penalizes experts receiving a disproportionate share of routing probability, so that all experts get used. The weighting coefficients α₁–α₄ in the combined loss are not given numerical values in the paper, nor are the block size m, the sliding window size w, or the number of selected experts used outside the ablation.

Why This Matters

The paper's argument is that speed and safety are usually traded against each other in end-to-end ADSs — prior work such as DriveAdapter and PlanKD reduces latency at the cost of planning quality — and that MoE can improve both simultaneously. It also argues that most prior MoE work in driving is confined to the planning module or to narrow subtasks, and is sensitive to perception and planning noise. ExpertAD's distinctive move is to place the MoE mechanism across both perception and prediction, and to use task-specific sparse-attention patterns rather than shared low-rank experts.

Real-world applications:

• Deployed autonomous vehicle stacks, where a conventional end-to-end model is often retained as a fallback even when LLMs or world models are used. The paper notes this explicitly as motivation.
• Latency-constrained embedded driving compute, since the reported latency reductions are measured on a single NVIDIA GeForce RTX 3090 rather than a data-center GPU.
• Urban driving in unfamiliar cities, given the cross-city Boston/Singapore generalization tests.
• Rare-scenario handling such as emergency braking, merging, and traffic-sign compliance, where the paper reports the largest scenario-group gains.

Industry relevance: The framework is designed as a drop-in addition to existing transformer-based end-to-end ADSs, and the paper demonstrates this by integrating it with three publicly available baselines, which lowers the barrier for adoption by teams already running UniAD, VAD, or VADv2-style pipelines.

Future Directions

1. Close the reasoning gap in overtaking and giving-way scenarios. The paper itself notes these demand complex, human-like reasoning and that rule-based fallback systems remain necessary — a natural target for future work.
2. Combine ExpertAD with LLMs or world models. The paper frames ExpertAD as enhancing the end-to-end model that currently serves as a fallback behind LLMs and world models; how the two layers would interact is not explored.
3. Reduce the parameter overhead. Parameters increase for every baseline (e.g., ~89M to ~125M for UniAD), described as minimal, but whether this can be trimmed further is an open question.
4. Explain and stabilize routing. The paper notes that minor input variations in dynamic scenes can destabilize expert activation, motivating its noise-injected gating; how well this holds under distribution shift beyond the Boston/Singapore tests is not reported.
5. Determine the right number of active experts in general. The ablation compares Top-4 and Top-8 for UniAD only; a principled rule for choosing k, and for the loss weights and attention hyperparameters left unspecified, is not established.

Target Audience

Researchers and engineers working on end-to-end autonomous driving, particularly those interested in efficiency–accuracy trade-offs, modular perception–prediction–planning pipelines, or applying Mixture-of-Experts outside of language and vision foundation models. It is also relevant to practitioners deploying driving models under latency constraints, and to readers tracking how MoE ideas transfer from LLMs into robotics and safety-critical control. Some familiarity with BEV encoders, transformer queries, and nuScenes-style evaluation metrics is assumed.

Authors’ abstract

Recent advancements in end-to-end autonomous driving systems (ADSs) underscore their potential for perception and planning capabilities. However, challenges remain. Complex driving scenarios contain rich semantic information, yet ambiguous or noisy semantics can compromise decision reliability, while interference between multiple driving tasks may hinder optimal planning. Furthermore, prolonged inference latency slows decision-making, increasing the risk of unsafe driving behaviors. To address these challenges, we propose ExpertAD, a novel framework that enhances the performance of ADS with Mixture of Experts (MoE) architecture. We introduce a Perception Adapter (PA) to amplify task-critical features, ensuring contextually relevant scene understanding, and a Mixture of Sparse Experts (MoSE) to minimize task interference during prediction, allowing for effective and efficient planning. Our experiments show that ExpertAD reduces average collision rates by up to 20% and inference latency by 25% compared to prior methods. We further evaluate its multi-skill planning capabilities in rare scenarios (e.g., accidents, yielding to emergency vehicles) and demonstrate strong generalization to unseen urban environments. Additionally, we present a case study that illustrates its decision-making process in complex driving scenarios.