Embodied Cognition Augmented End2End Autonomous Driving

AI summary

Overview

Research area: Robotics / autonomous driving, specifically vision-based end-to-end (E2E) planning, combined with EEG-based brain-computer interfaces and embodied cognition.

Technical level: Advanced. The paper assumes familiarity with Bird's Eye View (BEV) representations, contrastive learning, transformer decoders, and EEG signal preprocessing.

Scope: The paper introduces a paradigm called E³AD that uses contrastive learning between a video feature extractor and a frozen large EEG model (LaBraM) to transfer human driving cognition into existing end-to-end driving planners, then evaluates that transfer on nuScenes (open-loop) and Bench2Drive/CARLA (closed-loop).

What This Paper Is About

Existing vision-based end-to-end driving models learn their visual representations from manually annotated labels tied to downstream subtasks such as 3D perception, motion prediction, and semantic segmentation. The authors argue this supervision is too narrow compared with how human drivers use embodied reasoning to anticipate hazards and adapt to new situations. Their goal is to give the feature extraction network a broader, human-derived learning signal by aligning video features with the brain activity recorded from real drivers, and then to test whether that added "driving cognition" actually improves planning.

Key Contributions

1. A new paradigm, E³AD, that performs contrastive learning between a spatio-temporal visual feature extraction network (the "Driving-Thinking Model") and the general EEG large model LaBraM, so the visual branch learns latent human driving cognition from paired video–EEG data. The authors state this is the first work to integrate human driving cognition for improving end-to-end autonomous driving planning.
2. A self-collected multimodal embodied cognitive dataset. Data were gathered from 27 subjects driving a fixed route in complex traffic, recording CAN bus data, EEG, heart rate, skin conductance, and front-facing camera footage. After filtering and variable control, 20 male drivers (10 experts, 10 novices) were analyzed, split 80:10:10 into 1894 training clips (1037 expert, 857 novice), 236 validation clips (144 expert, 92 novice), and 237 test clips (118 expert, 119 novice), cut into 2 s clips.
3. Three integration frameworks with corresponding hypotheses for how driving cognition might enhance planning: attaching cognition to spatio-temporal (BEV) features, interacting it with the ego query, and interacting it with planning features. Each represents a hypothesis at an increasing level of cognitive difficulty and complexity.
4. A two-stage training design that keeps inference fair. EEG is used only in stage one; in stage two and at inference only the mainstream driving datasets and the same inputs as baseline models are used, with the Driving-Thinking Model frozen. The authors report substantial planning gains at only a tiny increase in computational cost.

Main Findings

• Open-loop collisions drop on nuScenes. In Table 1 the paper reports that average collision rates of UniAD and VAD-Base decreased by 0.08% (25.8% relatively) and 0.04% (18.2% relatively), and convergence L2 errors were reduced by 0.08 m (11.1% relatively) and 0.05 m (6.7% relatively).
• Concrete open-loop numbers. E³AD(VAD-Base) reaches 0.64 m average L2 and 0.18% average collision (compared with VAD-Base at 0.72 m and 0.22%); E³AD(UniAD) reaches 1.03 m average L2 and 0.23% average collision (versus UniAD at 1.03 m and 0.31%); E³AD(VAD-Tiny) reaches 0.68 m and 0.35% (versus VAD-Tiny at 0.78 m and 0.38%).
• Gains transfer beyond autoregressive planners. E³AD(GenAD) reaches 0.47 m average L2 and 0.24% average collision under ST-P3 metrics and 0.87 m / 0.53% under UniAD metrics, improving on both the GenAD official checkpoint (0.49 m / 0.26%; 0.91 m / 0.56%) and the authors' reproduction (0.49 m / 0.30%; 0.90 m / 0.60%). E³AD(LAW) keeps average L2 at 0.61 m while lowering average collision from 0.30% to 0.22%.
• Closed-loop gains on Bench2Drive. E³AD(UniAD-Base) scores DS 50.07 and SR 20.12% with 0.69 m average L2, versus UniAD-Base at DS 45.81, SR 16.36%, 0.73 m. E³AD(VAD) scores DS 47.63 and SR 19.54% with 0.86 m average L2, versus VAD at DS 42.35, SR 15.00%, 0.91 m. The paper describes the route completion changes as 3.76% (23.0% relatively) and 4.54% (30.3% relatively), and the driving score increases as 4.26 (10.1% relatively) and 5.28 (12.5% relatively). AD-MLP is listed as a comparison at 3.64 m, DS 18.05, SR 0.00%, and UniAD-Tiny at 0.80 m, DS 40.73, SR 13.18%.
• Contrastive learning with LaBraM is the source of the benefit. The ablation in Table 3 shows that freezing the Driving-Thinking Model and running contrastive learning with both expert and novice EEG gives the best result (0.64 m average L2, 0.18% average collision). Using the Video Encoder alone as a pretrained feature extractor without contrastive learning gives 0.70 m and 0.25%, and unfreezing the Driving-Thinking Model in stage two gives 0.69 m and 0.25% — neither improves the baseline. The authors conclude the improvement comes neither from extra visual features nor from Video Swin Transformer capacity, but from the contrastive learning process with LaBraM.
• Expert EEG helps L2 most. In the ablation, using expert EEG alone yields 0.61 m average L2 and 0.21% average collision; using novice EEG alone yields 0.67 m and 0.21%; mixing both yields 0.64 m and 0.18% while training on a larger volume of EEG data.
• Higher-level cognition helps most. Among the three frameworks (Table 4), attaching to spatio-temporal features gives 0.69 m and 0.22%, interacting with the ego query gives 0.69 m and 0.20%, and interacting with planning features gives 0.64 m and 0.18%. The paper states the third framework achieved the greatest improvement and that the first did not yield significant improvements.
• L2 error can be misleading. Appendix C reports that doubling the L2 loss coefficient in VAD-Base's source code reduced L2 error by 0.15 m (20.8%) but increased the collision rate by 0.07% (31.8%), and that Ego-MLP, which plans using only the vehicle's historical state with no visual input, achieves L2 error comparable to VAD. The authors therefore treat L2 error as an indicator of model convergence rather than the goal.
• Inference cost stays low. E³AD(VAD-Tiny) runs at 8.8 FPS measured on the authors' NVIDIA A100 versus 9.8 FPS for VAD-Tiny on the same GPU; E³AD(VAD-Base) is 3.7 FPS versus 3.8; E³AD(GenAD) is 6.6 versus 6.7; E³AD(LAW) is 16.1 versus 16.5.

Methodology in Plain English

The work proceeds in two stages.

Stage one — teaching a video network to think like a driver. The authors paired short video clips from real drives with the EEG recorded from the driver at the same moment. A video branch (a Video Swin Transformer plus a learnable linear projection head) and a frozen large EEG model, LaBraM, both produce embeddings that are L2-normalized. The two embeddings are compared with a learnable temperature, and the whole thing is trained with a symmetric InfoNCE loss in the style of CLIP. Only the video branch's parameters are updated; LaBraM stays frozen. In effect, the video network is pushed to produce features that match the brain's response to the same moment — the "Driving-Thinking Model."

Stage two — plugging that cognition into existing planners. The Driving-Thinking Model is frozen and inserted into popular E2E-AD architectures without changing their inputs. No EEG is used from this point on, including at inference, so the comparison to baselines stays fair. Three insertion points were tried: (1) using cognition as an attention gate over BEV features to select brain-aware sparse visual representations, which are then cross-attended with the ego query and concatenated with the baseline's planning features before the planning head; (2) using cognition to enrich the ego query via cross-attention before it interacts with the baseline's supervised spatio-temporal features; and (3) using a transformer decoder with learnable positional encodings to let cognition reason over the preliminary planning features directly. In all cases a high-level driving command is fed to the planning head to produce the predicted trajectory.

Data preprocessing. EEG was re-referenced to M1 and M2, band-pass filtered at 0.1–50 Hz, notch filtered at 50 Hz, cleaned with Independent Component Analysis to remove ocular, cardiac, channel and muscle artifacts, downsampled from 1000 Hz to 200 Hz, and normalized by dividing amplitudes (±0.1 mV) by 0.1 mV. Continuous sessions were manually split into 14 condition-specific segments and cut into 2 s clips.

Training details. Video Swin Transformer extracts features from 2 s clips at 2 fps; both branches start from pretrained weights and are fine-tuned with the CLIP-style objective. Each branch's output passes through a two-layer MLP adapter into a shared 200-dimensional space. Training used aligned 2 s video–EEG pairs, batch size 16, 120 epochs, Adam with learning rate 2e-5 on both backbones and adapters (lr ratio 1:1), dropout 0.01, weight decay 1e-5, taking approximately 12 h on a single NVIDIA A100 40 GB GPU. Open-loop testing used nuScenes (1,000 urban scenes: 700 train, 150 validation, 150 test, ~20 s each, keyframes at 2 Hz, six cameras with 360° horizontal FOV plus LiDAR, radar and IMUs) with L2 displacement error and collision rate; closed-loop testing used Bench2Drive under CARLA Leaderboard 2.0 with the base set of 1000 clips and the official 220 routes.

Why This Matters

Research impact. The paper offers a route to supervise driving models with signals that are not hand-annotated labels, and it argues that the benefit comes from the brain-derived feature space rather than from extra visual capacity. It connects the EEG foundation-model literature (LaBraM, pretrained on 2,500+ hours of EEG and setting state of the art on downstream BCI tasks such as anomaly detection and emotion recognition) to autonomous driving planning, and it proposes three testable hypotheses about how cognition enters a planner, with the strongest effect from the highest-level intervention.

Real-world applications:

• Safer trajectory planning in complex urban traffic, where collision rate rather than L2 error is the metric that matters.
• Data-efficient training of autonomous driving stacks that need less hand annotation for BEV and planning features.
• Human-cognition-informed driver monitoring and advanced driver assistance, using physiological signals such as EEG, heart rate and skin conductance as a window into driver state.
• Brain-inspired model design more broadly, including embodied AI and human-intelligence augmentation in other robotics domains.

Industry relevance. The method is designed as a drop-in addition to existing end-to-end stacks: it is applied to VAD-Tiny, VAD-Base, UniAD, GenAD and LAW, it requires no extra inputs at inference, and it costs only about 0.2–1.0 FPS in the reported measurements. The work was supported by the Beijing Natural Science Foundation (L233033), the Beijing Municipal Science and Technology Project (Nos. Z231100010323005), and funding from Horizon Robotics, and code is promised at https://github.com/AIR-DISCOVER/E-cubed-AD.

Future Directions

• Expand the EEG–video dataset. The authors identify the paired dataset as relatively small because of the cost and time of EEG acquisition and processing, and commit to publicly releasing the dataset and data collection procedures to facilitate expansion.
• Explain the mechanism. The paper states that the precise mechanisms by which EEG alignment enhances planning have not been fully explored, and that future work will investigate the models' underlying mechanisms.
• Scale to more subjects and drivers. Only 20 male drivers (10 experts, 10 novices) were analyzed after filtering, so testing whether expert-versus-novice effects and the expert/novice mixing result hold on larger and more varied pools remains open.
• Generalize the intervention further. The three frameworks were validated on VAD-Base for framework comparison, with broader model coverage in the main tables; determining which framework suits which planner architecture, and whether cognition should be injected at other points in the pipeline, is left open.

Target Audience

Researchers and engineers working on end-to-end autonomous driving and BEV-based planners; brain-computer interface and EEG foundation-model researchers interested in real-world driving cognition; and robotics or embodied-AI practitioners exploring brain-inspired supervision, human-intelligence augmentation, or cognitive datasets for training complex autonomous systems.

Authors’ abstract

In recent years, vision-based end-to-end autonomous driving has emerged as a new paradigm. However, popular end-to-end approaches typically rely on visual feature extraction networks trained under label supervision. This limited supervision framework restricts the generality and applicability of driving models. In this paper, we propose a novel paradigm termed $E^{3}AD$, which advocates for comparative learning between visual feature extraction networks and the general EEG large model, in order to learn latent human driving cognition for enhancing end-to-end planning. In this work, we collected a cognitive dataset for the mentioned contrastive learning process. Subsequently, we investigated the methods and potential mechanisms for enhancing end-to-end planning with human driving cognition, using popular driving models as baselines on publicly available autonomous driving datasets. Both open-loop and closed-loop tests are conducted for a comprehensive evaluation of planning performance. Experimental results demonstrate that the $E^{3}AD$ paradigm significantly enhances the end-to-end planning performance of baseline models. Ablation studies further validate the contribution of driving cognition and the effectiveness of comparative learning process. To the best of our knowledge, this is the first work to integrate human driving cognition for improving end-to-end autonomous driving planning. It represents an initial attempt to incorporate embodied cognitive data into end-to-end autonomous driving, providing valuable insights for future brain-inspired autonomous driving systems. Our code will be made available at Github