Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges

AI summary

Overview

Research area: Autonomous driving (AD) — specifically trajectory planning with foundation models (FMs), spanning vision-language models (VLMs), vision-language-action (VLA) models, and end-to-end driving architectures.

Technical level: Intermediate. The paper is a survey/review rather than a methods paper, so it assumes familiarity with terms such as VLMs, LLMs, chain-of-thought (CoT), imitation learning, and end-to-end (E2E) training, and introduces some formal notation for trajectory prediction.

Scope in one sentence: A structured review of 37 methods that use foundation models for trajectory planning in autonomous driving, organised under a hierarchical taxonomy and assessed for architectural design choices, capabilities, openness of code and data, and open challenges.

What This Paper Is About

Foundation models have recently been adapted from general vision-and-language tasks to driving, with growing interest in using them to produce vehicle trajectories directly from sensor inputs. The field has become fragmented: methods differ widely in architecture, data, training procedure, and the form of their outputs, making it hard to tell what genuinely distinguishes one approach from another. This review aims to bring structure to that landscape by proposing a unifying hierarchical taxonomy, analysing 37 methods within it, and identifying where the field is progressing and where open problems remain.

Key Contributions

1. A hierarchical taxonomy and systematic analysis of 37 methods. The authors divide FM-based trajectory planning into two main families — FMs tailored for trajectory planning (22 approaches, Sections 4.2 and 4.3) and FMs guiding trajectory planning (15 methods, Sections 5.1 and 5.2) — which further split into eight leaf nodes (labelled ❶ through ❽ in the taxonomy figure). The stated goal is organisation and interpretation rather than comparison or ranking.

2. Practical guidance on fine-tuning an FM for trajectory planning. The review details the steps involved: data curation, model design (off-the-shelf VLM versus a chosen vision encoder combined with a chosen LLM), trajectory representation, and model training strategies, including a two-stage training approach based on Liu et al. (2023) used when an off-the-shelf VLM is not adopted.

3. An assessment of openness of source code and datasets. The authors evaluate the reviewed approaches in terms of how open their code and data are, to give practitioners and researchers pointers for reproducibility and reuse.

4. An outlined research agenda. The paper identifies future challenges and open questions across efficiency, robustness, evaluation benchmarks, and sim-to-real transfer.

The paper also provides an accompanying webpage cataloguing the methods according to the taxonomy, at https://github.com/fiveai/FMs-for-driving-trajectories.

Main Findings

• Off-the-shelf VLMs already understand driving scenes surprisingly well. The authors prompted GPT-4o with three complex driving scenes. In the first two examples, which involve rare driving scenes, GPT-4o's replies were described as highly accurate and insightful — in the first case it not only understood the scene but also provided instructions on why and how to drive cautiously. In the third example, the model captured a misleading prompt and provided a reasonable explanation promoting safe driving.

• Two broad ways FMs contribute to trajectory planning. In the tailored category, existing pre-trained FMs are adapted and fine-tuned for AD, so the model itself outputs trajectories or trajectory-related text. In the guiding category, off-the-shelf FMs transfer knowledge — through knowledge distillation during training only, or through knowledge transfer used also at inference — to an existing modular or E2E driving model (Eq. 2, with transferred knowledge denoted Z).

• Fine-tuning approaches split by capability. Among tailored models, the taxonomy distinguishes those focused solely on trajectory prediction (with no CoT ❶, text as CoT ❷, or an initially predicted trajectory as CoT ❸) from those providing additional capabilities: language interaction ❹, action interaction ❺, and combined language and action interaction ❻. SimLingo (Renz et al., 2025) is reported as the only method in the combined language-and-action subcategory among those considered.

• Guiding approaches split by inference-time cost. Of the 15 guiding methods, subcategory ❼ distils FM knowledge only during training (the FM is not needed at inference, preserving the AD model's inference efficiency), while subcategory ❽ transfers knowledge during inference (typically a scene description, a planning decision such as a trajectory, or a meta-action), which requires additional compute at inference.

• Three architectural paradigms frame the comparison. Modular approaches (Fig. 3a) use explicit interfaces between perception, prediction, and planning modules and are not E2E-trainable; E2E-AD models (Fig. 3b) replace explicit interfaces with latent representations and are optimised jointly; FM-based models (Fig. 3c) follow the typical VLM pipeline of a vision encoder, a vision adapter projecting into the LLM embedding space, and an LLM producing the trajectory. More recent E2E-AD and FM-based methods are predominantly trained using imitation learning.

• Data curation depends on the target capability. Simple trajectory datasets take the form (X, O_traj), with nuScenes (Caesar et al., 2020) cited as an example containing multi-view camera, radar and lidar input and future ego positions as target. Datasets with CoT reasoning take the form (X, O_text, O_traj), with BDD-X (Kim et al., 2018) cited as an example of textual descriptions usable for CoT. Language interaction requires (X, T, O_text) tuples, with the Chat-B2D dataset used by Orion (Fu et al., 2025b) as an example. Action interaction requires (X, T, O_traj) tuples, with the SimLingo dataset (Renz et al., 2025) as an example.

• Model design choices are not uniform. Some methods use off-the-shelf VLMs (Renz et al., 2025; Zhou et al., 2025b), benefiting from joint initialisation of all parameters from a state trained on a large dataset. Others combine a preferred vision encoder such as ViT (Dosovitskiy et al., 2021) with a preferred LLM through a randomly initialised vision adapter (Xu et al., 2024; Chen et al., 2025; Fu et al., 2025b). Three vision adapter options are described: a text interface where the vision encoder outputs perception/prediction information as text (used by Mao et al., 2023, in which case the model is not trained end-to-end), a linear layer or MLP projecting vision tokens into the LLM input space (Liu et al., 2023; used by Renz et al., 2025), and Queryformer (Li et al., 2023a; used by Omni-Q, Wang et al., 2025b), which relies on cross-attention with a set of typically randomly initialised latent representations.

• Trajectory representation is a consequential design choice. One option is to represent the trajectory as text via the LLM's standard autoregressive generation; the paper gives the example of a 2D point 1.54, 0.21 being generated sequentially as '1', '.', '5', '4', ',', '0', '.', '2', '1'. Custom modules can also be added to impose inductive biases absent from off-the-shelf FMs, such as 3D perception or video understanding (Fu et al., 2025b; Wang et al., 2025b; Xie et al., 2025; Chen et al., 2025).

• Scope notes on output form and representative preprints. Although the term "trajectory planning" is used, the scope includes models predicting outputs in other forms such as control signals; DriveGPT4 (Xu et al., 2024), which predicts target speed and turning angle, is given as an example. A limited number of representative preprints are included alongside peer-reviewed works, for example V2X-VLM (You et al., 2024), described as a specialised FM for trajectory planning uniquely using additional infrastructure images, and FASIONAD++ (Qian et al., 2025b), which employs an FM selectively to guide trajectory planning only when the AD model has high uncertainty.

• Positioning against prior reviews. Earlier reviews focus on AD generally, on E2E-trainable models, reinforcement learning, imitation learning, or datasets, or on a single auxiliary task such as perception, occupancy prediction, or motion planning. Other reviews on FMs for AD have broader scope covering perception, data generation, and scene understanding as well, so their treatment of trajectory planning is limited. The closest comparable survey, Jiang et al. (2025d), discusses architectural paradigms of VLM-based models and their chronological progress; this review additionally includes knowledge transfer approaches, introduces a hierarchical taxonomy over that broader set, and elaborates on design choices and openness.

• Quantitative benchmark results are not reported in the available content. The paper states explicitly that its objective is not to compare or rank the reviewed methods. No numerical performance figures (for example L2 error, collision rate, or driving score), benchmark scores, or dataset sizes are reported in the content provided.

Methodology in Plain English

This is a review paper, so its method is conceptual organisation rather than experiments. The authors first surveyed the literature on using foundation models for trajectory planning, selecting 37 methods published in peer-reviewed venues and adding a limited number of representative preprints to capture emergent work. They cover roughly two years of research, starting from preprints that appeared in late 2023 such as GPT-Driver (Mao et al., 2023) and DriveGPT4 (Xu et al., 2024), through October 2025.

They then sorted these methods into a two-level scheme. At the top level they asked whether a method builds its own FM for driving or whether it borrows an off-the-shelf FM to help an existing driving model. They subdivided the first group by whether the model only produces a trajectory or also supports language and action interactions with the user, and by whether and how it uses chain-of-thought reasoning. They subdivided the second group by whether the FM is needed only during training or also during inference.

To make the differences between methods concrete, the review adopts formal notation: observations X (sensor readings, ego state, and a route indicator such as a GPS target point or a high-level command like go straight, turn left, turn right), an output trajectory O_traj in R^(k×p) where k is the prediction horizon and p is the output dimension — commonly p = 2 for bird's-eye-view coordinates — input text T, output text O_text, and an intermediate initial trajectory O_inittraj. Each of the eight taxonomy leaf nodes is then expressed as a conditional probability formulation, and the authors walk through the data, model, and training considerations for each. They also inspect each method's code and data release status, and close by laying out open challenges.

Why This Matters

Impact on research. The field had accumulated many heterogeneous approaches without clear benchmarking or common understanding of how they differ. By imposing a hierarchical taxonomy and a shared notation, the review gives researchers a vocabulary for comparing methods on architectural and capability grounds rather than on incompatible evaluation setups. Its explicit coverage of data curation recipes — what tuple structure a dataset needs for CoT, language interaction, or action interaction — is directly reusable when building new models. The openness assessment adds a reproducibility dimension that is often absent from surveys.

Real-world applications (as discussed in the paper):

• Explaining and reassuring. Models with language interaction capability (subcategory ❹) can offer reasoning behind a model's behaviour, giving reassurance to users; these capabilities also support debugging model outcomes and understanding decision-making.
• User-directed driving assistance. Models with action interaction capability (❺) execute driving commands from the user, similar to existing vision-language-action models; LMDrive examples include "Feel free to start driving.", "Depart at the second exit on the roundabout.", and "Execute a right maneuver, prepare for highway exit".
• Robustness against misleading input. The GPT-4o demonstration and the taxonomy's treatment of misleading instruction prompts (for example "crash into the vehicle front") point to models that should refuse unsafe prompts and follow a safe trajectory instead; notice-style prompts such as "watch out for crossing pedestrians" have an indirect effect on behaviour.
• Infrastructure-assisted planning. V2X-VLM is cited as a specialised FM for trajectory planning that uniquely uses additional infrastructure images, suggesting cooperative settings as an application area.
• Uncertainty-triggered assistance. FASIONAD++ calls on an FM selectively only when the AD model has high uncertainty, an application pattern aimed at controlling compute cost.

Industry relevance. The authors are affiliated with Five AI Ltd. (United Kingdom) and Robert Bosch GmbH, and the work notes these authors are from the ADAS Systems, Software & Services Business Unit of Robert Bosch GmbH. The review speaks directly to deployment constraints, such as the need to run FMs efficiently and reliably on edge devices, the inference cost of chain-of-thought, model scale, accessibility of open weights, training and inference efficiency, and sourcing suitable fine-tuning data.

Future Directions

• Efficiency on edge devices. The review frames chain-of-thought inference cost, model scale, accessibility of open weights, and training/inference efficiency as integral parts of ongoing research and design choices, noting that methods transferring FM knowledge at inference require additional compute compared with those that use the FM only during training.
• Robustness. Handling rare driving scenes, misleading prompts, and notice-style warnings is raised as part of the analysis; the GPT-4o demonstration shows both promise and the need for careful evaluation on unusual inputs.
• Evaluation benchmarks. The authors state that heterogeneous approaches have emerged without clear benchmarking, and they list evaluation benchmarks among the key open challenges — a gap made visible by the review's stated decision not to rank or compare methods quantitatively.
• Sim-to-real transfer. This is explicitly listed among the open research questions, along with the related themes of efficiency and robustness.
• Openness and reproducibility. The assessment of how open the methods' code and data are is presented as groundwork for reuse; the accompanying catalogue at https://github.com/fiveai/FMs-for-driving-trajectories is intended to support that.

Target Audience

This review is most useful for practitioners and researchers working on autonomous driving who are deciding whether and how to incorporate foundation models into trajectory planning — particularly those weighing an off-the-shelf VLM against a custom vision-encoder-plus-LLM design, choosing a trajectory representation, or curating fine-tuning data for CoT, language interaction, or instruction following. It is also relevant to engineers evaluating deployment trade-offs such as edge-device efficiency, chain-of-thought cost, and open-weight availability, and to researchers entering the area who need a structured map of 37 methods and the open questions that remain. Readers seeking quantitative benchmark comparisons would need to consult the original publications, since this review does not compare or rank methods numerically.

Authors’ abstract

The emergence of multi-modal foundation models has markedly transformed the technology for autonomous driving, shifting away from conventional and mostly hand-crafted design choices towards unified, foundation-model-based approaches, capable of directly inferring motion trajectories from raw sensory inputs. This new class of methods can also incorporate natural language as an additional modality, with Vision-Language-Action (VLA) models serving as a representative example. In this review, we provide a comprehensive examination of such methods through a unifying taxonomy to critically evaluate their architectural design choices, methodological strengths, and their inherent capabilities and limitations. Our survey covers 37 recently proposed approaches that span the landscape of trajectory planning with foundation models. Furthermore, we assess these approaches with respect to the openness of their source code and datasets, offering valuable information to practitioners and researchers. We provide an accompanying webpage that catalogues the methods based on our taxonomy, available at: https://github.com/fiveai/FMs-for-driving-trajectories