PlannerForge: LLM Agents for Scenario-Based Testing of Motion Planners in Autonomous Driving

AI summary

Overview

Research area: Autonomous driving validation, specifically scenario-based testing of motion planners, combined with LLM-agent systems.

Technical level: Intermediate. The paper assumes familiarity with autonomous driving testing concepts (scenario databases, motion planners, simulation formats like CommonRoad and SUMO) and with LLM agent techniques such as tool calling, chain-of-thought prompting, and in-context learning. The formal notation in Section 3 can be skimmed without losing the argument.

Scope: The paper presents a single unified LLM-agent framework that automates the entire scenario-based testing pipeline for autonomous driving motion planners, from scenario generation through planner tuning, and benchmarks it against prior dedicated tools.

What This Paper Is About

Testing an autonomous driving system safely requires simulating the rare, dangerous situations it might encounter, because driving millions of real-world miles to find them is impractical. Today this testing is done as a fragmented chain of separate tools: one program generates scenarios, another filters a database, a script runs the test, and a human interprets the results. The authors ask whether a single conversational LLM-agent framework can automate that whole chain, and whether off-the-shelf language models — with no domain-specific training — are good enough to do each stage reliably.

Key Contributions

1. PlannerForge, a full-lifecycle LLM framework. To the authors' knowledge, this is the first framework to unify all six classical scenario-based testing stages (Scenario Source, Generation, Database, Selection, Test Execution, ADS Assessment) in one chatbot-driven pipeline, plus two new stages enabled by LLMs: ADS Enhancement (natural-language planner cost tuning) and ADS Benchmarking (cross-planner comparison under shared scenarios).

2. A broad empirical evaluation of off-the-shelf LLMs. Ten model backends — five commercial APIs (Qwen3.6-plus, Deepseek-v3.2, Glm-5, Gemini-3-flash, Gpt-5.4-mini) and five open-source models (Qwen3.6:35B in think/no-think variants, Gemma4:31B in think/no-think variants, Gpt-oss:20B) — are tested across eight task slices under five prompt conditions, totaling roughly 80,000 calls with no fine-tuning.

3. A unified multi-planner interface. Two classical planners — the sampling-based Frenetix and the learning-based MP-RBFN — are wrapped behind a common interface so the same generated or modified scenario can be run against both for comparative evaluation.

4. Head-to-head comparisons with the strongest prior tools at each pipeline stage, demonstrating where LLM-driven stages beat rule-based, keyword-based, and earlier LLM-based baselines, and at what latency and token cost.

Main Findings

• Module scores range from 0.88 to 1.00. Generation reaches 0.957 (Glm-5 with context prompting plus chain-of-thought, versus 0.783 baseline), Selection reaches 0.880 joint satisfaction (Qwen3.6-plus, versus 0.180 baseline), the Module Router reaches 0.997 (Gemma4:31B, versus 0.722 baseline), and Planner Testing and Enhancement reach a perfect 1.000 (Gpt-5.4-mini, versus 0.675 baseline). Modification hits roughly 100 percent on headline structural checks.

• The full pipeline composes, but with measurable leakage. Chaining all stages end-to-end on 200 seed queries retains 83 percent (commercial backend) and 78 percent (open-source backend). Selection and Modification are the two leak points; Generation starts both runs at a 96 percent pass rate. Average cost per scenario is about 126 seconds and 42.5k tokens (commercial) versus 99 seconds and 32.6k tokens (open-source).

• Open-source 20–35B models are competitive. Qwen3.6:35B matches commercial APIs on three of the five core tasks. Open-source models do not match commercial ones at every stage, but they can drive the whole pipeline.

• Extra prompting can hurt reasoning models. Applying context prompting plus chain-of-thought plus in-context learning to models with native reasoning ("Think" variants) disrupts their internal reasoning, increasing latency and token usage while degrading accuracy. On Selection, adding chain-of-thought on top of context prompting dropped every reasoning-enabled model (for example, 65.0 to 40.0 percent) while lifting the non-thinking Qwen3.6:35B (63.5 to 83.0 percent).

• Prompt technique choice should match the failure mode. Context prompting saturates closed-vocabulary tasks; in-context learning is what enables correct refusals of out-of-vocabulary requests (0 percent baseline, 44.4 percent with context prompting only, 100 percent with in-context learning); chain-of-thought helps tasks involving joint constraints.

• Generation beats Scenario Factory 2.0 on attributes, loses on speed. PlannerForge produced 193 executable scenarios out of 200 versus 144, realised 92–96 percent of requested city, road, and vehicle attributes (which SF 2.0 cannot target at all), and used seven traffic-participant classes instead of four — but takes 21.6 seconds versus 2.2 seconds per scenario. It also induces far more planner collisions (20.0 versus 6.1 percent), which is desirable for safety-critical testing.

• Selection beats BM25 at rank 1. PlannerForge satisfies 92.0 percent of queries at rank 1 versus 67.5 percent for keyword search, and 96.5 versus 86.0 percent anywhere in the top five — at a cost of 21.6 seconds and 8.7k tokens per query versus effectively zero.

• Modification beats From-Words-to-Collisions on physical validity. At least 94 percent of PlannerForge edits are kinematically valid versus 31 percent for FWtC, because every edit routes through a SUMO simulation round-trip. Participant edits produce 58 valid new collisions and push mean risk from 1.84 down to 1.20, versus only 16 new collisions and a drop to 1.69 for FWtC.

• LLM cost tuning substantially improves planner safety. Against a hand-set default configuration, LLM retuning raised planner success by 17.6 to 21.3 percentage points and cut collisions by 10.5 to 13.3 points across batch sizes from 50 to 400, without any domain-specific fine-tuning. At N=400, success went from 50.4 to 70.2 percent and collisions from 19.0 to 8.4 percent. All 15 tuning rounds returned the identical configuration.

• A schema-checked harness is essential. Because LLMs hallucinate structured outputs (invented tags, invalid map identifiers, misrouted module calls, unsupported analysis claims), every scored module parses, schema-validates, and executes output before accepting it.

Methodology in Plain English

The system is built as a chatbot with a Gradio front end and a LangChain back end, keeping three levels of state: conversation memory, visual chat history, and in-session storage of intermediate module outputs.

Six modules do the work. The Router reads each user message and classifies it into one of five actions — modify, tune, test, analyse, or general question answering — using LLM function calling, then dispatches the extracted arguments to the right module. This breaks the rigid linear testing workflow and lets users navigate freely.

The Generation module parses a natural-language request like "Munich intersection with light traffic, focus on a turning truck" into a structured JSON intent (location, road classes, traffic density, vehicle mix, duration), queries OpenStreetMap through the Overpass API, simulates traffic microscopically in SUMO, converts the result to CommonRoad format, and then attaches an ego vehicle initial state and goal region to form a solvable planning problem.

The Selection module runs a five-step dialogue, extracting one metadata slot per step (location, tags, road network, obstacles, velocity) and intersecting the matching sets to prune a CommonRoad scenario database indexed in a Chroma vector store. If exact matching fails, it falls back to SentenceTransformer semantic similarity over the whole database.

The Modification module converts scenarios back into SUMO network and route files, lets the LLM emit an edited file, and converts back — which guarantees kinematically feasible trajectories. Four edit categories are supported: Trajectory, Behaviour (six car-following presets from Aggressive to Eco), Population (adding or removing vehicles), and Goal (moving the ego vehicle's goal region).

The Testing and Enhancement module runs either Frenetix or MP-RBFN behind a unified interface, returning trajectory, collision flag, and cost log. The tuner translates qualitative requests like "drive safely" into schema-conformant YAML weight overrides.

The Analysis module takes batch results plus a user question and assembles a prompt containing aggregate statistics, chronological per-scenario logs, the active configuration, and the raw CSV path, then returns metrics, a failure-mode breakdown, and parameter-adjustment recommendations that feed back into the tuner.

All modules share three prompting techniques — contextual prompting (injecting output schema and constraints), chain-of-thought (explicit reasoning steps), and in-context learning (few-shot examples including negative refusal cases) — evaluated in five combinations from bare zero-shot up to all three combined.

Why This Matters

Safety validation is the bottleneck for deploying Level 4 autonomous driving. If the validation pipeline itself is fragmented and partly manual, it cannot keep up with the number of edge cases that need coverage. This paper shows that a general-purpose LLM agent stack, with no domain fine-tuning, can automate the whole pipeline while beating specialised tools at several individual stages. It also contributes a reusable negative finding: prompt engineering techniques interact badly with natively reasoning models, and prompts should be matched to specific failure modes rather than stacked indiscriminately. That insight transfers well beyond autonomous driving to any structured-output agent task.

Real-world applications:

• Automotive OEM and supplier test teams can replace GUI scenario editors, hand-crafted database filters, and separately scripted batch runs with a single conversational interface, lowering the expertise barrier for creating safety-critical test cases.
• Regulatory and homologation bodies working under frameworks like ISO 21448 (SOTIF) need reproducible, attribute-faithful scenario generation and documented comparative planner evaluation; this framework produces both with schema-checked outputs.
• Motion-planning researchers get a unified multi-planner benchmark that runs the same scenario set against sampling-based and learning-based planners, making architectural comparisons more direct.
• Simulation tool developers can adopt the CommonRoad-plus-SUMO round-trip approach to guarantee that LLM-generated edits remain physically valid, a problem that undermined earlier LLM modification tools.

Industry relevance: The cost profile is the honest counterweight. PlannerForge takes roughly 20 to 60 seconds and tens of thousands of tokens per operation where classical tools take under a second to a few seconds. It buys attribute control, rank-1 retrieval precision, physical validity, and a measurable planner safety lift in exchange. For teams already running large batch test campaigns on GPU infrastructure, that trade is likely acceptable; for high-throughput regression suites, it is not. The finding that 20–35B open-source models can drive the pipeline on most tasks matters commercially, since it removes dependence on closed APIs and keeps test data on-premises.

Future Directions

• Closing the loop with reactive agents. The framework currently runs open-loop: other traffic participants follow recorded or SUMO-exported trajectories and never react to the ego vehicle. True adversarial falsification requires closed-loop simulation where surrounding agents respond to what the planner does.
• Extending to other simulators. The authors explicitly name CARLA as the next target. This would test whether the module architecture generalises beyond CommonRoad, or whether parts of it are tied to that format's metadata schema and SUMO bridge.
• Fixing the Selection bottleneck. Selection is both the hardest task per module (0.880 joint satisfaction) and the pipeline's main leak point, with tag over-prediction causing a 6 to 11 percent drop. Better slot extraction or a less brittle matching strategy than strict five-stage intersection is needed.
• Validating the Analysis module. The result-interpretation module is not scored against ground truth, so its recommendations are currently unverified. Measuring whether its diagnoses match actual failure causes would close an evaluation gap.

Target Audience

This paper is most useful to autonomous driving validation engineers and researchers who need to scale scenario-based testing, and to applied LLM researchers interested in multi-module agent systems that must emit schema-conformant structured outputs. It is also relevant to motion-planning researchers seeking a comparative benchmark harness, and to graduate students in robotics or vehicle engineering looking for a concrete case study of where LLM agents succeed and fail in a safety-critical engineering pipeline. Readers without background in autonomous driving simulation can follow the high-level findings but will need some familiarity with CommonRoad, SUMO, and planning cost functions to appreciate the design choices.

Authors’ abstract

Ensuring the safety of autonomous driving is a critical challenge. Scenario-based testing is a systematic process used to validate Autonomous Driving Systems (ADSs), but it remains a fragmented modular pipeline in which scenario generation, retrieval, modification, ADS execution, and results analysis are performed by separate tools with little interaction. Large Language Model (LLM) agents have shown promise across ADS sub-systems such as perception, planning, and control. However, no prior work covers the whole scenario-based testing pipeline for ADSs with a unified LLM-agent framework. We present PlannerForge, an LLM-agent framework that extends all scenario-based testing stages (from Scenario Generation to ADS Assessment) and adds two further LLM-enhanced stages: ADS Enhancement and ADS Benchmarking. We evaluate PlannerForge with 10 off-the-shelf LLMs across all tasks (Generation, Selection, Modification, Module Routing, Planner Testing, and Enhancement) under 5 prompt conditions. Best-per-task scores range from 0.88 to 1.00, and open-source 20-35B backends match commercial APIs on most tasks. Open-source models such as Qwen3.6:35B match commercial APIs on three of the five tasks. Chaining the modules end-to-end retains 83% / 78% of seed queries (commercial / open). It outperforms Scenario Factory 2.0 (Finkeldei et al., 2025) on natural-language generation (193 vs. 144 executable of 200) and realises 92-96% of requested city, road and vehicle attributes. It outperforms BM25 (Robertson and Zaragoza, 2009) at rank 1 selection (92.0% vs. 67.5%) and From-Words-to-Collisions (Gao et al., 2025) on physically valid edits (>=94% vs. 31%). At N=400, cost-tuning lifts planner success from 50.4% to 70.2% and cuts collisions from 19.0% to 8.4%, without domain-specific fine-tuning.