KnowVal: A Knowledge-Augmented and Value-Guided Autonomous Driving System

AI summary

Overview

Research area: Autonomous driving (Robotics / computer vision), specifically end-to-end planning, vision–language reasoning, retrieval-augmented knowledge grounding, and value alignment for trajectory selection.

Technical level: Advanced. The paper assumes familiarity with end-to-end driving stacks (BEV perception, trajectory decoding), Vision–Language–Action models, retrieval-augmented generation, knowledge graphs, and Transformer-based planning.

Scope: The paper introduces KnowVal, an autonomous driving system that couples open-world perception with retrieval from a hand-built driving knowledge graph (traffic laws, defensive driving principles, ethics) and uses a human-preference-trained Value Model to score candidate trajectories.

What This Paper Is About

Existing autonomous driving systems — both end-to-end planners and Vision–Language–Action models — learn almost entirely from data, so they struggle to capture the logic behind safe decisions and cannot easily express why one action is better than another. The paper's goal is to build a driving system that reasons over explicit driving knowledge (laws, defensive driving practice, moral norms) retrieved on the fly from what the car perceives, and that then judges candidate future trajectories using a learned value model rather than imitation alone. KnowVal is presented as a system that can be layered onto existing planners while adding these reasoning and value-assessment capabilities.

Key Contributions

1. A new autonomous driving system, KnowVal, which performs visual–language reasoning through mutual guidance between perception and knowledge retrieval: perception drives retrieval queries, and retrieval results that reveal missing information are fed back to prompt the perception module to look again.
2. A comprehensive driving knowledge graph covering traffic laws, defensive driving principles, and moral considerations, together with an LLM-based retrieval mechanism tailored to driving. The graph is built from a "knowledge forest" of source texts that is turned into a graph by LLM-driven entity extraction and edge weighting, while retrieval is filtered to return only verbatim "native" clauses rather than LLM-summarized text.
3. A planner that integrates a world model for future-state prediction with a Value Model for outcome evaluation, enabling value-aligned decision-making. This is supported by a curated human-preference dataset used to train the Value Model.
4. Extensive experiments showing compatibility with existing methods, the lowest collision rate on nuScenes, and state-of-the-art performance on Bench2Drive and NAVSIM (labeled NVISIM in the paper's tables).

Main Findings

• Lowest collision rate on nuScenes: Across the methods compared in the paper's nuScenes planning table, KnowVal variants post the lowest average collision rates — KnowVal-HENet++ at 0.12 in the UniAD-metric column (baseline HENet++ 0.13) and 0.03 in the VAD/STP3-metric column (baseline HENet++ 0.05). KnowVal-GenAD reaches 0.33 average collision rate versus 0.43 for GenAD in the UniAD-metric column.
• L2 error is slightly higher, by design: KnowVal-GenAD shows an average L2 of 1.51 m versus 0.91 m for GenAD, and KnowVal-HENet++ 1.32 m versus 1.29 m for HENet++. The authors argue this reflects deviation from human demonstration trajectories rather than worse planning, and that the system "tends to discover safer or more efficient driving strategies that may not be reflected in human demonstrations."
• State-of-the-art on Bench2Drive: KnowVal-SimLingo achieves a Driving Score of 88.42 and Success Rate of 69.03, a gain of 3.35 in Driving Score and 1.76% in Success Rate over the VLA-based SimLingo (85.07 / 67.27).
• State-of-the-art on NAVSIM: KnowVal-DiffusionDrive scores a PDMS of 90.9 versus 88.1 for DiffusionDrive (a +2.8 gain), and KnowVal-iPad scores 92.9 versus 91.7 for iPad (a +1.2 gain). Sub-metrics for KnowVal-DiffusionDrive are NC 99.0, DAC 98.3, TTC 96.3, Comfort 100, EP 84.3.
• Ablations on nuScenes (baseline GenAD): Baseline collision rate 0.43 with L2 0.91; adding knowledge retrieval and the value model gives 0.38 / 0.89; adding open-world perception gives 0.34 / 1.47; adding retrieval-guided perception gives 0.33 / 1.51.
• Ablations on Bench2Drive (baseline SimLingo): Moving from the baseline (DS 85.07, SR 67.27) through knowledge, values, heuristic rules, and combinations, performance rises to DS 88.42 and SR 69.03 for the full configuration.
• Qualitative behavior: In the shown examples, KnowVal correctly applies the ethical commonsense rule "When passing through accumulated water, slow down to avoid splashing pedestrians" and the legal regulation "Inside tunnels, vehicles must not overtake across solid lines," where baselines fail.
• Benchmark blind spots: The authors state that current benchmarks lack sufficient evaluation data and appropriate metrics for many forms of improper driving, noting that failing to yield to pedestrians or overtaking inside tunnels does not affect standard metrics such as collision rate or driving score.

Methodology in Plain English

KnowVal has three interacting parts.

Looking and asking. The perception stack has three layers: standard specialized perception for common categories (vehicles, pedestrians, drivable areas); open-ended 3D perception for rare objects such as fire trucks or standing water, implemented using VL-SAMv2 and OpenAD; and abstract concept understanding via a VLM to judge contextual properties such as whether the scene is a bridge, a tunnel, daytime, or nighttime. Retrieval from the previous timestep can flag what was missed, and the perception module is then prompted to look specifically for those elements.

Building and querying a knowledge base. The authors gather traffic laws and regulations, principles extracted from defensive driving instructional videos (using Gemini 2.5-Pro with designed prompts), driver interview transcripts, and ethical guidelines. Laws and regulations already have a hierarchical structure, which becomes the initial "knowledge forest," with each clause node storing raw, unaltered text and marked as "native." An LLM (e.g., GPT-4 or Qwen) then extracts entities in predefined categories — Traffic-Sign-Device, Road-User, Driving-Maneuver, Road-Condition — creates entity nodes, and links them to the clauses containing them, also assigning semantic weights, edge descriptions, and concise text keys. At inference, a templated "perception verbalizer" turns bounding boxes, semantic-map or occupancy results (segmented into connected blocks via breadth-first search with a dynamic connectivity threshold), abstract concepts, navigation data, and user instructions into a text query. A lightweight LLM extracts two layers of keywords (macro-level context such as "Driving Security," and specific entities such as "Pedestrians" or "Rainy Weather"), the graph is expanded with Top-K nearest neighbors, and only "native" nodes are kept — so retrieved knowledge is verbatim source text, not an LLM paraphrase. The top N_K items are embedded as vectors for the value stage.

Choosing a trajectory by value, not just imitation. The planner is extended to act as a world model: distinct Gaussian noise vectors are injected into ego-vehicle features to produce N_T distinct queries and trajectories, with an added loss penalty (τ − e) whenever the L2 distance e between two candidate trajectories drops below a threshold τ, to encourage diversity. A Value Model with a Transformer Encoder–MLP Decoder then scores each trajectory against each retrieved knowledge entry. For each (trajectory, knowledge) pair, a query token is formed from the knowledge feature plus the trajectory's positional encoding and ego features; future-state tokens serve as key–value pairs; after L iterative steps, the MLP produces N_K scalars in [−1, 1], where −1 means behavior inconsistent with the rule, 1 means positive compliance, and 0 means the rule is irrelevant. Per-rule scores are pooled with a weighted decay, Score(Tᵢ) = (1/Z) Σ γ^(j−1) · sᵢ,ⱼ with Z = Σ γ^(j−1), so more relevant rules weigh more. The highest-scoring trajectory is output.

Training the value model. The Preference Dataset for Values contains 160K trajectory–knowledge pairs, each with scene states (feature vectors and BEV renderings) and ground-truth value annotations; 80% is used for training and 20% for validation and testing. Preliminary compliance scores are generated automatically by Qwen-VL-Max under a detailed four-step prompt (applicability check, evidence check, adherence check, risk assessment), then manually reviewed to correct errors and outliers. Training minimizes mean squared error between predicted and annotated scores. Configuration: Qwen2.5-3B for retrieval and knowledge embedding, N_K = 16 retrieved entries, N_T = 20 candidate trajectories, L = 3 value-model reasoning steps, γ = 0.7. nuScenes baselines are fine-tuned for three epochs on eight A100 GPUs; Bench2Drive baselines for one epoch on the same setup; the Value Model is trained separately for 50 epochs on eight V100 GPUs with AdamW and cosine annealing.

Why This Matters

Impact on research. The paper argues that imitation learning and handcrafted rules cannot fully capture the logic of safe driving, and demonstrates an alternative: grounding decisions in retrievable, verbatim legal and ethical knowledge plus a learned value model. It also points out a measurement gap — existing benchmarks do not penalize many socially or legally improper behaviors — which is a challenge to the evaluation community as much as to system builders.

Real-world applications.

• Highway and urban driving systems that must respect jurisdiction-specific traffic law and documented ethical norms.
• Fleet safety and insurance telemetry that requires an auditable, per-rule explanation for why a maneuver was chosen (each decision traces back to specific retrieved clauses and their scores).
• Driver-assistance features aimed at defensive driving, using principles distilled from instructional material and driver interviews.
• Simulation and validation pipelines that stress-test planners against long-tail objects (fire trucks, standing water) and contextual conditions (tunnels, night) that standard perception taxonomies omit.

Industry relevance. Because KnowVal is presented as modular — its knowledge graph, retrieval, and Value Model can be attached to existing planners such as GenAD, HENet++, SimLingo, DiffusionDrive, and iPad — it suggests a path to adding interpretability and value alignment to deployed end-to-end stacks without retraining them from scratch. The reported collision-rate improvements on nuScenes and the Driving Score/Success Rate gains on Bench2Drive are the kind of metrics that matter directly for safety cases and regulatory review.

Future Directions

• Expanding knowledge coverage and measuring it: The knowledge graph currently draws on laws, regulations, defensive-driving video, interviews, and ethical guidelines; how well it generalizes to new regions, rule changes, and unanticipated edge cases is an open question, and the paper's Appendix C analyses of varying knowledge-entry counts, candidate-trajectory counts, and noisy knowledge items are referenced but not detailed in the main text.
• Benchmarks that capture social and legal compliance: The authors explicitly note that failing to yield to pedestrians or illegal overtaking in tunnels leaves standard metrics untouched, motivating evaluation protocols with data and metrics for these behaviors.
• Closing the L2 gap without losing safety: Understand and control the trade-off between deviating from human demonstrations and finding safer trajectories, since the paper attributes higher L2 to that deliberate divergence.
• Scaling and efficiency: With N_T = 20 candidate trajectories and N_K = 16 knowledge entries evaluated per trajectory by a Value Model, how the system scales to longer horizons, larger knowledge graphs, and real-time onboard constraints is not reported in the available content.

Target Audience

Autonomous driving and robotics researchers working on end-to-end planning, Vision–Language–Action models, and knowledge-grounded decision-making; practitioners building interpretable or regulation-aware planning stacks that must be layered onto existing architectures; safety, ethics, and evaluation researchers interested in value alignment and in the shortcomings of current driving benchmarks; and graduate students who already know the basics of BEV perception, trajectory planning, and retrieval-augmented generation.

Authors’ abstract

Visual-language reasoning, driving knowledge, and value alignment are essential for advanced autonomous driving systems. However, existing approaches largely rely on data-driven learning, making it difficult to capture the complex logic underlying decision-making through imitation or limited reinforcement rewards. To address this, we propose KnowVal, a new autonomous driving system that enables visual-language reasoning through the synergistic integration of open-world perception and knowledge retrieval. Specifically, we construct a comprehensive driving knowledge graph that encodes traffic laws, defensive driving principles, and ethical norms, complemented by an efficient LLM-based retrieval mechanism tailored for driving scenarios. Furthermore, we develop a human-preference dataset and train a Value Model to guide interpretable, value-aligned trajectory assessment. Experimental results show that our method substantially improves planning performance while remaining compatible with existing architectures. Notably, KnowVal achieves the lowest collision rate on nuScenes and state-of-the-art results on Bench2Drive and NVISIM.