ScenicRules: An Autonomous Driving Benchmark with Multi-Objective Specifications and Abstract Scenarios

AI summary

Overview

Research area: Autonomous driving evaluation and formal methods for safety — specifically, benchmarks that score driving systems against multiple, prioritized traffic objectives in formally modeled environments.

Technical level: Advanced. The paper assumes familiarity with Signal Temporal Logic, preorders/priority relations, and simulation-based falsification, though its main ideas are explained with concrete driving examples.

Scope: The paper introduces ScenicRules, a benchmark that pairs a multi-objective, priority-ordered specification framework (Hierarchical Rulebook) with abstract, probabilistically parameterized driving scenarios written in the Scenic language, and validates both against human driving preferences.

What This Paper Is About

Autonomous vehicles routinely face situations where their goals conflict — for example, avoiding a collision may require leaving the lane, or entering the oncoming lane may improve passenger comfort. Existing benchmarks either test a single objective or combine several objectives into one weighted score, which does not express explicit priorities among them. This paper builds a benchmark that formalizes 19 driving objectives with quantitative violation measures, arranges them into a priority hierarchy that can be adapted to different driving contexts, and tests driving agents in a compact set of Scenic scenarios that includes reconstructed near-accident cases.

Key Contributions

1. ScenicRules, a combined benchmark. The authors state this is the first benchmark to combine a multi-objective, priority-based specification framework (a Rulebook) with an expressive scenario modeling notation (Scenic) for evaluating autonomous driving approaches and systems.

2. A formalized set of driving objectives. They collect and formalize 19 autonomous driving objectives with precise, quantitative violation measures, including alternative formulations for selected rules to examine how different definitions change evaluation outcomes.

3. A Hierarchical Rulebook framework. Instead of defining pairwise priorities among all 19 rules — for which the paper notes there are more than 10^35 possible orders — rules are grouped by functional similarity and priorities are set between groups, keeping the specification interpretable, extensible, and adaptable.

4. A curated scenario testbed. They assemble common scenarios using an automated Scenic program generator plus coreset selection, and reconstruct near-accident scenarios from California DMV collision reports using an LLM-assisted pipeline.

Main Findings

• The rulebook aligns with human judgment. On the Reasonable Crowd dataset (92 scenarios with pairwise human preference labels), the base hierarchical rulebook reached 80.9 ± 1.8% accuracy against human preferences, compared with 78.4 ± 2.8% for the rulebook + decision tree baseline, 82.3 ± 2.4% for random forest, and 79.1 ± 2.5% for logistic regression from prior work. The authors emphasize that their rulebook is designed independently of the dataset, whereas the learning-based methods rely on training data.

• Rule parameters improve alignment. After rule-parameter optimization, the hierarchical rulebook reached 81.8 ± 2.0% accuracy, up from the 80.9 ± 1.8% base.

• Part of the gap to human preferences is attributed to human variability. The authors report that the median agreement rate between individual annotators and the aggregate preference is roughly 84% in prior work, so perfect alignment is inherently unlikely. They attribute a significant portion of the roughly 20% discrepancy to this variability.

• Data-driven priority optimization converges quickly. On the scenario-specific priority optimization task, both the restricted version (only adjacent groups may be swapped) and the unrestricted version converged in 2 greedy iterations, reaching 85.08% and 85.97% respectively. A brute force search over all possible priority permutations reported the same best accuracy of 85.97%.

• Alternative rule formalizations change which behaviors are preferred. Changing the vehicle clearance rule to sum violations rather than take the maximum changed 6 preferences and altered the key determining rule once, with 80.92% accuracy. Classifying front/left/right vehicles by ego heading instead of future trajectory changed 23 preferences and the determining rule 251 times (81.63%). Checking only the ego centroid rather than the whole polygon for correct-side-of-road changed 69 preferences and the determining rule 171 times (78.95%). The original formalization on the whole dataset scored 80.97%.

• The scenario space is large and the generator is systematic. With up to three surrounding agents, the authors state there are over one million possible combinations of agent types, maneuvers, and spatial relations, motivating coreset selection over exhaustive testing.

Methodology in Plain English

The authors start from driving rules used in prior work and write each one as a formal formula with a matching violation score, so that a system can be judged on both whether it broke a rule and how badly. Many rules use Signal Temporal Logic, which expresses constraints like "always stay inside the drivable area" or "eventually reach the target," but the authors also use objective functions where appropriate, such as minimizing deviation from the lane centerline. Each rule is implemented as a Python function that takes simulation results and returns a violation score.

Because a flat list of 19 rules has an unmanageable number of possible priority orderings, they group similar rules — for instance, left and right vehicle clearance — into categories such as safety-critical, road compliance, safety-enhancing, social interpretability, precautionary measures, progress, and passenger comfort. Priorities are then set between groups, with each group internally ordered by a fixed principle. This makes it practical to swap group priorities for different contexts, such as emergency vehicles, dense traffic, or degraded braking performance. They also describe an algorithm that automatically refines group priorities by greedily swapping them to better match annotated human preferences.

For scenarios, they built a generator that takes a JSON-style description of the map, the ego maneuver, and other agents' attributes and outputs an executable Scenic program. Scenic lets a single program represent a distribution over parameter ranges, so one program covers many concrete situations. To choose a compact subset from the huge program space, they encode each scenario as a vector and apply k-Center Greedy selection using Hamming distance, which favors scenarios that are far from all already-selected ones. Separately, they summarize California DMV collision reports into short natural-language paragraphs and use few-shot prompting with Gemini 2.5 Flash to generate Scenic programs, each taking about 30 seconds with typically only minor human refinement.

Why This Matters

Impact on research. The paper provides a shared testbed for evaluating driving agents under explicit, prioritized, multi-objective specifications rather than a single scalar score. It also supplies evidence that formalized rules can match human preference judgments at a level comparable to learned models, while remaining interpretable and dataset-independent. It follows what the authors call the Verified AI approach, pairing formal probabilistic environment modeling with multi-objective formal specifications.

Real-world applications.

• Safety validation of autonomous vehicle stacks, including perception, prediction, planning, and end-to-end systems, since Scenic supports evaluating any component of the stack.
• Simulation-based falsification, where the benchmark and Rulebook specifications are used to surface agent failures in critical or near-accident situations.
• Regulatory and incident analysis, drawing on the reconstruction of collision reports published by the California Department of Motor Vehicles into testable scenarios.
• Context-specific behavior tuning, such as prioritizing progress over road compliance for on-duty emergency vehicles, emphasizing social interpretability in dense intersections, or adding clearance when braking performance is degraded.

Industry relevance. Because the benchmark runs on Scenic, which interfaces with widely used simulators including CARLA and MetaDrive, the scenarios can be plugged into existing simulation pipelines. Developers can adopt the default priority ordering as a baseline or adjust group-level priorities to reflect their own product preferences, and the benchmark is published openly at https://github.com/BerkeleyLearnVerify/ScenicRules/.

Future Directions

• Learning preferences directly from data. The authors state that accurately learning preferences from data remains a non-trivial challenge and name it as a direction for future work, even though their greedy priority-swapping algorithm already converges quickly.

• Extending the rule set. The collection of 19 rules is described as not exhaustive but the most extensive collection of formalized rules to date, and is designed for future extensibility; users can add custom rules as Python functions.

• Reconciling human disagreement. Since individual annotators agree with aggregate preferences only about 84% of the time, it remains open how a single rulebook should handle genuinely divergent human judgments in identical contexts.

• Standardizing which formalization is correct. The experiments show that sum-based versus max-based clearance, heading-based versus trajectory-based vehicle classification, and centroid-based versus polygon-based correct-side checks change which trajectories are preferred, leaving open how such choices should be validated or standardized.

Target Audience

Researchers and engineers working on autonomous driving evaluation, safety benchmarking, and runtime verification, particularly those already comfortable with formal methods and temporal logic. It is also relevant to simulation and test-infrastructure teams who need a compact, representative, and reproducible scenario suite, and to policy or safety analysts interested in encoding prioritized traffic rules in an interpretable form. Readers seeking beginner-level tutorials on autonomous driving or deep learning for perception will find the paper's framing more formal than introductory.

Note: The provided text is truncated during the rule parameter optimization discussion in Section V, so results reported after that point are not available in this content.

Authors’ abstract

Developing autonomous driving systems for complex traffic environments requires balancing multiple objectives, such as avoiding collisions, obeying traffic rules, and making efficient progress. In many situations, these objectives cannot be satisfied simultaneously, and explicit priority relations naturally arise. Also, driving rules require context, so it is important to formally model the environment scenarios within which such rules apply. Existing benchmarks for evaluating autonomous vehicles lack such combinations of multi-objective prioritized rules and formal environment models. In this work, we introduce ScenicRules, a benchmark for evaluating autonomous driving systems in stochastic environments under prioritized multi-objective specifications. We first formalize a diverse set of objectives to serve as quantitative evaluation metrics. Next, we design a Hierarchical Rulebook framework that encodes multiple objectives and their priority relations in an interpretable and adaptable manner. We then construct a compact yet representative collection of scenarios spanning diverse driving contexts and near-accident situations, formally modeled in the Scenic language. Experimental results show that our formalized objectives and Hierarchical Rulebooks align well with human driving judgments and that our benchmark effectively exposes agent failures with respect to the prioritized objectives. Our benchmark can be accessed at https://github.com/BerkeleyLearnVerify/ScenicRules/.