Monte Carlo Expected Threat (MOCET) Scoring

AI summary

Overview

• Research area: AI safety evaluation, specifically biosecurity risk assessment for large language models (LLMs) and AI Safety Level (ASL) threat measurement.
• Technical level: Intermediate. The paper combines probabilistic modeling (Bernoulli trials, Monte Carlo simulation), k-nearest-neighbor classification over sentence embeddings, and standard LLM benchmark evaluation.
• Scope: The paper proposes and demonstrates a single metric, MOCET, for translating LLM-generated bioweapon protocols into interpretable expected-casualty estimates.

What This Paper Is About

Existing benchmarks such as LAB-Bench, BioLP-bench, and WMDP can measure model uplift and domain knowledge, but the authors argue they do not contextualize "real-world risks" or scale in an open-ended way as models advance. The authors introduce Monte Carlo Expected Threat (MOCET) scoring, which treats each step of an LLM-generated protocol as a probabilistic success-or-failure event and converts the resulting success probability into a harm-weighted score analogous to expected casualties per incident. They then demonstrate the metric on an open-source model with reduced safety guardrails.

Key Contributions

1. The MOCET framework itself: a "doubly-scalable" metric (automatable and open-ended) that models the multi-step "Build" phase of a non-state actor's attack chain, identified by the authors as the critical bottleneck in their threat model.
2. A harm-weighted, contextualized risk score: MOCET (expected threat per incident) and Cumulative MOCET (expected threat per population per annum), the latter produced by scaling MOCET by a rate of occurrence approximated from FBI mass murder data.
3. A data-driven, instance-based step-probability estimator: a k-Nearest Neighbors model over semantic embeddings (all-mpnet-base-v2 from Sentence-Transformers) that assigns each generated step its own success probability instead of relying on manual or broad categorical assignments.
4. An empirical case study: a zero-shot evaluation of a fine-tuned Llama-3-8B model (Dolphin 2.9 variant) on bioweapon-relevant prompts, including comparison against two PhD-level human annotators.

Main Findings

• Benchmarks showed a capability drop, but risk rose: The Dolphin-2.9-Llama3-8B variant scored lower than Llama-3-8B-Instruct on every reported benchmark — MMLU (57.15% vs 63.77%), WMDP-Bio (65.99% vs 71.01%), WMDP-Chem (46.32% vs 47.06%), and GPQA (27.46% vs 29.46%). The authors state that this might suggest slight capability degradation, yet MOCET analysis showed the model's potential for misuse was "dangerously unlocked" by reducing guardrails.
• Non-zero threat scores across multiple bioweapon categories: As shown in Figure 4, the Dolphin model produced instructions leading to non-zero threat scores in several categories.
• Sarin scores: Prompts concerning Sarin yielded a MOCET score of 18.94, corresponding to a Cumulative MOCET of 568.17 expected casualties per year.
• Anthrax scores: Prompts for Anthrax resulted in a MOCET of 0.58 and a Cumulative MOCET of 17.50.
• Model vs. human success estimates diverged differently by agent: For Anthrax, the model estimated E[Y] of 1.18% (described as conservative) while human experts perceived 16.5%. For Sarin, the model's E[Y] of 0.82% was slightly more optimistic than the human rating of 0.5%.
• k-NN validation on academic benchmarks: The k-NN model's predicted accuracy for a given statement was significantly higher for correct answers than incorrect ones (p << 0.01 for k=10, 20, 40) on benchmarks including MMLU, GPQA, and WMDP.
• Robustness to probability error: The framework is reported to have an estimated ~10% deviation in step probabilities resulting in only a ~1% error in the final score.
• Contextualization against public safety statistics: The paper notes that a per-incident MOCET score may be compared to 18.86 casualties per incident using guns, and a cumulative MOCET score may be compared to 44,534 motor vehicle traffic deaths.

Methodology in Plain English

The authors model a bioweapon-building protocol as a chain of steps, each treated as a coin flip that either succeeds or fails. If each step has its own probability of succeeding, the overall chance the whole protocol works is the product of those step probabilities (with steps grouped into categories, per Eq. 8). Running this as a Monte Carlo simulation over N trials produces a distribution of outcomes rather than a single number.

Each successful outcome is then weighted by a harm function W, defined as expected casualties derived from historical data on relevant bioweapons (Table 1). This produces the MOCET score, an estimate of expected threat per incident (Eq. 5). To contextualize the risk at population scale, the score is multiplied by a real-world occurrence rate — approximated using FBI data on mass murder incidents, specifically the 30 mass murders in 2017 — producing the Cumulative MOCET score (Eq. 6).

The hardest part is estimating each step's success probability. Rather than assign steps to broad manual categories, the authors convert each step's text description into a high-dimensional semantic embedding using all-mpnet-base-v2. For a given step, they find its k nearest neighbors in a historical dataset of steps with known outcomes (success or failure) and take the mean success rate of that neighborhood as the estimated probability (Eq. 16). They first validated this k-NN approach on general academic and technical benchmarks, confirming it distinguished correct from incorrect answers.

For the case study, they selected a fine-tuned Llama-3-8B model trained on the publicly available Dolphin 2.9 dataset, which is based on the Orca methodology of learning from complex explanation traces of more powerful models. They chose it because it would be reasonably accessible and non-compute-intensive to a non-state actor. All model evaluations used the lm-evaluation-harness. In a zero-shot setting, they prompted the Dolphin model with queries representative of those a non-state actor might use, then used the validated k-NN model with k=20 to predict success probability for each step in the outputs. Two PhD-level annotators independently rated the likelihood of success for the generated protocols to ground the automated calculations. The authors state that the code for the MOCET framework and the prompts used are available upon request.

The paper's threat model (Figure 1) partitions non-state actor biosecurity risk into four stages — Deploy, Build, Procure, Research — noting levels of possibility or impossibility and estimated cost. The "legal" branch is described as most probable, and the Build stage (informed by the Research stage) and its implied n substeps are identified as the greatest bottleneck.

Why This Matters

• Impact on research: The paper argues that standard academic benchmarks are inadequate for capturing catastrophic risks, since a model can lose benchmark performance while gaining misuse potential. MOCET is presented as complementary to LLM-as-a-judge methods and aligned with risk preparedness and scaling policy frameworks laid out by OpenAI, Anthropic, and the National Institute of Science and Technology (NIST), by offering a quantitative, iterative, and transparent risk assessment tool.
• Real-world applications:
  • Safety cases for public-use LLM releases, by measuring and mitigating risks before incidents occur.
  • Evaluating the effectiveness of safety interventions and guardrail reductions.
  • Informing stakeholders and policymakers creating and steering safe AI systems.
  • Informing capability reports and safeguard evaluations across frontier AI development.
• Industry relevance: The finding that MOCET yielded a non-zero risk estimate for an open-source LLM suggests that even with current technological constraints, such models can meaningfully lower barriers to access for malicious actors. The authors state this underscores the importance that AI development firms and governments approach the implementation and release of open-source LLMs with caution and responsibility.

Future Directions

• Improving probability and harm estimates: The authors state that MOCET's accuracy depends on accurate estimations of individual step probabilities and the weighting function used to assess harm, both of which require more real-world empirical data to determine accurately.
• Addressing adversary behavior assumptions: MOCET currently assumes the actor would be unable to fact-check and would not use best-of-n or multi-turn prompting; relaxing these assumptions is an open direction.
• Testing the "correctness is sufficient" assumption: The framework assumes that correctness of information provided is sufficient to estimate risk, which the authors list as a limitation.
• Extending beyond order-of-magnitude use: The authors note that the score should be considered an order-of-magnitude estimate, but argue MOCET is inherently a monotonic measurement and thus reliable for assessing safety measures — a property that could be tested further against real-world data.

Target Audience

AI safety researchers and evaluators working on biosecurity and frontier model risk; policy analysts and government stakeholders involved in AI regulation and safety cases; model developers at AI labs deciding whether and how to release open-source models; and biosecurity or public health experts who need an interpretable, casualties-based framing of LLM-related risk.

Authors’ abstract

Evaluating and measuring AI Safety Level (ASL) threats are crucial for guiding stakeholders to implement safeguards that keep risks within acceptable limits. ASL-3+ models present a unique risk in their ability to uplift novice non-state actors, especially in the realm of biosecurity. Existing evaluation metrics, such as LAB-Bench, BioLP-bench, and WMDP, can reliably assess model uplift and domain knowledge. However, metrics that better contextualize "real-world risks" are needed to inform the safety case for LLMs, along with scalable, open-ended metrics to keep pace with their rapid advancements. To address both gaps, we introduce MOCET, an interpretable and doubly-scalable metric (automatable and open-ended) that can quantify real-world risks.