The Promises and Limitations of Evaluation-Based AI Governance: Evidence from Expert Interviews

AI summary

Overview

Research area: AI safety governance and evaluation policy (CS/CY — computers and society), specifically qualitative empirical research on how frontier AI evaluations are expected to reduce societal-level risk.

Technical level: Intermediate. No technical AI background is required, but readers benefit from familiarity with concepts like dangerous capability evaluations, benchmarks, red lines, safety cases, and sandbagging.

Scope: A thematic analysis of 37 anonymized semi-structured expert interviews that reconstructs the "theories of change" behind evaluation-based AI governance, the assumptions those theories rest on, and where participants expect those assumptions to break.

What This Paper Is About

Evaluations of AI capabilities have become the main empirical tool that governments and companies use to identify and manage frontier AI risk, yet the reasoning for how a test score actually translates into reduced societal harm is rarely spelled out. The authors interview 37 experts across governance, evaluation development, technical research, and government advising to make that implicit reasoning explicit. The goal is to identify the mechanisms experts believe connect evaluations to risk reduction, the assumptions each mechanism depends on, and the challenges that threaten them.

Key Contributions

1. Reconstructs three explicit "theories of change" for evaluation-based governance from expert accounts: evaluations make risks concrete (raised in 18 of 37 interviews), evaluations form the basis for governance commitments and regulation (18 of 37), and evaluations shape safety incentives inside AI companies (16 of 37).

2. Maps the foundational and pathway-specific assumptions underlying these mechanisms into a structured table (Table 1), covering assumptions such as that evaluators can elicit capabilities near the performance ceiling, that evaluation performance corresponds to real-world risk, that models do not strategically underperform, and that results reach and are acted upon by decision-makers.

3. Documents and ranks the challenges experts raised against those assumptions, ordered by frequency: diminishing informativeness as capabilities improve (21), measurement limitations possibly inherent to the paradigm (18), no guarantee results will be acted upon (16), and developer control over evaluation design, disclosure, and response (12).

4. Surfaces a counterproductive pathway: some participants (4) argued evaluation-based governance could function as a permission structure that absorbs political energy and legitimizes continued capability development rather than constraining it.

Main Findings

• Advance warning and political momentum are the core of the "wake-up" pathway. Participants argued internal results can let developers and officials "see capabilities up to a year before these capabilities would be made public" (US24), and that "waiting for large real-world impacts is probably too late" (US27). Evaluations were seen as more credible than demos because audiences react as though "the fancy Stanford professor did science on the model" (P15).

• Evaluations are expected to ground commitments, red lines, and an industry floor. Participants described tying results to if-then safeguards, to pausing development, and to peer pressure among labs (e.g., Anthropic, OpenAI, Google DeepMind cited by US37), with evaluation-based governance potentially serving "first as industry standards" and then "getting into legislation" (US34).

• Internal incentives may improve safety practice. Evaluations can refine safeguards, force attention onto neglected risk domains (US35 noted no evaluations exist for multi-agent collusion), and create accountability records — including leverage for risk-minded employees to "say, 'slow down'" (A3).

• Measurement limits are seen as potentially inherent. "Just because your model passes all your evals, it doesn't mean it's necessarily safe. We don't have any bounds or guarantees on that" (A5); "no one's satisfied with the evaluations that we currently have" (I12).

• There is no reliable path from information to action. Voluntary commitments were widely doubted across eight participants (A2, A6, EU10, P15, US24, US28, US35, US37); A2 noted companies "can kind of just change it at any time, as multiple companies have done. And no one really cares." Exit clauses, commercial pressure, motivated reasoning, dual-use justifications, absent political will, and racing dynamics were all cited. US37 called safeguard readiness "probably the most fragile assumption."

• Developers retain substantial control over the evaluation ecosystem. They control reporting (no standardized system cards), thresholds and responses ("really ad hoc" and "somewhat arbitrary" per US23), external access (query-only access is "extremely limiting" because "all the state of the art red teaming tools use white box access" — A1), and which evaluations exist. External testers typically see a version that is not final (UK18).

• Evaluations may lose informativeness as capabilities scale. Benchmark saturation outpaces replacement (US23: models advancing "much more rapidly than we can advance the evaluations ecosystem"), complex agentic and multi-agent behavior lacks methodology ("there's not a science of this yet" — P15), and evaluation awareness and sandbagging could create "false security" (A2), framed as "an empirical race between evaluation awareness and the evolution of scheming."

• The 'inability argument' is currently doing much of the safety work. Participants noted that once models exceed human expert level and the inability argument no longer applies, "a lot of [evals] become much less useful" (UK19), leaving the situation "much more precarious."

• Governance may be counterproductive. Participants described evaluation-based governance as "essentially permission notes to do something very dangerous" (UK17) and as an arrangement that "looks pretty good from developers' perspective because it's better than harder regulation and in practice constrains their behavior basically not at all" (US27).

• The authors explicitly do not conclude that evaluations cannot support governance. They frame the findings as identifying institutional conditions that would be needed for evaluations to reliably reduce risk, with a closing discussion of the conditions that would improve the paradigm's prospects (details of that closing discussion are beyond the truncated text provided here).

Methodology in Plain English

The authors recruited experts purposefully and through snowball sampling, contacting roughly 140 people and completing 37 interviews across four groups: AI governance researchers, evaluation developers, technical AI researchers, and government AI advisors. Participants were based mainly in the US and UK, with some in the EU, and included people currently or formerly at frontier AI companies, government agencies, nonprofits, and academia. After 27 interviews, no new themes appeared, which the authors treat as saturation.

All interviews were run online by the first author between February and April 2025, lasting 60–70 minutes, and were anonymized. The protocol had four sections: background and definitions; theories of change and their required non-technical conditions; limitations and criticisms; and additional technical questions for researchers. The protocol was revised after three pilot interviews.

Transcripts were produced with a local instance of OpenAI's Whisper and then anonymized; no recordings or identifiable data went to any other AI system. Analysis used hybrid thematic analysis combining 30 deductive codes from the protocol and literature with inductive codes, producing a final codebook of 250 codes, 220 of them emergent. Two coders calibrated on three transcripts, then a fourth, and split the remaining 33, discussing about five jointly. The authors note that quote-based rather than segment-based coding made metrics like Cohen's kappa inappropriate. A short Likert component (five-point scale from −2 to +2) covered trust in developers to evaluate honestly, trust in developers to act on concerning results, and expectations about reliability as capabilities advance; 33 of 37 participants completed it (Figure 1). Assumptions were derived from three sources: those participants stated explicitly, latent assumptions from interpretive coding, and assumptions inferred from participants' criticisms.

Why This Matters

Impact on research. Prior interview work on evaluation-based governance is thin — the authors cite Wei et al. (2024), a 17-interview study of US AI policy experts, as the closest methodological precedent — and most practitioner studies examine fairness and bias evaluations inside industry rather than frontier-risk evaluations embedded in regulation. This paper supplies an empirical account of the mechanisms and assumptions that policy proposals like the EU AI Act, California's SB-53, and the Frontier AI Safety Commitments implicitly rely on.

Real-world applications:

• Informing how regulators design and staff third-party auditing arrangements, given participant claims that query-only access undermines external evaluation validity.
• Informing standardization of system-card reporting, since participants described no standardized way to report and wide variation across companies.
• Informing the design of if-then commitments and red lines, where participants described thresholds as ad hoc and subjective in wording.
• Informing oversight of voluntary commitments, especially the use of exit clauses and the absence of enforcement.

Industry relevance. The findings bear directly on frontier labs' frontier safety policies, on the credibility of safety cases, and on the perceived legitimacy of self-run evaluations. The "permission structure" critique is a reputational and regulatory risk for labs that lean heavily on evaluation-based governance.

Future Directions

• Specifying the institutional conditions under which evaluation-based governance works. The authors frame this as the paper's forward agenda; the specific conditions discussed in the closing section are not fully captured in the provided text.
• Studying the translation gap between evaluation results and policymaker understanding, given participant claims that most policymakers have not heard much about evaluation-based governance and that "people agree what the numbers are, but they don't agree what it means" (US36).
• Developing methodology for complex and agentic risk surfaces, including multi-agent interactions, longer time horizons, and collusion, where participants reported that "there's not a science of this yet" and that no evaluations exist.
• Tracking evaluation awareness and sandbagging empirically over time, since participants described this as a race whose outcome determines whether evaluation results can be trusted at all.

Target Audience

AI governance and policy researchers, frontier lab safety and policy staff, government AI advisors and regulators drafting evaluation-triggered rules, evaluation developers and third-party auditors, and safety researchers who rely on evaluations as evidence in safety cases. It is also useful for social scientists studying how procedural governance instruments function in practice, and for readers who want a structured, expert-sourced account of the weaknesses of the evaluation paradigm rather than a purely technical critique.

Authors’ abstract

We conducted 37 semi-structured expert interviews to examine how frontier AI evaluations reduce societal-level AI risks. Evaluations have become central to AI risk governance, despite their shortcomings being widely critiqued. Still, the case for how they reduce risks in practice remains largely implicit. Drawing on expert accounts, we reconstruct this case by identifying the mechanisms through which evaluations are expected to reduce risk, the assumptions underpinning those mechanisms, and the challenges those assumptions face. Our findings suggest that `evaluation-based governance' could contain foundational gaps which undermine risk-reduction strategies that depend on evaluations. Using thematic analysis, we identified three key pathways through which evaluations could reduce risk: by (i) making societal-level risks concrete to decision-makers and the public (raised in 18 out of 37 interviews), (ii) serving as a basis for governance commitments and regulation (18), and (iii) incentivizing a better risk culture within AI companies (16). However, these pathways often depended on shared assumptions that participants themselves doubted. The main challenges included, in order of frequency: (i) diminishing informativeness of evaluation results as model capabilities improve (21), (ii) measurement limitations that may be inherent to the evaluation paradigm (18), (iii) a lack of guarantees that evaluation results will be acted upon by companies or governments (16), and (iv) the degree of control and discretion AI companies retain over evaluation design, disclosure, and response (12). Participants also noted that evaluation-based governance could prove counterproductive, by functioning as a permission structure for developers' continued capability improvement (4). We finish by discussing the implications of our findings for evaluation-based governance, and what would improve its prospects.