Reproducibility: The New Frontier in AI Governance

AI summary

Overview

Research area: AI governance, metascience, and research reproducibility (with comparative case studies from economics, cancer biology, and psychology).

Technical level: Beginner-Friendly. This is a commentary/position paper rather than an experimental contribution; it contains no new models, datasets, or benchmark results, and its arguments are framed for a policy audience.

Scope: The paper argues that weak reproducibility standards in AI research degrade the information available to policymakers, and proposes preregistration, improved statistical power, and negative result reporting as governance tools, contextualised against historical reproducibility crises in other scientific domains.

What This Paper Is About

AI policymakers are expected to design effective governance for a technology whose research base is growing faster than almost any other field, yet the scientific literature they must rely on is often not reproducible. The authors argue that the combination of rapid publication in AI and weak reproducibility protocols lowers the "Signal-To-Noise Ratio" of AI research, which favours regulatory capture and leaves policymakers uncertain about which risks to prioritise. The goal of the paper is to show how adopting stricter reproducibility guidelines — in particular preregistration, higher statistical power, and publication of negative results — could strengthen AI governance and build consensus on the AI risk landscape.

Key Contributions

1. A framing of AI's coming reproducibility problem through the lens of prior crises in economics, cancer biology, and psychology, drawing parallels between irreproducible findings and harmful downstream policy and clinical outcomes.
2. An indicative mapping of scientific domains by publication speed against reproducibility standards (Figure 1), using Dimensions publication data (Table 1) and prior reproduction analyses, categorising domains into Dogma, Excessive, Progressive, and Optima quadrants under the broader labels "Alchemy" and "Science."
3. An analysis of NeurIPS papers mentioning GitHub between 2019 and 2024 (Figure 2) as an imperfect proxy for replicability, with the stated limitations of that proxy documented in the appendices.
4. A policy-oriented "Theory of Change" (Appendix B) describing how mandating reproducibility standards for policy-relevant AI research could raise the Signal-To-Noise Ratio, improve the signal available to governance professionals, and enable more effective policy — while acknowledging trade-offs in slower research, higher researcher burden, and increased compute requirements.

Main Findings

• AI is the fastest-growing domain in the comparison. Information & Computer Science publications grew 72.008% between 2019 and 2024 (from 475,933 to 818,642), the largest growth of the seven domains in Table 1, versus 26.284% for Biomedical & Clinical, 25.479% for Biological, 20.156% for Psychology, 34.284% for Economics, 13.478% for Mathematical, and 11.888% for Physical. The authors state AI publications have been growing on average at a circa 50% faster rate between 2019 and 2024 compared to most domains.

• Economics: mandatory data sharing was largely not honoured. A 2006 study of the Journal of Money, Credit, and Banking found only 25.56% (68/266) of authors had archival entries where required, and only 22.58% (14/62) of empirical studies could be replicated. A later study found successful replication of less than half of a selected group of papers published across top economic journals.

• Economics: the "Growth in The Time of Debt" case study. The original paper asserted that when external debt reaches 60 percent of GDP, annual growth declines by about two percent, and for higher levels growth rates are roughly cut in half. A replication attempt failed due to missing data and existing calculation errors; when analysed correctly the replicators found no trend for OECD countries conforming with respect to debt and growth, undermining the austerity policy justification drawn from the original findings.

• Cancer biology: very low reproducibility. An eight-year Center for Open Science study attempted to replicate 193 experiments from 53 high-impact preclinical papers published between 2010 and 2012. Only 2% of studies had open data, 0% had pre-requisite protocols that allowed for replication, and among the experiments that could be successfully reproduced (50/193) effect sizes were 85% smaller on average than the original findings. The paper also reports that 19 of 20 cancer drugs used for clinical studies do not demonstrate enough safety, efficacy, or commercial promise to achieve license.

• Psychology: partial replication with deflated effects. The Open Science Collaboration attempted to replicate 100 randomly selected studies from three prestigious journals over three years, reporting in 2015 that only 36% of successfully replicated studies had significance in the same direction as the originals, and that effect sizes were half those originally reported. This prompted the Transparency and Openness Promotion Guidelines and the TOP Factor metric.

• NeurIPS code sharing is rising but incomplete. Papers mentioning GitHub increased over the past five years, with the most notable change coinciding with the introduction of the NeurIPS Datasets and Benchmarks track in 2021. The paper states a large number of papers over the last five years do not mention any code base; the exact counts are not reported in the text provided.

• Reproducibility scores by domain are qualitative. The figure's reproducibility axis is assigned from prior studies rather than a new measurement: biomedicine is placed using evidence that 72% of surveyed researchers agreed there was a reproducibility crisis in biomedicine (27% calling it significant) and a Brazil study showing only 21% of experiments were replicable; computer science is placed using prior low-reproducibility and repeatability findings plus a study reporting 24% replication of ICML 2024 Oral/Spotlight papers by LLMs at significant computational cost and less than 50% reproducibility by PhD students; biological sciences uses a 2016 survey where 70% of researchers could not reproduce others' findings and circa 60% could not reproduce their own. Physics is assigned the highest reproducibility score partly because empirical particle physics adopts 5-sigma significance, and mathematics is placed high because it depends on proof rather than empirical study.

• Negative results are largely absent. Publication bias favours significant or positive results, driven by rejection of negative-result studies, the opportunity cost of writing them up, lack of citation incentive, and funding that favours positive outcomes, limiting oversight of AI's current limitations.

Methodology in Plain English

The authors do not run experiments on AI systems. They synthesise prior replication and reproducibility studies from economics, cancer biology, and psychology, then compare them to the AI field. For the comparative figure, they take publication counts by domain from the Dimensions database (data as of 11 April 2025, with publication counts covering 2019 to 2024) and plot them against reproducibility scores assigned by hand based on previously published reproduction analyses. For the AI-specific evidence, they count accepted NeurIPS papers between 2019 and 2024 whose PDFs mention GitHub in the main tracks (2019-2021) and the Datasets and Benchmarks track (2022-2024), treating a GitHub mention as a weak proxy for replicability. Both figures are explicitly labelled indicative rather than complete reconstructions, and the appendix documents the limitations: a single database, AI treated as a proxy for all computer science, GitHub mentions not guaranteeing available or working code, and exclusion of workshop papers and PDFs that produced analysis errors. Code to replicate the data and figures is provided at a public GitHub repository.

Why This Matters

The paper's central claim is that reproducibility is a governance instrument, not just a scientific norm. If policymakers cannot distinguish reproducible from irreproducible AI findings, they cannot accurately assess capabilities and risks, and industry actors may benefit from an information environment that favours regulatory capture through asymmetric information. The historical cases are presented as evidence that irreproducible findings can drive real policy harm, including austerity policies linked in cited studies to increased inequality and to hundreds of thousands of excess deaths in the United Kingdom.

Real-world applications implied by the paper:

• AI regulation and standards bodies: mandating reproducibility protocols for research used to justify policy positions on AI risk and capability.
• Machine learning conferences and journals: requiring preregistration, sample-size guidance, and negative-result tracks to raise the field's Signal-To-Noise Ratio.
• Clinical and healthcare AI: the cancer biology case is used to argue that integrating AI into clinical pipelines demands reproducibility standards comparable to those recommended for preclinical research.
• Compute policy: the proposal that government-backed AI Factories or Gigafactories provide open GPU and CPU access so that academic researchers can run high-statistical-power experiments.

Industry relevance: the paper frames reproducibility as a condition for trustworthy AI development and for the credibility of capability claims that feed into corporate deployment decisions, procurement, and regulatory compliance.

Future Directions

• Establish consensus reproducibility standards for AI publishing. The paper argues this requires agreement from publishing venues and reviewers, since current AI venues do not require preregistration; how such a mandate would be enforced is left open.
• Define sample-size and statistical guidance for AI experiments. The authors suggest guidance on sample sizes and appropriate statistics such as Standard Error of the Mean, but do not specify thresholds; developing them is presented as a task for the community.
• Build incentives and funding for negative results. The paper points to workshops such as the Science for Deep Learning NeurIPS debunking challenge and calls for increased funding for open science workshops, without detailing how incentive structures would be redesigned.
• Test the Theory of Change empirically. The proposed feedback loop from mandated reproducibility to a higher Signal-To-Noise Ratio, better governance information, and more effective policy is presented as a conceptual model rather than a measured outcome; the paper also notes unresolved trade-offs in speed, researcher burden, and compute cost.

Target Audience

AI policymakers, regulators, and civil servants who rely on AI research evidence; AI researchers and reviewers concerned with methodological standards; conference and journal organisers who set publication requirements; and metascience, open-science, and research-funding specialists interested in how lessons from economics, cancer biology, and psychology translate to machine learning. The paper is written at an accessible level and does not require a technical AI background.

Authors’ abstract

AI policymakers are responsible for delivering effective governance mechanisms that can provide safe, aligned and trustworthy AI development. However, the information environment offered to policymakers is characterised by an unnecessarily low Signal-To-Noise Ratio, favouring regulatory capture and creating deep uncertainty and divides on which risks should be prioritised from a governance perspective. We posit that the current publication speeds in AI combined with the lack of strong scientific standards, via weak reproducibility protocols, effectively erodes the power of policymakers to enact meaningful policy and governance protocols. Our paper outlines how AI research could adopt stricter reproducibility guidelines to assist governance endeavours and improve consensus on the AI risk landscape. We evaluate the forthcoming reproducibility crisis within AI research through the lens of crises in other scientific domains; providing a commentary on how adopting preregistration, increased statistical power and negative result publication reproducibility protocols can enable effective AI governance. While we maintain that AI governance must be reactive due to AI's significant societal implications we argue that policymakers and governments must consider reproducibility protocols as a core tool in the governance arsenal and demand higher standards for AI research. Code to replicate data and figures: https://github.com/IFMW01/reproducibility-the-new-frontier-in-ai-governance