Designing Incident Reporting Systems for Harms from General-Purpose AI

AI summary

Overview

Research area: AI governance and AI safety policy — specifically the institutional design of incident reporting systems for harms caused by general-purpose AI (GPAI), including large language models (LLMs), with a focus on the United States.

Technical level: Beginner-Friendly. The paper is a conceptual and policy-analysis work. It contains no model training, benchmarks, or quantitative experiments; its substance is a framework, a literature review, and comparative case studies of existing regulatory regimes.

Scope: The paper defines seven dimensions of incident reporting system design and reviews incident reporting in nine US safety-critical industries to extract design considerations for AI incident reporting, limited to safety and rights incidents caused by GPAI.

What This Paper Is About

As GPAI systems spread, they are producing real-world harms — the paper cites a $25.6 million financial scam, assistance in planning explosives attacks, explicit deepfakes, accidental deletion of an entire company's code, demonstrated blackmail and deception capabilities, and election disinformation in the US and worldwide. The authors argue that pre-deployment governance such as audits will not prevent all incidents, because complex systems make severe incidents inevitable over time and emergent LLM capabilities can create unanticipated incident types after deployment. Their goal is to fill a gap in the AI governance literature: little guidance exists on the institutional design of AI incident reporting systems — how organizations, rules, and norms should be constructed to shape tasks and responsibilities among actors.

Key Contributions

1. A seven-dimension framework for institutional design. The authors define incident reporting systems across: policy goal (safety learning or accountability for harm); actors submitting and receiving reports; type of incidents (safety, rights, or security); level of risk materialization (hazards, situations, near misses, or harm events); enforcement of reporting (voluntary or mandatory by law); anonymity of reporters (open, confidential, or anonymous); and post-reporting actions (information sharing, information disclosure, audit, or regulatory action).

2. Clarification of two frequently confused dimensions. The paper singles out the level of risk materialization, presenting a lifecycle from hazards to situations to near misses to harm events, and the actors dimension, mapping which sets of actors participate in different system types.

3. Nine case studies of US safety-critical industries. The authors review incident reporting systems in nuclear power, aviation, pesticides, pharmaceuticals, cybersecurity, dams, rail, workplace safety, and healthcare, and classify specific systems against the framework.

4. Systematic design considerations for AI incident reporting. The paper extracts lessons on regulatory versus non-regulatory system operators, near miss reporting, mandatory thresholds versus voluntary channels, enabling safety learning after reporting, information sharing, and clarifying legal frameworks. The paper describes this as the first systematic examination of design considerations for AI incident reporting in the US.

Main Findings

• Third-party AI databases are insufficient on their own. The Artificial Intelligence Incident Database (AIID), the AI, Algorithmic, and Automation Incidents and Controversies Repository (AIAAIC), and the AI Vulnerability Database (AVID) are voluntary, crowdsourced databases that lack stakeholder buy-in and often lack the information needed for safety learning. Eight of the nine industries examined (all except dams) have implemented reporting regimes that go beyond independent databases. Most top contributors to the AIID are from civil society.

• Policy goals can conflict. Learning and accountability often point toward opposing design choices — for example, a voluntary learning-oriented system would be undermined if reporters feared fines. Pursuing both may require creating multiple single-goal systems, such as an FAA-style voluntary information sharing system for learning plus a separate citizen reporting system enabling regulatory audits.

• Information is dispersed across actors. Coverage — the audiences from whom reports are solicited — matters because incident information is often spread across parties. One emergency room study found healthcare provider reports most often helped identify new incidents, but reports by patients, families, and hospital risk management also helped.

• Regulators versus non-regulators behave differently. Regulators generally oversee reporting by companies, industry employees, citizens, third parties, or product users, and usually oversee reporting when industry actors are legally mandated to report. Systems reporting to non-regulatory agencies are generally non-punitive and oriented toward learning, and because such agencies cannot punish reporters they may engender more trust. Regulators may also have perverse incentives to avoid soliciting reports that reflect poorly on them.

• The ASRS is the cited gold standard for voluntary reporting. The FAA outsourced administration of the Aviation Safety Reporting System (ASRS) entirely to NASA. Because NASA does not regulate airlines, this helped promote trust, and the system can guarantee reporters anonymity and some liability protections.

• Multiple systems can raise coverage. The FAA maintains no fewer than eight distinct voluntary reporting programs for industry employees, covering dispatchers, air traffic controllers, pilots, flight attendants, technicians, airport employees, and FAA employees. The IAEA runs three separate systems for nuclear power plants, fuel cycle facilities, and research reactors. Pharmaceuticals uses one primary system (MedWatch) with many reporting formats for different parties.

• Information flows between systems need design. Options include a "federated" regime with a unified reporting standard across government, industry, and non-profits, or a more centralized regime such as nuclear or aviation. The US Coast Guard's National Response Center offers a single unified point of contact that re-routes reports to relevant agencies.

• The pesticide regime shows what high coverage looks like. Pesticide reporting and data sharing involves agriculture workers, poison control centers, doctors, medical labs, hospitals, most US state governments, the Department of Agriculture, the Environmental Protection Agency, the Bureau of Labor Statistics, the World Health Organization, the Intergovernmental Forum on Chemical Safety, and the UN Food and Agriculture Organization.

• Overlapping reporting systems corroborate each other. A study of wildlife collisions with commercial aircraft estimated that 91% of such incidents were reported via FAA incident reporting systems, but at least 10 other systems also contributed reports. Air Traffic Organization employee reports comprised 7% of all reports and supplemented another 7% that were duplicated from other sources.

• Mandatory reporting typically targets high-severity harm events. Mandatory reporting generally applies to harm events such as death or serious injury. Government-mandated reporting thresholds have improved regulatory visibility in industries such as pharmaceuticals and civilian aviation. Mandatory requirements alone may be insufficient for safety learning, and voluntary near miss reporting can be significant.

• Clear thresholds and definitions are a practical prerequisite. Reporting systems usually need clear, well-scoped thresholds and definitions to be useful. As of July 2025, no consensus definition for AI "incidents" has emerged.

• Incident reporting is a first step, not the endpoint. Subsequent analysis, mitigations, and monitoring are needed to enable safety learning. Incident investigation and analysis remain immature in AI, and it is unclear whether independent databases can facilitate these practices within developers or regulators.

• Anonymity is context-dependent. Anonymity can be warranted to promote safety learning, or when the policy goal is accountability and reporting parties fall outside the chain of incident responsibility — but not in all systems.

• Legal frameworks can support or block reporting. Policymakers can clarify legal frameworks up front, set clear reporting incentives, communicate guidelines to potential reporters, and close legal loopholes that permit avoiding reports.

• Current AI mandates are narrow. As of November 2025, China, the EU, California, and South Korea are the only jurisdictions that have adopted GPAI-specific incident reporting mandates. China's August 2023 generative AI regulation requires service providers to create user reporting channels and to report "illegal content" to the government. The EU AI Act will require AI service providers to report "serious incidents" to national regulators. California's SB-53, enacted September 2025, requires frontier developers to report certain safety and security incidents to the state and creates a public reporting hotline. Non-governmental initiatives include the AIID, AIAAIC, and AVID, while the OECD's Expert Group on AI Incidents launched the "AI Incident Monitor" in 2023.

Methodology in Plain English

The authors use a three-step case study approach adapted from prior work by Raji et al. (2022b), Ayling and Chapman (2022), and Stein et al. (2024). First, they select nine safety-critical industries with robust incident reporting regimes, identified through seed articles. Second, they conduct a background literature review of incident reporting in those industries alongside the AI governance literature, and from that they build the seven-dimension framework. Third, they purposively select specific incident reporting systems from the nine industries and classify each one against the framework, using these classifications to identify best or common practices and to reason about when particular design choices are appropriate for AI. Table 3 in the paper presents the classification of each system, with hyphens marking where no publicly available information was found.

The stated scope exclusions are notable: the paper covers institutional design only, not the design of AI systems themselves or operational-level details of incident reporting. It covers safety and rights incidents caused by GPAI, not security incidents in which GPAI systems could be compromised by external actors, and it excludes AI whistleblowing.

Why This Matters

Impact on research. The paper provides a shared vocabulary for comparing AI incident reporting proposals that currently use conflicting definitions and design choices. It links AI governance to decades of established practice in other safety-critical industries, giving researchers a structured basis for evaluating whether existing AI incident databases can support safety learning or accountability, and where the gaps are.

Real-world applications:

• Designing regulatory reporting mandates for GPAI, including setting reporting thresholds and deciding which incidents are mandatory versus voluntary.
• Standing up voluntary near miss reporting channels for industry employees, users, third parties, and citizens.
• Building internal complaint intake and investigation systems at AI model developers and service providers.
• Structuring information sharing and routing across multiple reporting systems, including standardization and interoperability, so reports reach the right stakeholders.

Industry relevance. AI developers and service providers are directly implicated: the paper suggests they may benefit from internal systems for accepting and investigating complaints from product users. Regulators at the federal, state, and local level, incident database operators, and insurers also have a stake — the paper notes that incident reporting can support risk estimation for insurance policies, evaluations of existing safety measures, and real-world system feedback.

Future Directions

• Is aviation the right model for AI? Some commentators have called for voluntary near miss reporting systems for AI like those in US aviation, but the paper points out it is unclear whether aviation is an appropriate model because the same model has failed in the rail industry.

• Defining and thresholding AI incidents. No consensus definition of AI "incidents" existed as of July 2025. The paper's own working definition is described as not necessarily ideal or operationalizable as-is, leaving open how to build practically useful definitions and reporting thresholds.

• Closing the analysis gap. Incident investigation and analysis remain immature in AI, and it is unclear whether independent databases can facilitate these practices within AI developers or regulatory agencies. How to build that capability is unresolved.

• Designing multi-system and federated regimes. With several possible system types — regulator-facing, non-regulator-facing, citizen hotlines, employee near miss channels — open questions remain about how information should be aggregated, routed, standardized, and made interoperable, and whether a federated or centralized regime is more appropriate for AI.

Target Audience

US researchers and policymakers considering incident reporting as a mechanism for mitigating harms from GPAI will benefit most, along with AI governance scholars, AI developers and service providers designing internal complaint and safety processes, operators of AI incident databases, and analysts comparing AI oversight to established regimes in aviation, nuclear, pharmaceuticals, and other safety-critical fields.

Note: The full paper content provided for this summary is truncated mid-way through Section 4.2. Sections covering the remaining design dimensions, additional case study results, and the paper's concluding discussion are not included in the available text, so findings from those parts are not summarized here.

Authors’ abstract

We introduce a conceptual framework and provide considerations for the institutional design of AI incident reporting systems, i.e., processes for collecting information about safety- and rights-related events caused by general-purpose AI. As general-purpose AI systems are increasingly adopted, they are causing more real-world harms and displaying the potential to cause significantly more dangerous incidents - events that did or could have caused harm to individuals, property, or the environment. Through a literature review, we develop a framework for understanding the institutional design of AI incident reporting systems, which includes seven dimensions: policy goal, actors submitting and receiving reports, type of incidents reported, level of risk materialization, enforcement of reporting, anonymity of reporters, and post-reporting actions. We then examine nine case studies of incident reporting in safety-critical industries to extract design considerations for AI incident reporting in the United States. We discuss, among other factors, differences in systems operated by regulatory vs. non-regulatory government agencies, near miss reporting, the roles of mandatory reporting thresholds and voluntary reporting channels, how to enable safety learning after reporting, sharing incident information, and clarifying legal frameworks for reporting. Our aim is to inform researchers and policymakers about when particular design choices might be more or less appropriate for AI incident reporting.