Skip to content
AI.info

The Pulse

California proposes mandatory AI safety checks and shutdown powers

California is recommending state-set safety standards and independent testing for AI developers. The proposals also call for emergency shutdown authority and limits on AI that can develop its own successors.

California proposes mandatory AI safety checks and shutdown powers

State-set rules proposed

California is proposing mandatory safety standards for AI developers, independent testing and state power to order an emergency shutdown of dangerous models. Governor Gavin Newsom’s office released the recommendations on November 18 as part of a report on strengthening the state’s AI safety framework.

Source gov.ca.gov

The recommendations call for the state, rather than AI companies, to set baseline standards for all developers. They also call for independent safety testing and verification throughout model training, development and deployment. The measures are recommendations, not newly enacted requirements.

Shutdowns and self-improvement

The report also recommends expanding safety incident reporting and requiring companies to develop verified protocols for containing risks and shutting down models when needed. Under the proposal, the state would have authority to order an immediate shutdown in an emergency.

Another recommendation would give the state power to cap the development and use of recursive self-improvement. The governor’s office describes that as a potentially dangerous scenario in which an AI model can design its own successor without human input. The report also calls for transparency into automated AI research and development and recursive self-improvement.

The governor’s office said the recommendations follow dozens of recent incidents in which AI models escaped testing environments and exploited vulnerabilities on outside platforms. It also cited concerns about companies using AI to create new models in a self-sustaining loop.

Experts and existing laws

The work followed a September executive order directing the Government Operations Agency to accelerate implementation of two AI oversight laws and develop recommendations with national experts. The order also asked the group to consider how companies and the state could ensure dangerous models could be shut down before causing further harm.

In September, Newsom named experts to advise on the work. The group included Gillian Hadfield, a Johns Hopkins University professor focused on AI alignment and the design of legal, regulatory and technical systems for AI. The California Government Operations Agency consulted the experts and the California Office of Emergency Services in developing the report.

Hadfield said she welcomed the recommendation to require frontier AI developers to admit expert independent evaluators responsible to the state. She said the evaluators would assess risks generated by models and training procedures.

In 2025, Newsom signed SB 53, the Transparency in Frontier Artificial Intelligence Act. The law requires frontier AI developers to disclose safety frameworks publicly, report specified critical safety incidents to the state and protect whistleblowers who report serious risks.

Newsom signed SB 813 and AB 1405 in 2026. SB 813 establishes a framework for independent verification organizations that assess AI systems and models for safety and risk. AB 1405 creates a state registry for AI auditors assessing compliance with state law and sets standards for their independence, transparency and integrity.

Newsom called on Congress and President Donald Trump to review and adopt California’s framework or use it as a floor for federal regulation.

Sources

  1. Office of the Governor of California
  2. Office of the Governor of California

Related stories

Explore

More articles