The Pulse
Former AI Employees Warn NYC Council as OpenAI Defends Its Safety Pledge
Former employees of Anthropic, OpenAI and Google DeepMind warned the New York City Council that advanced AI could escape human control. OpenAI representative Morgan Dwyer said she could not quantify catastrophic risk as council members weig
AI.info Team · Written with AI ·
Former AI employees told the New York City Council on Monday, October 5, that companies are pushing ahead with systems they may not be able to control. OpenAI representative Morgan Dwyer, pressed to estimate the chance of catastrophe, said she did not know the odds but argued that any such risk would be unacceptable. The exchange put two competing claims before lawmakers: former insiders warned that humanity could lose control, while company representatives emphasized safety work and left the scale of the danger unquantified.
Coxon says the AI race could outrun control
Jacob Coxon, a former Anthropic engineer, urged council members to reject a culture of moving quickly and fixing problems later. “The companies run on a startup mindset: Move fast, break things, fix them later. That works for a photo sharing app. It does not work for building the most powerful technology ever built,” he said.
“On the current path, I think it is more likely than not that humanity loses control to these AIs and it could end in human extinction,” Coxon said.
Jacob Coxon, former engineer at Anthropic, as quoted by The Associated Press
Employees who had recently left Anthropic, OpenAI and Google DeepMind described what they saw as a reckless corporate culture, according to the Associated Press. Coxon said AI could bring major benefits, but argued that without transparency and safeguards, events were more likely to spiral out of control. His forecast was a warning offered to lawmakers, not a finding established at the hearing.
Dwyer rejects a numerical estimate
City Council Speaker Julie Menin asked company representatives to quantify the risk in a worst-case catastrophic scenario. Dwyer replied that she could not provide a figure, then said the difference between a 1%, 10% or 20% chance did not change the company’s responsibility to prevent catastrophe. Menin called the response “flippant at best.”
“I would say that the percentage chance doesn’t matter. What matters is the commitment to safety, and that is what OpenAI is committed to doing,” Dwyer said. The Associated Press reported that representatives from OpenAI, Anthropic, Google and Meta also said their systems were already helping people and stressed their focus on safety. Anthropic’s Logan Graham said the company welcomed “smart regulation” and that state and local governments had a role in shaping rules.
NYC weighs validation, a kill switch and whistleblower payments
The hearing followed a package of proposed city measures announced by Menin on September 25. One proposal would require third-party validation for AI systems marketed, sold or deployed in New York City, including checks on data quality, bias, privacy and security. It would also require a human override capable of shutting down each system, with a proposed $25,000 penalty for each instance of deployment without validation or with falsified validation.
Other proposals would let whistleblowers receive a share of penalties recovered from AI companies and allow people to sue companies over foreseeable harm caused by third parties exploiting inadequate safeguards. The package also includes proposed reporting requirements for AI safety incidents involving city contracts. The Council had announced that the October 5 hearing would convene all 51 members and include testimony from major AI firms; it used the prospect of subpoenas to secure several companies’ participation.
The warning now meets a legislative test
Former employees asked lawmakers to treat loss of control as a serious possibility. Company representatives said safety matters but did not offer a shared estimate of how likely a catastrophe might be. That disagreement gives the Council’s proposals a practical test: whether city rules can require measurable checks and clear responsibility without treating the former employees’ worst-case forecast as an established outcome.
The measures remain proposals for Council consideration, not enacted city law. The October 5 hearing brought company representatives and former employees into the same public debate, while leaving lawmakers to decide whether the proposed validation, shutdown and whistleblower provisions should become binding requirements.