The Pulse
Hinton, Bengio Warn AI Research Could Accelerate Beyond Human Control
A Cambridge Programme on AI Science & Policy working paper, co-authored by Geoffrey Hinton and Yoshua Bengio, says AI could automate most AI research and development within a few years. It urges governments to track that automation and prep
AI.info Team · Written with AI · fact-checked ·
Automation promises speed; the report warns of lost control
AI systems could help deliver medical breakthroughs and other advances sooner. The same acceleration could leave governments and the public struggling to keep pace, according to a working paper released Monday by the Cambridge Programme on AI Science & Policy (CASP). Its authors, including Geoffrey Hinton, Yoshua Bengio, OpenAI chief scientist Jakub Pachocki and Anthropic co-founder Jack Clark, argue that automating AI research and development could compress years of progress into months or less.
The report describes a possible feedback loop, not one already under way: AI systems take on more work developing AI, then contribute to building more capable systems that can take on still more research. The authors write:
“Although there remains much uncertainty about these possibilities, the high stakes warrant serious further attention.”
Alan Chan, Sören Mindermann and co-authors including Geoffrey Hinton and Yoshua Bengio, “What if automating AI R&D triggers an intelligence explosion?”
The authors say the threshold has not arrived
The paper’s abstract says AI systems are on track to automate most AI R&D work within a few years, and possibly all of it. That forecast sits alongside an important qualification: the authors write that productivity gains from AI R&D automation have not yet reached the threshold needed to trigger an intelligence explosion, though gains from newer systems are “likely approaching that threshold”.
That distinction separates a warning about a plausible future from evidence of an ongoing runaway process. The report points to AI’s growing role in work at the companies that build these systems, saying AI now writes most of the code inside those companies; it cites Anthropic’s figure that AI systems’ share of approved code rose from low single digits to over 80% between January 2025 and May 2026. But writing code is not the same as independently directing research or deciding which results are valuable; the concern is how much more of the research pipeline systems may take on.
Faster progress could bring faster risks
The authors set out both sides of the possibility. Automated research could bring forward AI’s benefits, but a sharp rise in capabilities could outpace society’s capacity to respond. The report warns of possible loss of human control over superhuman systems and of weakened checks on power within and between states, companies and branches of government.
The paper also warns that an intelligence explosion could bring forward AI-enabled biological and cyber attacks, leaving society less time to adapt where threats accelerate faster than the measures needed to counter them, and that a state could turn a modest lead in military R&D or operations, such as in cyberspace, into a decisive one. Those outcomes are scenarios, not claims that such harms are already occurring. The authors emphasize uncertainty while arguing that the stakes justify preparation before the feedback loop becomes stronger.
CASP calls for visibility and ways to intervene
The report urges policymakers to obtain better visibility into how AI is being used for AI research, develop ways to steer or constrain an intelligence explosion, and prepare society for its effects. Its proposed steps include standardized reporting of AI R&D indicators to governments and third-party auditors, independent evaluators embedded in some AI companies, tools to verify compliance with future agreements that pace AI progress, and options, worked out with data center operators, to pause specific AI R&D workloads.
Those recommendations address the gap between noticing acceleration and retaining the ability to respond. The report’s central policy claim is that governments should build that visibility and capacity while the process remains uncertain—not wait until AI systems can complete more of the research needed to improve their successors.