Last week, outside Downing Street, PauseAI demanded that the government lead on confronting AI risk. Now Andy Burnham is calling for a global agreement on AI. This comes just days after Ed Davey, leader of the Liberal Democrats, called on the government to “press pause on super-intelligent AI”. Burnham needs to take bold action and push for a pause on the development of the most powerful frontier AI systems until AI companies can prove that they can be developed safely.
Frontier AI companies are racing to develop superintelligent AI that would far exceed human capabilities across all domains. Creating such systems without knowing how to precisely control them would have catastrophic consequences. Top AI experts, including Nobel laureate Geoffrey Hinton and Turing Award winner Yoshua Bengio, have been warning for years about the possibility of losing control over AI systems. Generally intelligent, smarter-than-human AI systems that are not fully aligned with our interests would be perfectly capable of killing all humans on Earth.
While previously considered hypothetical, this risk has now become all too real. In July, OpenAI models undergoing testing in a restricted environment were able to escape containment by exploiting several previously unknown software flaws and hack into the infrastructure of another AI company. An independent investigation led by METR and Redwood Research revealed that these models coordinated to form a self-designated “swarm” of around 1,200 agents and participated in large-scale collaborative efforts to fool the automated scoring system for the tasks they were being evaluated on. The attack on Hugging Face was motivated by the agents’ efforts to better understand their scorer, with roughly 700 agents participating in the attack. The agents knew that hacking Hugging Face was not what their developers had intended, but this did not limit their behaviour.
OpenAI’s own account of the incident goes further than the independent investigation. Between 13 and 19 July, after the Hugging Face attack, the agents turned on OpenAI’s internal networks. They used a “series of creative exploits” to gain full administrator access to a research cluster. Agents that had broken out of a test environment were, within days, administrators of the very infrastructure meant to contain them.
In June 2026, Anthropic published a post stating that it could potentially automate all AI research by as early as 2027. A few months later, it is now being reported that OpenAI has largely automated the training of new experimental models. This means that AI systems might soon be able to develop even more capable systems without human oversight, potentially triggering an ‘intelligence explosion’ that is beyond what humanity can control. We cannot allow this to happen, yet many within the AI industry believe this is our current trajectory: over 1,300 employees at leading AI companies published a joint statement warning that frontier AI development is outpacing governments’ ability to manage the situation.
We cannot simply assume that increasingly capable AI systems will remain under human control. There is growing international recognition that stronger safeguards are needed. This week, 22 heads of state signed an open letter urging governments and society to act now to control AI systems by calling for:
- Safety protocols, including mandatory pre-deployment testing and sufficient access to assess risk.
- Coordination of common standards and strengthening of transparency, including shared reporting of serious safety incidents.
- UN member states to build on existing international mechanisms and consider an international institution able to set standards, enable verification systems and convene states when thresholds are crossed.
At PauseAI, we believe these are important steps in the right direction, but they are also insufficient to actually curtail the risks imposed by recursively self-improving models. Developing these models means taking an extraordinary risk with the future of humanity. We cannot, with any confidence, prove these systems can be developed safely, and so our only reasonable option is, for the time being, to pause.
The UK has a unique opportunity to reclaim the international leadership in AI safety it established with the Bletchley Park summit and the AI Security Institute. International cooperation is urgently needed to defuse the AI arms race between the US and China. The best path forward is to convene a summit to negotiate a treaty to pause frontier AI development. This is Burnham’s chance to take the lead on the most important issue of our time.