The safety debate that erupted when whistleblowers last week warned that the world’s most advanced AI poses an existential threat to humanity should come with a health warning: side effects include whiplash. Just seven days ago, AI enthusiasts were having fun playing with OpenAI’s latest GPT-6 Astra, which was marketed in part as a lifestyle tool to help people book tennis courts, run fashion companies and order takeout.
By Wednesday, advanced AI began to look much less attractive, with retiring anthropology researcher Jacob Coxon warning that “the people making AI honestly believe it could kill us all by the end of the decade.” Yes, he was talking about artificial superintelligence—an as-yet-defunct version of technology that outperforms humans in most areas—but his comments, and the chorus of experts who echoed him, frayed many nerves.
Another shift came when Anthropic said it had found that users were evading controls and using existing models “in ways that support the development of biological weapons”—think new mosquito-borne viruses or bird flu. All this was enough for lawmakers in London and Washington to demand brakes and bans on the most potentially dangerous types of AI development.
And then there was another dramatic turnaround this weekend in the form of a three-point plan from Anthropic boss Dario Amodei to “get past the line.” Putting aside the fact that his own employees had caused border panic just days earlier, Amodeus was here to save us from fears of AI Armageddon, but not before he issued his own juicy warning that rising capabilities meant a swarm of AI agents “could take over the entire Internet” in six to 12 months. Skeptics doubted this claim, but for some it was enough to declare it a “Code Red” moment.
In a sign that a coordinated slowdown may be more than just talk, some of Amodei’s precautionary ideas quickly gained support from his rivals OpenAI’s Sam Altman and SpaceX’s Elon Musk. There may have been reasons for hope after a difficult week: at least artificial intelligence companies were talking about slowing down before anyone got seriously hurt.
In short, Amodeus proposed three moves. First, every US AI company provides ongoing access to built-in third-party evaluators to verify compliance with security obligations, report incidents, and ensure the correctness of new AI models. Until now, such monitors have only been used at the behest of companies or given limited access to systems to investigate when something goes wrong. In the absence of any independent regulator at government level, this is just the beginning.
Second, he wants all companies in democracies creating advanced AI models to set common safety standards, as well as limit the speed of uncontrolled AI progress.
Third, the world’s democratic AI powers will coordinate with autocracies—especially China—to control race. This will probably be the most difficult step. Amodei suggested that a first step could be a narrow agreement banning clearly dangerous uses of AI, such as biological weapons. Donald Trump’s meeting with Xi Jinping in Washington on September 24 will test the prospects for any Sino-American cooperation.
The immediate political problem is that Donald Trump seems reluctant to admit to any concerns about AI. Amodei’s plan requires action by the US government, and Trump said last week he had no concerns about AI leading to human extinction. On Sunday, he doubled down, saying people are “talking about things that won’t happen.” Losing to China appears to be a big concern in Trump’s AI calculations.
“There will be no day after tomorrow if China wins on this issue,” Scott Bessent, the US Treasury secretary, said last week. “If they turned their backs on us on AI, then nothing else would matter.”
People close to Trump also question why Amodeus and the rest of the AI leaders can’t accelerate AI development themselves.
“The easiest way not to create superintelligence is to agree not to create it,” said David Sachs in response to Amodei’s blog. Sachs is co-chair of Trump’s Council of Science and Technology Advisors.
after promoting the newsletter
“Demanding your preferred regulatory framework because the cost of doing so would look like blackmail of the public and the political system,” he told Amodei.
Another criticism of Amodei’s call to “slow down the pace” of AI progress to ensure safety measures and guardrails are respected came from Professor Stuart Russell, one of the world’s leading experts in AI. Amodei suggested slowing down the pace of growth to free up time to improve systems and ensure AI is aligned with human interests. But Russell said it was “quite the opposite.”
“We’re not just setting a slower pace of capability development and then hoping that gives enough time to get the security right,” he said. “We set safety requirements, and further progress only happens when they are met. Imagine if a pharmaceutical company said, ‘We’re going to come out with a new cancer drug every year, and we hope that gives us enough time to complete some clinical trials and get good results.’
AI experts who don’t work in big labs have also been suspicious of a plan to create a global regulatory framework for artificial intelligence, developed by the leader of the world’s most valuable artificial intelligence company.
“Too little, too late,” said David Kruger, an AI professor, safety campaigner and former founding director of the UK government’s AI Safety Institute. He added: “We need an immediate, indefinite international moratorium on advanced artificial intelligence development.”