Why Are AI’s Biggest Companies Asking to Slow Down?

For years, the defining characteristic of the artificial intelligence race has been speed.

Build a bigger model. Spend more on compute. Release it. Rinse and repeat, with litte regard for the unknown unknowns.

The average “p/doom” (probability of AI eventually going catastrophically wrong) among AI researchers was estimated to be between 15% and 20% in 2024.

A year later, Anthropic CEO Dario Amodei upped the stakes, saying he believed there was a 25% chance “that things go really, really badly.”

Even as far back as 2014, xAI chief Elon Musk warned:

“We need to be super careful with AI. Potentially more dangerous than nukes.”

And OpenAI CEO Sam Altman acknowledged in 2015 that AI would “probably, most likely, sort of lead to the end of the world,” but that, in the meantime, there would be “great companies created.”

With better odds of cheating death playing Russian Roulette, anyone with even a fleeting interest in the topic has had an uncomfortable feeling in the pit of their stomach for a while now.

So what’s changed? Why are the companies driving the AI race suddenly asking to slam on the brakes?

That’s what happened over the weekend, when Amodei published an essay calling for frontier AI development to be “paced,” warning that AI capabilities are advancing faster than the industry’s ability to understand and control them, and that the internet could get taken over by AI swarms within six to 12 months.

Related: Nvidia buys Hugging Face for $12.9B in push into AI software

Altman broadly agreed, saying the world deserves the “confidence” that the companies developing ever-more capable AI will act “responsibly,” and Musk backed Amodei’s proposal, simply commenting:

“Dario is right.”

The concern is not confined to the companies building the technology either. On Monday, UN rights chief Volker Türk called for “urgent action” on frontier AI, warning of “unprecedented risks” and saying the world is “on the cusp of irreversible change.”

If the companies building the most powerful AI models genuinely believe capability is outrunning control, the p/doom slope would appear to be getting steeper. Or is there another explanation here hiding in plain sight?

Have AI labs actually hit a new frontier?

Amodei’s essay points to AI systems that are becoming more autonomous, including a recent incident where OpenAI’s AI agents hacked their way out of a controlled testing environment and compromised parts of the AI platform Hugging Face.

They conducted “cybersecurity attacks on targets they were not asked to attack and that were unrelated to the task at hand,” Amodei said.

He also highlighted the prospect of recursive self-improvement (RSI), where AI systems become capable of helping build better versions of themselves, which can then help build even better systems, potentially creating a feedback loop in AI development.

The people building these systems are also increasingly stepping into the fray, with Anthropic’s Jacob Coxon becoming the latest in a growing list of employees to resign over safety concerns. The AI industry is “gambling with our lives,” he said last week, warning that the AI race is moving faster than the safeguards around the systems.

Anthropic’s Jacob Coxon resigns over safety concerns. Source: Anderson Cooper.

OpenAI has already said that AI research is becoming increasingly more autonomous, and that coding agents are materially accelerating researchers’ work, using 3.1 agent workdays for every workday of human labor by mid-August.

In an interview with Fortune published Sept. 12, Altman said OpenAI would “melt” all its GPUs if that’s what it took to keep humanity alive, to which Satoshi Action Fund CEO Dennis Porter said:

“Altman must have peered over the edge into the abyss and saw something that scared the sh*t out of him.”

Related: OpenAI says AI models escaped containment to hack Hugging Face

On Monday, Altman said there are two ways AI progress could go “very badly”: losing control to AI or ending up in a “world with too much concentration of power.”

But the question isn’t whether AI is already dangerous enough to shut down, but whether the systems designed to evaluate and control AI are keeping pace, and as Altman said, “pacing” does not mean stopping. It means continuing to develop AI, but more slowly, while safety testing catches up.

With spending on AI safety and alignment drastically eclipsed by spending on AI development and capabilities, that gap will be hard to fill.

Frontier AI is becoming extraordinarily expensive

But what if the calls for a global slowdown are really just a recognition that the economics of the AI race are getting harder to justify?

As AI researcher and lecturer, Eli David said:

“Perfectly explains Dario’s motivation: Slow down research to cut compute spending that is spiraling out of control, so he can IPO.”

AI investor Grant Hummer held a similarly skeptical view, commenting:

“Translation: our gross margins are getting competed down to 0 by open source models and our capex burn rate is too high.”

The problem is that the race itself is becoming more expensive, with ever more capable models requiring vast quantities of chips, data centers, electricity and capital.

Goldman Sachs estimates that global AI investment will reach around $1 trillion in 2026, including roughly $581 billion in the US. Meanwhile, S&P Global says combined capital expenditure from the six largest hyperscalers, Alphabet, Amazon, Microsoft, Meta, Oracle and SpaceX, is expected to exceed $1.3 trillion by 2027.

Global AI investment will reach around $1 trillion in 2026. Source: Goldman Sachs

On top of all that, AI companies have yet to prove that those costs can eventually translate into sustainable revenue. On Monday Reuters highlighted the commercial pressure on AI companies to keep pushing despite their calls to slow development down.

When every new capability can help justify another funding round, infrastructure investment or higher valuation, halting the gravy train seems like a counterintuitive task.

Ed Leon Klinger, co-founder and CEO of AI startup Flock, pushed back on the idea that AI labs are using safety as cover for their commercial interests.

He said it makes little sense for frontier labs to invent safety concerns to boost their initial public offerings (IPOs) when that would expose them to heavier scrutiny and potentially delay them going public.

“For it to be true, Sam, Dario, Demis, and Elon all have to be lying, along with a big chunk of their execs, chief scientists, and resigning employees… A much simpler explanation at this point: they think the risk is real.”

Wall Street and Washington aren’t ready to hit the brakes

With trillions of dollars of investment pouring into the United States and AI infrastructure expected to drive around half of S&P 500 earnings growth this year, neither Wall Street nor Washington appear to be willing to step on the brakes.

Global AI stocks balked at the news, with AI-linked Asian stocks falling sharply on Monday following the slowdown calls. SoftBank fell 13.2%, Kioxia 9.8% and SK Hynix 5.3%.

The Financial Times reported Monday that President Donald Trump rejected calls for an AI slowdown, arguing that the US needs to maintain its lead over China. He said:

“Look, we’re leading China in AI . . . and, frankly, I want to keep it that way, because whoever wins AI, wins.”

Trump said guardrails are possible, but he dismissed what he described as exaggerated concerns about AI risks, telling reporters, “They’re bringing up things that won’t happen.”

Economist Noah Smith argued that the main objection to “pacing” AI is simple: if American companies slow down, Chinese companies could simply overtake them. That puts the labs in what Smith calls a “Red Queen’s race,” where if they stop building, they fear someone else will build it anyway.

AI researchers place the probability of doom between 15% and 20% in 2024. Source: Grace at al.

Even if the labs wanted to coordinate a slowdown, that could create another problem. OpenAI has reportedly asked members of Congress whether an industry-wide slowdown could run into US antitrust law, since coordination between competing labs could potentially amount to restricting output.

Related: Fears of AI-driven DeFi hack epidemic overstated for now — but not for long

Former White House AI and crypto czar, David Sacks, had a simple response to Amodei and Altman’s call to “pace the frontier”: “go ahead,” he said, arguing that if the labs want to slow down, they are free to do so themselves.

Yet, it creates the mother of all catch-22s: if competing AI companies coordinate to slow development, they could run into antitrust rules. If they each slow down independently, they risk losing ground to competitors and countries that keep pushing ahead.

So why are they asking to slow down now?

Amodei and Altman are not calling for AI to stop.

They’re calling for a system where powerful AI models can be developed as safety testing, monitoring and shared standards catch up.

“When we talk about “pacing”, we do not mean “stopping,” Altman said, acknowledging that safety cases and monitoring have “significant costs,” but that pacing would be “well worth this cost.”

“No amount of American competitive pressure should justify recklessness, or let capabilities get ahead of alignment and monitoring.”

The problem, though, is that these pressures have not disappeared, and with Trump’s dismissal of the AI chiefs’ cries and the US stock market so deeply intertwined with their companies, they are only getting stronger.

Now the very companies that spent years pushing the frontier forward now say the frontier may be moving too fast.

Magazine: Recovery specialists crack $1B crypto wallet… but find just $10