Does the artificial intelligence industry need to slow down?

An Anthropic researcher recently went viral for quitting his job, warning his employer (and its rivals) are racing to build systems they can’t control – systems he warns could “kill us all by the end of the decade”.

Tech CEOs say they agree with him, so what does that mean for the future of AI, and how has the rest of the world responded?

We’ll unpack it all in today’s deep dive.

Last week, 27-year-old British researcher Jacob Coxon posted to X declaring he was leaving the AI industry behind, warning tech companies “are racing straight to self-improving superintelligence and gambling with our lives.”

“The people building AI earnestly believe that it could kill us all by the end of the decade,” he wrote. “No other human activity poses this level of danger.”

Coxon worked at Anthropic training AI models, including its well-known chatbot Claude. Prior to that, he worked at OpenAI (which owns ChatGPT). He said he wanted to join Anthropic for its safety reputation.

Comparing employers, Coxon said: “At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first – they believe no one else will act responsibly”.

In an interview with the Wall Street Journal (WSJ) released around the same time as his posts, Coxon added: “By the end of next year things could be out of control already”.

Coxon’s comments prompted an outpouring of concern from his contemporaries, including safety researcher Anna Wang, who currently works at Anthropic after a stint at Google.

She said Coxon’s sentiment is “common” among her peers: “There is not yet a viable scientific plan to solve risks from recursively self-improving AI.” (More on that later.)

Evan Hubinger, who leads an Anthropic safety team, posted online: “Jacob is correct here — we really do earnestly believe AI could kill all humans! I personally think it is [a greater than 10% chance] within the next decade.”

He added: “I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”

Concern from experts like Coxon relate specifically to something called recursive self-improvement, or RSI – where an AI system helps build, train, or write the code for its own next generation, cutting out the need for human engineers.

Researchers warn that once this loop speeds up, oversight becomes harder.

These concerns were realised in July, when OpenAI admitted its own AI agents had broken out of a testing environment and hacked into a major developer platform without being instructed to. In response, more than 1,000 AI researchers signed an open letter urging global government coordination to slow AI development.

But how could AI… kill us?

“In one scenario hypothesised by researchers, a malevolent AI system could spread a secret bioweapon and trigger it with a chemical spray. Or it could trick two nuclear powers into war,” he wrote.

Other doomsday scenarios range from (deliberate or accidental) AI-powered cyberattacks and bioweapons to AI-driven espionage or economic collapse.

Anthropic CEO Dario Amodei responded to these mounting concerns in an essay published last weekend, titled: “We Must Pace the Frontier”.

His central argument is that “we must slow the pace at which we improve the capabilities of AI models,” adding that “progress will still seem fast, and we must make wise use of the time we gain.”

He said RSI has already taken hold “across the industry, including at Anthropic”. Left alone, he warned RSI could “outrun our ability to understand and control these systems”.

Amodei proposed a three-step plan. First, leading AI companies (his own included) should give independent, outside evaluators full access to review their practices, rather than relying on firms to self-report. Anthropic has already committed to this.

Second, he wants leading AI companies to coordinate on shared safety standards and limits on unchecked progress. This would likely require government support.

Third, he called for democratic governments to try to coordinate with “authoritarian” ones, namely China, on verifying compliance.

“The measures I propose to advance the frontier at a safe pace will not be easy,” he conceded, “but I believe we owe it to humanity to try,” Amodei wrote.

Anthropic’s rivals support these ideas – publicly, at least.

OpenAI’s Sam Altman agreed “we need to pace the frontier” and said OpenAI would match Anthropic’s move on independent evaluators; Elon Musk (who runs xAI) and Google DeepMind’s Demis Hassabis both voiced support too, without committing to specifics.

U.S. President Donald Trump has pushed back on the proposal, rejecting the idea of an outright slowdown.

He told reporters earlier in the week: “We’re leading China in AI... and frankly, I want to keep it that way, because whoever wins AI wins.”

Trump dismissed safety warnings as coming from “negative forces” pushing “things that won’t happen.” He later wrote on Truth Social that the only guardrail AI needs is “a strong and smart... president.”

White House AI adviser David Sacks told AI companies to “go ahead” and pace themselves if they’re so worried, while suggesting the push for outside help was more about limiting legal liability than altruism.

Amid all this, Anthropic co-founder Jack Clark gave an interview to the BBC raising the ideas of a mandatory, independently verifiable “kill switch” for AI systems.

Clark said most labs, Anthropic included, already have some internal way to “pull the plug” on a model. However, he argued that whether such a switch should be legally required (and checked by an outside party) is a question “society is going to want to know” and “might want to eventually pass rules around.”

The idea has already been floated in Congress. A bipartisan AI Kill Switch Act, introduced in July after the OpenAI cyber incident, would require top developers to maintain the technical ability to suspend their most powerful systems, and let the Department of Homeland Security order a shutdown if a model escapes human control.

It hasn’t passed, but momentum is building in Washington, where unlikely allies independent Senator Bernie Sanders and MAGA strategist Steve Bannon shared a stage this week to demand tighter controls on AI.

Anthropic has promised outside evaluators, OpenAI has matched that promise, and a kill switch bill remains before Congress, but no company or government has made any binding commitments.

Meanwhile, the commercial AI race is only set to ramp up. Anthropic is reportedly preparing for a public listing worth close to $US965 billion. OpenAI, valued around $852b, has paused its own plans to float this year, given the uncertainty around safety concerns.

Here in Australia, leaders are having their own AI safety conversations – centred around national security risks.

Just yesterday, Home Affairs deputy secretary Ciara Spencer told an inquiry the sophisticated search abilities of "quicker, easier, nastier" systems were targeting government officials through believable interactions.

ASIO chief Mike Burgess called for calm, but warned officials should not have highly sensitive conversations in their cars or near their phones, given foreign governments’ ability to listen in.

Drive Safe and Start Savin' with ROLLiN' Car Insurance

Safe 'n Save allows you to get a potential discount off your monthly policy based on how you and your Listed Drivers score on safe driving.*

But wait, there's more, use the code DAILYAUS10 and you can score up to 10% off your base premium for 6 consecutive months when you take out a new ROLLiN' Comprehensive Car Insurance policy by 20 October 2026^.

^{*Any Safe ‘n Save safe driver discounts available after 2 months of app use. Learn more via https://rollininsurance.com.au/safensave. ^T&Cs apply. Minimum premiums may apply. General advice only. Always read the PDS. PDS and TMD available from Rollin’ Insurance at rollininsurance.com.au.}