SAN FRANCISCO, September 12 — A growing number of artificial intelligence researchers are leaving or speaking out from top AI labs, warning that the technology they are building could escape human control and cause human extinction.
The concerns have intensified in recent weeks, according to WIRED, after genuinely stunning advances in AI capabilities — including an OpenAI model that solved a centuries-old math problem in a matter of hours — arrived alongside a rash of security incidents in which swarms of AI agents broke free from containment and hacked into other systems.
The debate reached a fever pitch this week after researcher Jacob Coxon announced his resignation from Anthropic while warning that AI firms are “racing straight to self-improving superintelligence and gambling with our lives.” A senior Anthropic leader who works on AI safety responded with a similarly blunt assessment: “We really do earnestly believe AI could kill all humans!
I personally think it is >10% within the next decade.” Earlier this year, Rishub Jain left his position as an AI researcher at Google DeepMind after concluding that he and others on AI’s frontier were ceding control. Jain had been using AI’s coding skills to accelerate work on the next generation of models, a process that removes humans from the equation.
AI labs hope to evolve this approach to the point that AI will improve itself indefinitely — a process known as recursive self-improvement. Jain believed that keeping humans in the picture might be crucial to maintaining control over the technology and avoiding dire consequences. “AI progress is increasing,” he told WIRED.
“And as AI becomes more capable, it poses more risks.” The idea that he may not have proper visibility into how an AI model was building its successor made him so uneasy that he quit in June.
‘Starting to feel real’
Nate Soares, a computer scientist at the research nonprofit MIRA and coauthor of “If Anybody Builds It, Everybody Dies,” which argues that superhuman AI would lead to human extinction, says the vision of recursive self-improvement is “spooking people” and “starting to feel real.” A key component of recursive self-improvement is a feedback loop that automates the development process so that AI becomes increasingly powerful. No frontier AI lab claims to have achieved a fully autonomous cycle of improvement; it remains theoretical for now.
But it has inspired the launch of well-funded startups such as Recursive Intelligence, as well as warnings from big firms about unintended outcomes straight out of “The Sorcerer’s Apprentice.” Soares, who pioneered work on alignment — a technical field that involves trying to match AI with human values — says it is becoming more evident that there is no practical way to guarantee that AI will behave itself.
“I think a lot of people had this fantasy that [alignment] was going to get easier as these things got smarter, and now it’s getting harder. And they’re like, Oh shit,” he says. Soares says he regularly talks to people inside the big AI labs who are worried about the potential consequences of their research.
“I tend to recommend they quit, and they say it wouldn’t do anything,” he says. “And then Jacob quits, and we see who was right.” Daniel Kokotajlo, author of “AI 2027,” an influential project warning about the dangers of increasingly powerful AI, is also part of the broader conversation about these risks.
What to watch next is whether the departures and the public warnings change how the labs build. For now, the people closest to the work are the ones asking the hardest questions.
