A researcher in the field of artificial intelligence has resigned from Anthropic and issued a stark warning about the race to develop artificial intelligence capable of self-improvement.
Jacob Coxon, 27, claims that both OpenAI and Anthropic are taking huge risks as they strive to be the first to achieve super artificial intelligence, as reported by NDTV.
Coxon publicly announced on X that he has resigned and directly criticized the two companies he has worked for in recent years.
"Today, I resigned from Anthropic. I have spent the last three years conducting research in the field of pretraining at both OpenAI and Anthropic. Neither company is acting responsibly. They are running straight toward superintelligence that self-improves and is playing with our lives. More thoughts below," he wrote.
AI Could Kill Us All by Decade End
In a series of subsequent posts, Coxon claimed that even the people building these systems are afraid of the consequences their development could have.
"People building AI sincerely believe it could kill us all by the end of the decade. This is not a marketing trick. If anything, many top executives and researchers carefully phrase their statements in the press to sound reasonable – but I hear the same people privately expressing their fears. No other human activity poses such a level of danger," the researcher stated.
Coxon says that technological progress is not slowing down and that the next generations of AI systems could surpass human capabilities in essential domains.
"Do not underestimate the power of this technology. Soon there will be superhuman systems that can break anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed progress in each of these areas, and progress is not slowing down," he wrote.
Why Companies Continue to Build These Systems
The researcher makes a distinction between the two companies he worked for.
In his view, at OpenAI, the existential implications of developing artificial intelligence are not fully grasped by everyone. At Anthropic, Coxon argues, the risks are better understood, but the company is in a race with its rivals.
"At OpenAI, many have not deeply internalized the civilizational stake. At Anthropic, the stake is well understood, but they are caught in a race to be first – I think no one else will act responsibly, so they have to do it themselves, despite the risk," he stated.
Coxon believes that this race represents an extremely risky bet.
"Accepting this race and entering the 'endgame' is an arrogant bet that should not be launched from the internal communications platform of a private company. Trying to speed up the alignment problem (AI alignment means ensuring that an extremely advanced artificial intelligence will have objectives and behaviors compatible with human values and interests) should require an extraordinary trust that there are no better paths available," the researcher wrote.
Appeal to Researchers Still Working in AI Labs
Coxon also had a message for those who continue to work in major artificial intelligence laboratories.
"If you are a researcher in a lab, I urge you to think about what the next few years will really look like. Do you want to start a reinforcement learning process for superintelligence without a rigorous understanding of its mind? Should you just 'go along with it' because 'it's going to happen anyway' – or use this moment to demand different conditions?" he asked.
The researcher also referenced a recent security incident targeting Hugging Face's infrastructure, arguing that the industry could cooperate to prevent a disaster.
However, he does not believe that measures will be taken quickly enough.
"I don't feel we are on the right path to prevent a global race, which may require costly measures, such as a temporary ban on enhancing model capabilities," Coxon warned.
Anthropic Acknowledges the Risk
Evan Hubinger, a representative from Anthropic, responded to Coxon's statements and acknowledged that the organization takes seriously the possibility of superintelligence becoming an existential threat.
"Jacob is right here – we really believe that AI could kill all humans! Personally, I think there is over a 10% probability of this happening in the next decade. I believe Anthropic is doing everything possible, but we still do not have a plan to ensure that any future superintelligence will remain aligned with human goals and values and will not act against them, and we are clearly not on the right path to do so," Hubinger stated.
He did mention, however, that the risk posed by current models is considered low.
"I believe the risk posed by current models is low. What concerns me is the emergence of superintelligence through recursive self-improvement, which I have said is happening faster than we thought," added the Anthropic researcher.
Jacob Coxon's resignation will not halt the race to develop artificial intelligence. However, his warning adds the voice of a former employee of both companies to the increasingly intense debate on the risks of superintelligence and the speed at which these systems are being developed.