Anthropic Researcher Warns of AI Existential Risk

AI researcher Jacob Coxon has resigned from Anthropic, a leading artificial intelligence company, to publicly voice concerns about the existential risks posed by advanced AI systems. Coxon stated in a social media thread on Tuesday night that frontier AI companies are "gambling with our lives" by developing systems that they "earnestly believe... could kill us all by the end of the decade." He elaborated that the primary danger lies not in current AI models but in the future development of "self-improving superintelligence," which could lead to "superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources." Coxon suggested that individuals working on these advanced AI systems either have not fully grasped the "civilizational stakes" involved or are driven by a belief that they must "speedrun" the race to superintelligence to preempt a less responsible entity from achieving it first. This perspective is not isolated within Anthropic; Evan Hubinger, Anthropic's Alignment Science Lead, publicly supported Coxon's assessment. Hubinger stated on social media that "Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade." Anthropic, founded in 2021 by former OpenAI members, focuses on developing safe and beneficial artificial intelligence. The company's mission centers on building reliable, interpretable, and steerable AI systems, with a significant emphasis on AI safety research. Their flagship models, such as the Claude series, are designed with constitutional AI principles to align with human values and avoid harmful outputs. The concerns raised by Coxon and Hubinger highlight a growing debate within the AI community regarding the long-term safety implications of increasingly powerful AI technologies. This debate involves researchers, ethicists, and policymakers grappling with how to ensure that the development of artificial general intelligence (AGI) and superintelligence proceeds in a manner that is beneficial and not detrimental to human civilization. The potential for AI to rapidly surpass human cognitive abilities and gain control over critical infrastructure or resources is a recurring theme in these discussions. The concept of a "superintelligence" refers to an AI that possesses intelligence far exceeding that of the brightest human minds across virtually all fields, including scientific creativity, general wisdom, and social skills. Such an entity, if its goals were misaligned with human well-being, could theoretically pose an insurmountable threat. The urgency expressed by Coxon and Hubinger, particularly the "greater than 10%" probability of existential catastrophe within the next decade, underscores the perceived immediacy of these risks by some within the field. This contrasts with the more gradualist views held by others who believe that such advanced AI capabilities are still distant or that the risks are manageable through careful development and oversight. The departure of a researcher like Coxon and the public endorsement by a lead scientist like Hubinger signal a significant internal acknowledgment of these profound safety concerns at a leading AI research organization.