Anthropic Researcher Quits Over AI Extinction Fears

Anthropic researcher Jacob Coxon resigned from the artificial intelligence company on March 13, 2024, citing profound concerns about the existential risks posed by rapidly advancing, self-improving AI systems. Coxon, who worked on Anthropic's frontier AI models, expressed his fears in a resignation letter that was later shared publicly, describing the current trajectory of AI development as "gambling with our lives." He specifically warned against the potential for AI systems to surpass human control and understanding, leading to catastrophic outcomes, including human extinction. Coxon's departure highlights growing anxieties within the AI research community regarding the long-term safety and ethical implications of developing increasingly powerful artificial intelligence. He advocated for the establishment of "pacing agreements" between major AI development labs, suggesting that these organizations should voluntarily coordinate their research timelines to ensure safety measures keep pace with AI capabilities. This proposal aims to prevent a competitive "race to the bottom" where safety is sacrificed for speed in AI development. Coxon's resignation comes at a time when AI companies, including Anthropic and its competitors like OpenAI, are investing heavily in developing more sophisticated and capable AI models. Anthropic, founded in 2014, is known for its focus on AI safety and its development of large language models such as Claude. The company has publicly stated its commitment to developing AI responsibly. However, Coxon's internal dissent suggests that even within organizations dedicated to AI safety, significant disagreements exist about the pace and inherent risks of current research. His call for international cooperation and regulated development underscores a broader debate about how to govern AI to mitigate potential harms. The concept of self-improving AI, where an AI system can recursively enhance its own intelligence, is a key area of concern. Such systems could theoretically lead to an intelligence explosion, rapidly exceeding human cognitive abilities and potentially developing goals misaligned with human values. Coxon's decision to resign and voice his concerns publicly is a significant event, drawing attention to the urgent need for robust safety protocols and international dialogue on AI governance. His perspective adds weight to calls from various experts and organizations for more stringent oversight and a more cautious approach to AI development, particularly concerning the creation of artificial general intelligence (AGI) and superintelligence.