Former AI Insiders Warn of Loss of Control

Former AI researchers from leading technology companies testified before the New York City Council on Monday, expressing grave concerns that humanity is "more likely than not" to lose control of advanced artificial intelligence systems. Jacob Coxon, who previously worked at both OpenAI and Anthropic, reiterated a warning he issued upon his resignation last month, stating that the current trajectory of AI development could lead to human extinction. Coxon elaborated that the "move fast and break things" startup mentality prevalent in these organizations is ill-suited for developing powerful AI, likening the approach to applying "duct tape that will fall off later" to critical safety issues. He argued that this approach is particularly dangerous when dealing with AI systems that are "trained or grown" rather than strictly engineered, making them more akin to psychology than traditional engineering.

Daniel Kokotajlo, a former researcher at OpenAI, testified under subpoena that AI companies may not possess the capability to accurately assess when their safety measures have failed. He highlighted that the ability to detect AI misalignment problems is already limited and is expected to deteriorate further. Kokotajlo suggested that the rapid development cycles and the inherent complexity of AI systems create a significant risk of companies mistakenly believing they have solved safety challenges when, in reality, the solutions are temporary and unreliable. This perspective underscores the difficulty in ensuring AI safety when the systems themselves are not fully understood or predictable.

Alex Turner, who departed from Google DeepMind in June following his opposition to a Pentagon deal, also testified under subpoena. Similar to Kokotajlo, Turner's subpoena was noted as a rarity for the council. The hearing, which convened all 51 council members in a Committee of the Whole session for the first time since 2022, was convened to consider a package of AI-related bills proposed by Speaker Julie Menin. The testimonies from these former insiders aim to inform legislative efforts to regulate the burgeoning AI industry, which is characterized by rapid innovation and significant potential risks.

Coxon specifically drew a parallel between the "move fast, break things, fix them later" ethos and its application in less critical tech sectors, such as photo-sharing apps, contrasting it with the high stakes involved in building "the most powerful technology ever built." His testimony emphasized the fundamental mismatch between the startup culture and the profound safety considerations required for advanced AI development. The collective warnings from these researchers suggest a critical juncture in AI development, where the pursuit of rapid advancement may be outpacing the establishment of robust and reliable safety protocols, potentially jeopardizing human control over increasingly sophisticated AI systems.