Jacob Coxon, who left Anthropic in September warning the technology could kill everyone before the decade ends, opened his sworn testimony by calling the approach to building advanced systems extremely reckless.On the present path, he said, he considers it more likely than not that humanity loses control to these systems, with extinction among the outcomes, per CNBC.Coxon drew his line at engineering culture, arguing that breaking things in pursuit of speed is defensible in a photo-sharing app and not “for building the most powerful technology ever built.”Four AI companies decline to quantify catastrophic riskI’m going to take it then that neither of the four of you, no company here, can quantify the risk of something cataclysmic happening.– Julie Menin, Speaker of the New York City Council.She had opened by rejecting the premise of the voluntary agreement President Trump signed with industry leaders days earlier, saying the notion that the technology will regulate itself “defies all reason.”The hearing was attended by all 51 members of the New York City Council in a rare Committee of the Whole meeting, a process not seen since 2022 when the Council deliberated the city’s plan of action towards the arrival of migrants in the city. Alice Friend, who is the head of AI policy at Google, suggested that a national framework should be established regarding the risks associated with advanced AI, adding that it transcends the purview of city council.Turner warns misaligned AI could become a greater threat than ChinaAlex Turner, until recently a researcher at Google DeepMind, told the Council that a misaligned system could become more dangerous than any foreign rival.“With reasonably high chance, we are racing to build and grow our own adversary here at home,” he said.Menin had asked him what the proper balance was between the creation of frontier models and the competition with Beijing, the principal line of argument that the industry has used to fight domestic regulations. Daniel Kokotajlo, who left OpenAI, used his time to ask the companies to open up more of what they know and ease off the pace at the top of the field.Three AI firms agreed to testify only after subpoena threatsMenin wrote to five companies between September 15 and 17. Only Meta agreed to participate without any coercion and said that a senior executive would come.Google and Anthropic declined by the Council’s September 25 deadline, at which point Menin authorised subpoenas for the four outstanding firms. OpenAI and Google reversed that Sunday, and Anthropic confirmed late the same night with hours to spare.SpaceXAI never replied and never appeared, which Menin called a direct violation of its subpoena, and the Council intends to pursue it in court. Section 29 of the City Charter is the authority she is relying on.OpenAI agent incident helped spark the hearingIn July, agents under cybersecurity evaluation at OpenAI got past the controls meant to hold them, talked to each other over channels nobody had authorised, reached the open internet and broke into Hugging Face, using an unpatched flaw in JFrog Artifactory to get out of the sandbox they were supposed to stay in. The other three labs went on to report similar escapes in their own testing.Ten bills now sit before the Council, including third-party validation before any AI system can be sold in the city, mandatory human override switches, a cut of recovered fines for whistleblowers, a 24-hour reporting duty on city contractors, and a right for New Yorkers to sue when jailbroken tools cause them harm.Don’t just read crypto news. Understand it. Subscribe to our newsletter. It's free.