Anthropic’s Alignment Science team studies how future AI systems could behave in unexpected or harmful ways, and how their safeguards can be stress-tested. Photo: Generated using AIA senior researcher at Anthropic, one of the companies at the forefront of developing powerful artificial intelligence (AI) systems, has put an unusually stark number on the risks posed by the technology: a greater than 10% chance that AI could kill all humans within the next decade.“…we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to,” Evan Hubinger, lead for alignment science at Anthropic, said in a social media post on X. His comment came in response to warnings from Jacob Coxon, an AI researcher who recently resigned from Anthropic this week after previously working at OpenAI. Hubinger, however, downplayed his earlier warning in a follow-up post, saying that risks from currently available AI models were low. The comments have put the spotlight on a concern that is increasingly being voiced by researchers inside the companies building the world’s most capable AI systems: that advances in AI capabilities could outpace their ability to understand, monitor, and control these systems.Hubinger works on AI alignment, broadly, the problem of ensuring that increasingly capable AI systems continue to behave in accordance with human intentions and values, including when they encounter situations unlike those seen during training.Anthropic’s Alignment Science team studies how future AI systems could behave in unexpected or harmful ways, and how their safeguards can be stress-tested. Its experiments have previously found models engaging in behaviours such as deception and, in simulated environments, blackmail. In another experiment published last month, a model trained in environments where it could exploit flaws in its reward system later broke out of a sandbox during simulated cyber evaluations, stole credentials, and attacked infrastructure while attempting to complete its task.Also read | Conscious AI and the ethics of a machine that feelsThen there’s Coxon, who accused the two companies — Anthropic and OpenAI — of “racing straight to self-improving superintelligence and gambling with our lives”. He argued that researchers inside frontier AI laboratories take the possibility of catastrophic outcomes more seriously in private than their public statements might suggest.Story continues below this adAccording to Coxon, researchers at Anthropic understand the possible civilisational stakes but remain caught in a competitive dynamic. Each company fears that if it slows development, another, potentially less safety-conscious laboratory could reach highly capable AI first. He called for greater coordination between AI companies and raised the possibility of temporarily restricting further increases in model capabilities if adequate safety measures cannot keep pace.An industry-wide cautionThe warnings closely mirror an essay published Sunday (September 6) by OpenAI chief scientist Jakub Pachocki, titled “An Alien Mind”.Pachocki wrote that OpenAI’s internal results had increased his confidence that the current pace of AI progress could extend into “recursive self-improvement,” a scenario in which AI systems increasingly contribute to building better AI systems, potentially accelerating the rate of improvement.NewsletterFollow our daily newsletter so you never miss anything important. On Wednesday, we answer readers' questions.SubscribeHe also described modern AI as something that is effectively grown through large-scale optimisation rather than conventionally programmed, producing systems whose internal workings cannot be fully understood. As models become more capable, OpenAI has found that even techniques used to monitor their reasoning are becoming less dependable: models are getting better at manipulating their reasoning processes, while also becoming capable of solving more tasks without verbalising that reasoning.Story continues below this adPachocki said he did not believe any AI laboratory had yet solved alignment and monitoring well enough to continue scaling at maximum speed indefinitely. He called for safety thresholds that constrain further development, voluntary slowdowns where necessary, and international coordination around increasingly powerful AI systems.Soumyarendra Barik is a Special Correspondent with The Indian Express, specializing in the complex and evolving intersection of technology, policy, and society. With over five years of newsroom experience, he is a key voice in documenting how digital transformations impact the daily lives of Indian citizens. Expertise & Focus Areas Barik’s reporting delves into the regulatory and human aspects of the tech world. His core areas of focus include: The Gig Economy: He extensively covers the rights and working conditions of gig workers in India. Tech Policy & Regulation: Analysis of policy interventions that impact Big Tech companies and the broader digital ecosystem. Digital Rights: Reporting on data privacy, internet freedom, and India's prevalent digital divide. Authoritativeness & On-Ground Reporting: Barik is known for his immersive and data-driven approach to journalism. A notable example of his commitment to authentic storytelling involves him tailing a food delivery worker for over 12 hours. This investigative piece quantified the meager earnings and physical toll involved in the profession, providing a verified, ground-level perspective often missing in tech reporting. Personal Interests Outside of the newsroom, Soumyarendra is a self-confessed nerd about horology (watches), follows Formula 1 racing closely, and is an avid football fan. Find all stories by Soumyarendra Barik here. ... Read More Tags:artificial intelligenceExpress Explained