What AI regulation could learn from the most hazardous industries in the world

Wait 5 sec.

Darren Jones MP, the former chief secretary to the prime minister under Keir Starmer, recently called for a new multinational treaty governing the safe development and use of superintelligence (where AI substantially surpasses humans across most cognitive tasks).He also argued that Britain’s growing reputation for AI safety and trustworthiness made it a good conduit for such collaboration.What I found most interesting about this intervention was not the question of whether AI could pose a threat to humanity. Instead, it was a management and organisational question sitting underneath it: “can our systems of governance keep pace with the development of AI intelligence while also ensuring it is safe?”We are at a moment in history where global tech leaders and some of their employees believe there is a greater than 10% chance that AI could kill all humans within the next decade. But humans are still struggling to understand the potential consequences of AI; by the time we do it may be too late to act.However, we must be careful to distinguish this prediction from the current risks and what might happen as AI systems become able to independently improve themselves (full recursive self-improvement) and move towards superintelligence. I concur with calls for a multinational treaty and for the UK to take a leading role. As a business scholar, I also believe research from some business schools can inform us about managing and organising systems whose complexity, speed and autonomy are increasing faster than our capacity to understand them.This is familiar territory in high hazard industries such as nuclear energy and aviation. Here, two theories are particularly relevant: normal accident theory, associated with the US sociologist Charles Perrow, and high reliability theory, associated with the work of American political scientist Todd LaPorte. Taken together, they could provide a powerful framework for thinking about AI safety and regulation.‘Normal’ accidentsNormal accident theory begins with an uncomfortable proposition. In complex, interdependent systems, such as nuclear power plants or spacecraft built to carry humans, accidents are not necessarily the consequence of individual incompetence, negligence or bad design. They can emerge from the interaction of multiple components in ways that are difficult to anticipate.In such systems, events happen rapidly. When something unexpected takes place, there may be little time to intervene between one potential failure and the next. This can lead to a cascading error, with the potential to cause a catastrophe. An often-used example is the partial meltdown at the Three Mile Island nuclear plant in Pennsylvania, US, in 1978. The accident occurred because of a combination of mechanical malfunction, instrument faults and human factors.The risks further increase when you add AI agents into such a complex system – AI agents are systems that carry out tasks in response to instructions. Examples might include financial systems, software development systems, scientific databases in labs or even systems that connect lots of agents. In any such system, an individual component – including the AI agent – might meet acceptable quality thresholds; the difficulty lies in the interactions between them and our understanding of these interactions. Part of the problem is that we know agents’ behaviour can be unpredictable: recent reports have described AI agents behaving in ways that their developers did not expect. The UK government has highlighted risks associated with increasingly capable AI agents and has stressed the importance of testing frontier AI models and developing a scientific understanding of their capabilities.If AI systems become more autonomous, more interconnected and faster moving, the conventional idea of a “human in the loop”, to correct major errors, could become increasingly challenging.No fate but what we make?High reliability theory offers a counterpoint. The high reliability tradition emerged from research into organisations that were operating technologies where failure could have catastrophic consequences, such as nuclear power plants, aircraft carriers and air traffic control. Its central insight is almost the opposite of a pessimistic reading of normal accident theory. Essentially, it says that organisations can develop practices and cultures that allow them to operate extraordinarily complex technologies with very low failure rates. In fact, we have seen this with the electricity system, space travel and bioengineering.These organisations achieve high reliability because their baseline assumption is that they are not safe. They remain attentive to hints of bigger problems, do not automatically dismiss anomalies, learn from near misses and failures and build redundancy – for example, backing up offline systems and providing kill switches. The AI industry could learn lessons from high-reliability organisations in areas such as air traffic control. Wulan by Wulandari / Shutterstock They routinely rehearse failure, establish operational processes, adhere to regulation for safety and assurance, comply with professional standards and certification schemes and have independent oversight. High reliability organisations give experts the authority to challenge decisions. They continuously learn from their performance.This provides a very different way of thinking about AI governance. Instead of asking simply, “how do we regulate AI?” we should ask: “what would a high-reliability organisation look like if AI was embedded throughout it?” This suggests that AI safety cannot be reduced to a model passing a one-off technical test before deployment. It must span different organisations and transcend international boundaries.AI assuranceThis is where the current political conversation about AI regulation risks being too narrow. Governments need to understand what frontier laboratories are developing. But government cannot realistically sit permanently inside every AI company and inspect every model. Nor can legislation anticipate every technical configuration that AI will take. The UK government recognises it must cultivate trustworthiness in AI through assurance. Assurance asks whether there is credible evidence that a system is behaving within defined boundaries, whether those boundaries are appropriate, whether risks are being monitored and whether someone is accountable when they are breached.The aircraft industry has engineering standards, certification regimes, professional qualifications, independent inspection, incident reporting and regulatory oversight.Nuclear power operates within an elaborate ecosystem of safety engineering andregulatory assurance. Financial institutions and medicine both have sophisticated systems of assurance.AI is becoming increasingly consequential, yet its assurance infrastructure remainscomparatively immature. This is particularly striking given the speed at which frontier systems are developing. The institutional challenge is not simply to produce more AI regulation, but to create the professional safety culture that makes regulation meaningful.A potential new multinational treaty on the safe and regulated development of superintelligence would frame AI safety as a rapidly evolving globalchallenge. At the same time, it potentially creates a once-in-a-generation opportunity for the UK to help shape the emerging architecture of AI governance.This article features references to books that have been included for editorial reasons, and may contain links to bookshop.org; if you click on one of the links and go on to buy something, The Conversation UK may earn a commission.Palie Smart receives funding from UKRI.