SEBASTIAN MALLABY2026年9月15日It takes courage to stand in front of an express train and yell at it to slow down. That is what Dario Amodei, the chief executive of the leading artificial intelligence lab Anthropic, has just done.站在一列高速列车前，大喊着让它减速，这需要很大的勇气。这正是领先的人工智能实验室Anthropic首席执行官达里奥·阿莫迪刚刚做过的事情。Anthropic has the world’s strongest A.I. models. It most likely has the fastest revenue growth in the history of capitalism. Its lead is set to grow because it is probably closest to the takeoff point of recursive self-improvement, when A.I. autonomously creates stronger versions of itself. Yet on Saturday Amodei published an essay calling for an A.I. slowdown. He declared that recursive self-improvement “must be pursued very carefully, if at all.”Anthropic拥有世界上最强大的人工智能模型。它很可能拥有资本主义历史上最快的收入增长速度。它的领先优势还将扩大，因为它可能最接近“递归自我改进”（人工智能自主创建更强大版本的时刻）的起飞点。然而，就在周六，阿莫迪发表了一篇文章，呼吁放缓人工智能的发展。他宣称，递归自我改进“必须非常谨慎地推进，如果真要推进的话”。It’s hard to think of another chief executive who has done something this gutsy — especially one simultaneously planning a blockbuster initial public offering. He deserves all due credit for grappling with a major problem. That’s not the same as saying he has the solutions.很难想出还有哪位首席执行官做过如此有胆量的事情——尤其是一位同时还在筹划轰动性首次公开募股的首席执行官。他勇于直面重大问题，理应获得充分的赞誉。但这并不等于说他已经找到了解决方案。Mr. Amodei is channeling his fear, and that of Anthropic’s internal brain trust, that a “swarm” of A.I. agents could be capable of “taking over the entire internet” in the next six to 12 months. He worries that, in the absence of advanced safety guardrails, “the scale of damage would continue to increase from there.”阿莫迪表达了他以及Anthropic内部智囊团的担忧：人工智能智能体“集群”可能会在未来六到12个月内具备“接管整个互联网”的能力。他担心，如果没有先进安全护栏，“破坏的规模将持续扩大。”He therefore proposes “pacing” — a slowing of A.I. capability to allow A.I. safety to keep up. As a first step, Anthropic will grant outside experts permanent access badges and system permissions. These embedded evaluators will monitor Anthropic’s safety practices and report incidents to the public.因此，他提出了“步调控制”——放缓人工智能能力的提升，让人工智能安全能够跟上。作为第一步，Anthropic将向外部专家授予永久访问证和系统权限。这些常驻评估者将监控Anthropic的安全实践，并向公众报告相关事件。The industry rejected earlier calls for slowing or pausing A.I. development. This time, because of the technology’s alarming progress, three rival A.I. executives — Sam Altman, Elon Musk and Demis Hassabis — have commended Mr. Amodei’s essay; and Mr. Altman says that he will follow Mr. Amodei’s example by embedding evaluators in his company, OpenAI.此前，业界曾拒绝过放缓或暂停人工智能发展的呼吁。而这一次，由于该技术令人担忧的进展，三位竞争对手公司的人工智能高管——萨姆·奥特曼、埃隆·马斯克和杰米斯·哈萨比斯——都对阿莫迪的文章表示赞赏；奥特曼还表示，他将效仿阿莫迪的做法，在其公司OpenAI内部引入常驻评估者。In the past, the A.I. labs didn’t know what they would do during a pause. They have reversed their position because they now have a to-do list as long as their arms.过去，人工智能实验室不知道在暂停期间该做些什么。如今它们改变了立场，是因为现在他们有了一份长长的待办事项清单。Many recent A.I. safety incidents could have been prevented if the labs had been more careful about operational details. Of five mishaps known to have occurred in quick succession over the summer, three involved errors in the configuration of so-called sandboxes — contained digital testing environments that A.I. agents are not supposed to be able to exit.如果各实验室在操作细节上更加谨慎，最近发生的许多人工智能安全事件本是可以避免的。在今年夏天已知连续发生的五起事故中，有三起涉及所谓的“沙盒”配置错误——沙盒是一种受控的数字测试环境，人工智能智能体本来是不应该能够从中逃脱的。To eliminate such glitches, the labs need to tighten their quarantine of experimental models, scrub training data of material that encourages models to behave badly and fix countless other protocols. “There is precedent for operating technologically complex, safety-critical systems millions of times without anything going wrong — for example, commercial airplanes,” Mr. Amodei observed in his essay. “But it takes time to get it right.”为了消除这些故障，实验室需要加强对实验模型的隔离，从训练数据中清除那些会诱导模型产生不良行为的内容，并修复无数其他的协议。“在技术复杂且安全至关重要的系统中，成百上千万次运行而未出任何差错是有先例的——例如商用飞机，”阿莫迪在他的文章中写到。“但这需要时间才能做好。”Some officials at Anthropic’s competitors, notably Jensen Huang, boss of the chip designer Nvidia, implicitly accuse Mr. Amodei of exaggerating A.I. risk for business reasons. By announcing that A.I. is dangerous, Mr. Amodei is really just trumpeting the fact that A.I. is powerful, Mr. Huang argues, without naming Mr. Amodei. Calling for safety, to Mr. Huang, is a marketing trick.Anthropic一些竞争对手的高管——尤其是芯片设计公司英伟达的老板黄仁勋——隐晦地指责阿莫迪出于商业原因夸大了人工智能的风险。黄仁勋在没有点名阿莫代先生的情况下称，通过宣布人工智能具有危险性，阿莫迪实际上只是在吹嘘人工智能十分强大。在黄仁勋看来，呼吁安全只是一种营销伎俩。But impartial authorities — academic experts, the British government’s respected A.I. Security Institute, the recent independent report on rogue A.I. agents’ hacking of the website Hugging Face — support Mr. Amodei’s claim that A.I. risk is genuine. Mr. Huang, who wants his company to sell lots more of its A.I.-enabling chips, is the one taking positions that accord with his company’s bottom line.但包括学术专家、备受推崇的英国政府人工智能安全研究所、最近关于失控的人工智能代理黑客攻击Hugging Face网站的独立报告在内，公正的权威机构都支持阿莫迪关于人工智能风险是真实存在的观点。黄仁勋希望自己的公司能售出更多支持人工智能的芯片，因此他所持的立场才是完全符合其公司经济利益的。A variant on Mr. Huang’s claim holds that Mr. Amodei is playing up A.I. risk to ensure government regulation, and that this is a cynical ploy to throttle competitors and cement Anthropic’s lead. But this “regulatory capture” story is also backward. The biggest reason to suspect that competitors might lose out from regulation is that they are often more dangerous. Many produce “open-weight” A.I. models — ones that allow users to remove safety guardrails.黄仁勋主张的一种变体认为，阿莫迪正在大肆渲染人工智能风险，以确保政府实施监管，而这是一种压制竞争对手并巩固Anthropic领先地位的冷酷伎俩。但这种“监管俘获”的说法也是本末倒置的。怀疑竞争对手可能会因监管而遭受损失的最大原因，是它们通常更加危险。许多公司生产“开放权重”的人工智能模型，这些模型允许用户移除安全护栏。The reasonable critique of Mr. Amodei’s proposal is that it wouldn’t result in the A.I. slowdown that he wants. He presents his embedded evaluators as a step toward coordinated pacing: If all labs in democratic countries embrace them, the evaluators can verify that no opportunist is taking irresponsible shortcuts.对阿莫代先生提议的合理批评在于，这一提议不会带来他所希望的人工智能发展放缓。他将常驻评估者作为迈向协同步调控制的一步：如果民主国家的所有实验室都接受这些评估者，他们就可以验证，是否存在投机者采取不负责任的捷径。But it’s not clear that all labs, or even most labs, will follow Anthropic’s example — or, if they do, that the evaluators will help. The Google subsidiary DeepMind previously invited independent evaluators to monitor its work on A.I.-enabled health products. Anxious to signal credibility, the evaluators exaggerated DeepMind’s shortcomings until Google dismissed them.但尚不清楚是否所有或大多数实验室都会效仿Anthropic的做法——或者，即使他们这样做了，评估者是否会有所帮助。谷歌子公司DeepMind此前曾邀请独立评估者监督其在基于人工智能的健康产品方面的工作。为了急于彰显公信力，这些评估者夸大了DeepMind的缺点，直到谷歌解雇了他们。To be fair to Mr. Amodei, he is proposing evaluators because Congress is unlikely to act quickly to create a government regulator that forces labs to act responsibly — the most obvious route to coordinated pacing. But he only briefly mentions another coordination mechanism that might prove useful: an industry-financed but government-endorsed self-regulatory body, as proposed two months ago by Mr. Hassabis, the chair of DeepMind.公平地说，阿莫迪提议引入评估者，是因为国会不太可能迅速采取行动，建立一个迫使实验室负责任地行事的政府监管机构，而这本是实现协同步调控制最明显的途径。但他只是简要地提到了另一种可能证明有用的协调机制：一个由行业资助但受到政府认可的自律机构，正如DeepMind主席哈萨比斯在两个月前所提议的那样。America’s A.I. titans should create such a body immediately. The government can help by ensuring that any potential antitrust concerns around this kind of industry coordination are minimized or waived.美国的人工智能巨头应该立即建立这样一个机构。政府可以通过确保将此类行业协调可能引发的反垄断担忧降至最低或予以豁免来提供帮助。Even this would not be enough, though. The larger coordination problem for any A.I. slowdown concerns China. “Pacing within democracies will be limited by the lead that U.S. companies have over authoritarian regimes, chiefly the Chinese Communist Party,” Mr. Amodei wrote. Since the U.S. lead over China stands at only a few months, Mr. Amodei is saying that a Western slowdown must be modest. But he also suggests that safer A.I. development might require additional breathing room of one or two years.然而，即使这样也是不够的。针对任何人工智能发展减速的提议，更广泛的协调难题都在于中国。“民主国家内部的减速节奏将受限于美国企业领先威权政权——主要是中国共产党——的幅度，”阿莫迪写道。鉴于美国对中国的领先优势只有几个月，阿莫迪的意思是，西方的放缓必须是适度的。但他也表示，为了更安全地发展人工智能，可能还需要一到两年的缓冲期。How to buy more time for safety without falling behind China? Here, Mr. Amodei restated his view that China should be denied the tools of A.I. progress. American controls on exports of chips and chip-manufacturing equipment should be tightened. “Distillation,” the practice of using advanced models to train new ones, should be combated, since China employs this shortcut ruthlessly. Security at Western labs should be strengthened to stop China from stealing A.I. secrets. These measures could expand the Western A.I. lead, enabling slower pacing.如何在不落后于中国的情况下为安全争取更多时间？在这里，阿莫迪重申了他的观点：应该拒绝向中国提供人工智能进步的工具。美国应收紧对芯片及芯片制造设备的出口管制。应该打击“蒸馏”（使用高级模型来训练新模型的做法），因为中国正不择手段地使用这种捷径。应该加强西方实验室的安全性，以阻止中国窃取人工智能机密。这些措施可以扩大西方在人工智能领域的领先地位，从而使放缓步调成为可能。This playbook has been tried already, and the results are not encouraging. The Biden administration imposed chip-export controls on China in 2022; the loopholes have been obvious for some time, but neither the Biden team nor the Trump team closed them enough to halt China’s progress.这一策略早已付诸实践，但结果并不令人鼓舞。拜登政府在2022年对中国实施了芯片出口管制；一段时间以来，漏洞一直很明显，但无论是拜登团队还是特朗普团队都未能充分堵住这些漏洞以遏制中国的进步。Meanwhile, Western frontier labs have enormous incentives to prevent distillation and guard against theft of their intellectual property. If they have not succeeded yet, it is probably because they don’t know how. Despite America’s best efforts to hobble China’s A.I. industry, Chinese models account for a growing share of A.I. usage in the United States.与此同时，西方前沿实验室有巨大的动力来防止蒸馏并防范其知识产权被窃取。如果它们尚未取得成功，那可能是因为它们不知道该怎么做。尽管美国竭力阻碍中国的人工智能产业，但在美国的人工智能使用量中，中国模型的占比却在不断增长。The alternative to keeping China down is to make China a partner in safety and pacing. Until now, Mr. Amodei has embraced the U.S. foreign policy consensus that negotiating an A.I. deal with China is near-impossible. So perhaps the most significant section of his essay is the last one, in which he appeared to soften his stance. He avoided calling out China’s techno-authoritarianism and oppression of ethnic minorities and listed a series of areas on which collaboration might be possible. During the Cold War, the United States competed fiercely with the Soviet Union. That did not prevent them from striking arms-control deals.除了压制中国之外，另一种选择是让中国成为安全与发展节奏方面的合作伙伴。在此之前，阿莫迪一直支持美国外交政策的共识，即与中国谈判达成人工智能协议几乎是不可能的。因此，他文章中最重要的一部分也许是最后一部分，在这一部分中，他似乎软化了自己的立场。他避免点名批评中国的技术威权主义和对少数民族的压迫，并列出了一系列可能进行合作的领域。在冷战期间，美国与苏联进行了激烈的竞争。但这并未阻止他们达成军备控制协议。On Sept. 24, President Trump is scheduled to meet China’s leader, Xi Jinping. The summit is expected to yield something modest on A.I. diplomacy, but the good news is that the leaders may meet twice more before the end of this year. The superpowers share an interest in preventing superhuman A.I. models from causing havoc, as China’s leaders clearly recognize. However difficult U.S.-China coordination, the consequences of not coordinating make it essential to try.特朗普总统计划于9月24日会见中国领导人习近平。预计这次峰会在人工智能外交方面成果有限，但好消息是，两国领导人可能在今年年底前再会面两次。超级大国在防止超越人类的人工智能模型造成严重破坏方面有着共同的利益，中国领导人清楚地认识到了这一点。无论美中协调有多么困难，不协调的后果都使得尝试协调成为必要。The icy state of U.S.-China relations is what makes an A.I. slowdown so elusive. The technology’s positive potential will be realized only if the Trump administration — and tech leaders like Mr. Amodei — throw their full weight behind A.I. talks with Beijing.美中关系的冰冷状态正是导致放缓人工智能发展变得如此难以实现的原因。只有当特朗普政府以及像阿莫迪这样的科技领袖全力支持与北京举行人工智能谈判时，这项技术的积极潜力才能得以实现。Sebastian Mallaby是美国外交关系委员会的高级研究员，也是《The Infinity Machine: Demis Hassabis, DeepMind, and the Quest for Superintelligence》一书的作者。他与他人共同主持该委员会的播客节目“The Spillover“。翻译：晋其角点击查看本文英文版。