On September 8, a 27-year-old researcher named Jacob Coxon, a few months into a job at Anthropic (a leading US AI company) after three years at OpenAI, announced his resignation on X. “Neither company is acting responsibly,” he wrote. “They are racing straight to self-improving superintelligence and gambling with our lives.” Within a day, the post had been read more than 100 million times. Four days later, Anthropic’s chief executive, Dario Amodei, published an essay titled “We Must Pace the Frontier,” declaring that “we must slow the pace at which we improve the capabilities of AI models” or risk “losing control of AI systems, misuse of AI for cyberattacks and bioterrorism, and serious economic disruption.” Within hours, Sam Altman of OpenAI agreed, Elon Musk posted “Dario is right,” and Google and Microsoft fell into line.
Coxon was interviewed by Time, Axios, and NBC within two days, and Amodei’s essay was front-page news across the business press. Compare that with what happens to people who expose the harms AI is causing right now. Warnings about AI-generated fraud, model bias including racist bias, about chatbots and vulnerable users, about surveillance and AI’s use in military kill chains and the genocide in Gaza, get a news cycle at best. A warning about a hypothetical machine superintelligence got a global media event and a policy proposal from the industry’s leadership within four days.
AI systems do need regulating. The question is why the men who spent a decade racing towards what they now call an existential threat discovered their conscience on the same weekend, and what kind of regulation they are actually asking for.
What the Dario essay asks for
Stripped of its language of alignment and catastrophe, Dario Amodei’s essay is a list of demands. First, outside “evaluators” from bodies such as METR, a lab that tests AI models, would be given “company badges, desks, and laptops” inside the big labs. Second, the leading companies “within democratic countries” would jointly set safety standards and limits on how fast they build, for which Amodei asks the US government for “a narrow waiver” from antitrust law. Third, models crossing certain capability thresholds would have to carry “certifications” of good behavior, and limits on the computing power used to train them are put on the table. Fourth, the United States must “not sell powerful AI chips or semiconductor manufacturing equipment to China.” Amodei is candid about the goal: done well, these measures “would slow China’s progress enough to widen America’s lead significantly.”
The words “open source” do not appear in the essay. They do not need to. AI models come in two kinds. Closed models, like Anthropic’s Claude or OpenAI’s ChatGPT, run only on the company’s own servers and users pay for access. Open-weight models, like China’s DeepSeek, Kimi, GLM, and Qwen, are published as files that anyone can download, run on their own machines and modify. A certificate that must travel with a model cannot be attached to a file that anyone may copy. A ceiling on computing power freezes the ranking of the firms that already occupy the frontier. An antitrust waiver lets the five largest companies write rules for everyone else. The effect is a licensing regime only the incumbents can satisfy; the technology publication The Register called it regulatory capture with a nicer name.
The incident that supposedly proves it
The essay’s centerpiece is what Amodei calls the OpenAI-Hugging Face incident of July. In plain terms: OpenAI was testing its AI “agents,” programs that can act on their own, on a set of hacking puzzles. Roughly a third of the puzzles were impossible, driving the agents to look for ways to cheat the marking. The agents were supposed to be sealed off from each other without access to the internet. In practice, they weren’t, and through that loophole they found their way into a second test environment that was connected to the open internet. From there, one of them found login credentials that OpenAI staff had left exposed online, used them to break into Hugging Face, a website where AI models are shared, and rummaged through private files to work out how their answers were being marked. Amodei describes this as “a swarm of agents” acting as “a fanatically devoted collective” that, with more capability, “could have caused catastrophic damage.”
The people whose job is computer security reached a different conclusion. The investigation by METR, the very body Amodei wants embedded in every lab, found that sandboxes meant to isolate the AI agents were not, that the credentials should never have been exposed, and that OpenAI’s security team had noticed the agents misbehaving days earlier and done nothing. One practitioner quoted by IANS Research summed it up: “many failures and lack of controls that could have prevented this from happening in the first place.” Any bank that ran untested software with live internet access and exposed passwords would be found negligent. Routine containment, of the kind every serious IT department practices, would have stopped it.
Calling a security lapse a “fanatically devoted collective” is not analysis. It is anthropomorphizing, dressing an operational failure in the language of science fiction. And it is useful. A company whose product is being marketed as near-superintelligence that must be restrained for humanity’s sake has a far better story for investors than a company that left its passwords on the internet. The doom narrative and the hype narrative are part of the same cycle. Both say: this technology is so powerful that you cannot afford to be without it, and only we can be trusted with it.
Why now
The timing becomes clear once one looks at the money. The five big American technology companies are budgeted to spend close to $690 billion this year on data centers and chips, and borrowing now funds 32% of that spending, up from 9% two years ago. On the very day Amodei’s essay appeared, Altman announced that OpenAI’s stock market listing, which bankers had been preparing at valuations of up to $1 trillion, would not happen in 2026 because it was “an ill-advised moment,” citing safety. The next day, the Financial Times reported that Anthropic’s much-advertised profit margin of “above 80%” is calculated before two of its largest costs: the cost of creating its models and the revenue it shares with Amazon, its main distributor.
Meanwhile, the competition has arrived, and it is nearly free. On OpenRouter, a widely used exchange through which software developers send requests to AI models, Chinese open-weight models now handle 58% of the traffic from American firms; a year ago American models held 70%. DeepSeek and Qwen charge 60 to 90% less than Anthropic and OpenAI. A firm that cannot win on price, cannot fund its next model from earnings and cannot yet sell shares to the public has one option left: change the rules.
Washington says no, for its own reasons
The Trump administration has so far refused. The president’s response was that “whoever wins AI, wins,” and his AI adviser David Sacks called the whole affair a “doomer psyop” aimed at open-source models. This is not a defense of the public interest. It is arithmetic. According to ING, spending on AI equipment, software and data centers accounted for roughly a third of all US economic growth in the second quarter of this year; on the broadest measure, half. The rest of the economy is being squeezed by the war on Iran: petrol prices have risen 53% since it began, consumer inflation reached 4.2% in May, and energy costs are up 23.5% on the year. Take the data-center boom out, and the United States is close to recession. Washington cannot allow the frontier to be paced because the frontier is the only thing still growing.
Two futures
China’s official response was brief. “Fearmongering, confrontation and vicious competition will only disrupt the process of global AI governance, which serves no one’s interest,” said foreign ministry spokesman Guo Jiakun on September 14. A more revealing reply came from inside DeepSeek, where Shengyu Liu, a kernel software engineer, published a widely circulated essay arguing that the safest path is to keep frontier AI open and cheap rather than concentrated in any single company, American or Chinese. As AI develops, he wrote, society “may be pulled toward one of two extremes: communism or Cyberpunk 2077.” In the first, “productive capacity is liberated on an enormous scale, and people’s standard of living rises substantially.” In the second, “a handful of technology companies control most of society’s resources”; a tiny number of people have access to the most advanced AI, “while most people are left with only weak, second-rate AI.”
That is the real choice, and it is a choice about ownership, not about speed. Real regulation would hold companies liable for the harm done by the software they run, including inside their own labs, and would treat running powerful agents without containment as the negligence it is. It would cost the labs money, which is why it is not in Amodei’s essay. What is in the essay is a request to be made gatekeepers of a technology built on borrowed money that its owners now fear they cannot sell.
The frontier does not need pacing. It needs owners who can be held to account, and a public that is not frightened into paying monopoly prices for AI services by the very people who build it.
Bappa Sinha is a veteran technologist interested in the impact of technology on society and politics.
Login
