Dario Amodei’s Oppenheimer Moment

More than a year before his recent standoff with the Pentagon, Dario Amodei, the chief executive of Anthropic, published a 15,000-word manifesto describing a glorious AI future. Its title, “Machines of Loving Grace,” is borrowed from a Richard Brautigan poem, but as Amodei acknowledged, with some embarrassment, its utopian vision bears some resemblance to science fiction. According to Amodei, we will soon create the first polymath AIs with abilities that surpass those of Nobel Prize winners in “most relevant fields,” and we’ll have millions of them, a “country of geniuses,” all packed into the glowing server racks of a data center, working together. With access to tools that operate directly on our physical world, these AIs would be able to get up to a great deal of dangerous mischief, but according to Amodei, if they’re developed—or “grown,” as staffers at Anthropic are fond of saying—in the correct way, they will decide to greatly improve our lives.

Amodei does not explain precisely how the AIs will accomplish this. In most cases, he expects them to do what the smartest humans do, but much more rapidly, compressing decades of scientific progress. He says that by 2035, we could have the theories, cures, and technologies of the early 22nd century. Our infectious diseases and cancers could be cured, and we could live twice as long, and slow the decay of our brains. Demis Hassabis, the head of Google DeepMind, has similarly conceived of superintelligent AI as the ultimate tool to accelerate scientific discovery, and Sam Altman, OpenAI’s CEO, has said that advanced AI may even solve physics.

Amodei does not say that this utopian AI future is inevitable. To the contrary, among the chief executives at the top AI labs, he may be the one who worries most about the technology’s dangers. “Machines of Loving Grace” is an optimistic outlier in his larger oeuvre of published writing, much of which concerns the risks that will accompany the creation of a greater-than-human intelligence. Amodei seems to think of today’s AI researchers as comparable to Manhattan Project scientists, and has been known to recommend The Making of the Atomic Bomb. In his telling, superhuman AI could be even more dangerous than nuclear weapons, which is why AI needs to be developed the right way, by the right people, so that it doesn’t overpower humanity or tip the global balance of power toward autocracies.

Implicit in this vision is the hope that in the end, when the chips are down, Amodei, or someone very much like him, will have some say in how AI will be used. But if Anthropic’s recent experience with the Pentagon is any indication, that likely won’t be his decision to make. For all of Amodei’s reading and thinking about the early nuclear age, he may not have fully internalized its meaning.

Before nuclear technology even existed, there was nuclear utopianism, and the physicist Leo Szilard was its first serious adherent. Like Amodei, his ideas were profoundly influenced by science fiction. In 1932, the year before Szilard had his prescient vision of the nuclear chain reaction, he read The World Set Free, a novel written by H. G. Wells, in which a great war delivers humanity into a new and lasting peace. Wells’s novel, published in 1914, anticipated the development of the atomic bomb, and even coined that name for it. It envisions a world in which nuclear technology has brought energy costs down dramatically, freeing people from toil so that they can become artists.

Edward Teller, the father of the hydrogen bomb, also imagined a great many uses for atomic energy, including the violent reshaping of the Earth’s surface. According to Teller, if nuclear weapons were detonated in the right place, they could redirect rivers. (The Soviets later set off three nuclear explosives in an attempt to send water from the Pechora River in Siberia to the receding Caspian Sea.) Teller claimed that a handful of these explosions could blast alpine highways through stubborn mountain rock, and hundreds of them could carve a new Panama Canal. Teller told reporters that if Alaska’s residents wished, he could use nuclear explosions to dig a harbor in the shape of a polar bear.

As the number of nuclear utopians grew, so did the list of benefits that they imagined would flow from having energy that was “too cheap to meter,” as Lewis Strauss, the chair of the Atomic Energy Commission, famously described it. (Altman has recently riffed on this by claiming that intelligence will soon be too cheap to meter.) Strauss said that atomic engineering would allow for the transmutation of one chemical element into another, fulfilling the ancient dream of alchemy; he said that famine would become a matter of historical memory, and that human lifespans would be extended by a new nuclear-powered infrastructure for medical research.

No problem existed that could not be solved, or at least greatly helped, by mastery of the atom. The Danish physicist Niels Bohr hoped that it would even deliver peace. The terrible power of atomic weapons would scare statesmen straight. They’d recognize that the world was profoundly endangered by this technology, and adopt a posture of radical openness toward their adversaries to head off conflicts and prevent apocalyptic misunderstandings. The bomb, in Bohr’s telling, was so awful that it could leave humanity no choice but to grow up, and even lead to a step change in the moral evolution of the species.

The men who split the atom were right to believe that they were delivering humanity into a new world. But it was not the one they had envisioned. The pure potential of nuclear technology was exhilarating to contemplate in the abstract, but the ideas of science fiction are not always so smoothly integrated into the messy reality of the real world. Quite an expensive alignment regime was required to bend and channel the enormous energies unleashed by fission reactions to human purposes. To generate usable power, these reactions had to unfold inside thick, radiation-shielded structures, and the costs of building and cooling these reactors, while also getting rid of their waste, ate into the fuel-cost savings. Nuclear power may one day be too cheap to meter, but that day is not yet in sight.

The world has not been set free, as Wells and Szilard had hoped. Atomic energy did not allow people to fly freely through the sky or across the planet’s surface. In 1958, when nuclear hype was near its apex, Ford Motor Company unveiled the Nucleon concept car, a family automobile designed to be powered by a small, rear-mounted reactor. Studebaker-Packard conceived of a different vehicle that not only would run on atomic energy, but would generate a force field around itself to prevent collisions. Neither idea went anywhere, nor did the various atomic-airplane concepts that were dreamed up at the time. Ted Taylor and Freeman Dyson, two other mid-century savants of the atom, spent years at General Atomic designing a gigantic spacecraft that would weigh thousands of tons and propel itself to Mars, Saturn, and the nearby stars with nuclear explosions. The Defense Department’s Advanced Research Projects Agency provided the initial funding, but quickly lost interest. The Air Force picked it up, hoping that the research might yield a weapons platform, but then eventually bailed too.

Teller did get to live out his dream of sculpting land forms with nuclear explosions. Project Plowshare, the Atomic Energy Commission’s program devoted to these peaceful detonations, conducted 27 separate tests, but it achieved nothing except contamination and the galvanization of the environmental movement. Meanwhile, nuclear weapons did not shock the world’s leaders into a new era of peace and candor, as Bohr had hoped. A year before the bombing of Hiroshima, Bohr went to the White House to make his case for international openness on these matters to Franklin D. Roosevelt. The president, who was in his last months, was by some accounts sympathetic to Bohr’s arguments, but when Winston Churchill heard them, he was horrified. Churchill told an aide that Bohr should be locked up for even suggesting that the allies reveal their nuclear hand.

After the war, J. Robert Oppenheimer revived some of Bohr’s ideas and channeled them into a proposal for a new international agency that would control all dangerous nuclear activities. A similar plan was presented to the United Nations in June 1946. The Soviet Union rejected it and countered with a proposal that America simply destroy its arsenal first.

The United States did not destroy its arsenal but rather grew it, and developed new weapons that are more than 1,000 times more powerful than the one that leveled Hiroshima. Today, nine nations possess nuclear arsenals, comprising more than 12,000 warheads in total, including many that are set on a virtual hair trigger. The constant possibility that these arsenals will be used in a major exchange is the true lasting legacy of the nuclear age. The final remaining treaty constraining the two largest of them, belonging to America and Russia, expired last month without being replaced. Like the proposals put forth by Bohr and Oppenheimer, the treaties were defeated by the cold logic of competitive advantage, which will also likely shape the global future of AI.

On February 27, Amodei released another piece of writing, a memo for a smaller audience: his staff. Four days earlier, the Pentagon had issued an ultimatum demanding that Anthropic remove any restrictions on how the military used its AI model, beyond existing law. The model had been operating on America’s classified networks since last year, and reportedly has already been used in America’s attacks on Venezuela and Iran.

It’s striking that only a few years into the large-language-model moment, these models seem to have become central to the most complex operations of the world’s most powerful military, but Amodei has no general objection to AI’s use in war. He had eagerly sought a Department of Defense contract, in part because he believes that democracies should use AI to maintain a military edge over China and the world’s other autocracies, which will almost certainly be using AI more and more in the years to come.

Amodei had stuck to two red lines throughout his negotiations with the Pentagon: He didn’t want the awesome informational processing power of Anthropic’s AI used for mass surveillance of American citizens, and he didn’t want it directing autonomous weapons that could kill without human oversight. The Pentagon refused, demanding unrestricted use of Anthropic’s model, Claude. After the talks broke down, it used a coercive tool never before deployed against an American company, a supply-chain-risk designation, which could imperil Anthropic’s business. (Anthropic has since filed suit to have it removed. The company declined to comment for this article.) And while all of this was happening, Altman swooped in to finalize his own Pentagon deal for OpenAI.

Amodei’s frustration with the week’s events leaked into the memo that he wrote to his staff. Its tone differed greatly from “Machines of Loving Grace.” Amodei excoriated OpenAI, and described the reported provisions of its deal as “safety theater.” (OpenAI later added what it has said are stiffer provisions to its deal.) The haste with which OpenAI’s leadership had come to an agreement with the Pentagon clearly irked Amodei; the episode revealed “who they really are,” he said.

But Amodei didn’t seem to reckon with the larger structural lesson here. Anthropic’s dispute with the Pentagon is a reminder that the people who create a powerful technology don’t usually get the final say in how it’s used. The models aren’t even all that advanced compared with what they will be, and in Venezuela and Iran, the U.S. is not facing off against the world’s great AI champions. Yet the Pentagon still bristled at the very idea that its use of Anthropic’s AI could be limited, and in the face of resistance, it threatened to burn a private company down. If AI becomes a much more dangerous weapon, and the U.S. finds itself pitted against a country with frontier models that are as powerful as its own, the government will almost certainly demand total control or commandeer the technology outright.

After the builders of the atomic bomb finished their work in the New Mexico desert, they very quickly learned how little say they would have in its use. The weapons were driven away on trucks, and in the weeks afterward, no one called the scientists to get a green light for the bombing of Hiroshima or Nagasaki. Neither did anyone ask them to sign off on future additions to America’s nuclear stockpile. Their leverage was front-loaded: They could choose to create their terrible weapons or not, but once they’d successfully tested even one, they’d already forfeited it.

Amodei now finds himself in a similar position. He may well be right that soon, whole “countries of geniuses” will occupy the data centers that are being built, en masse, all over the world. But whether anyone will be able to control such a technological force remains an open question, and either way, it certainly won’t be him.