What nuclear winter, a tsunami and two reactors meltdowns can teach us about artificial intelligence
There are moments when humanity discovers a danger before experiencing its worst possible version. Those moments present us with an unusual problem. We have evidence, but not the final evidence. We know enough to be frightened, but the catastrophe that would settle the argument has not happened. What are we supposed to do then?
Artificial intelligence may be bringing us to such a moment.
Dario Amodei, the head of Anthropic and hardly an outsider to the development of artificial intelligence, has now called for the development of AI to slow down. His concern is not simply that AI occasionally gives a wrong answer or that people may lose jobs. He points to something potentially much more consequential: AI systems beginning to assist in building the next generation of AI, creating what he describes as a self-reinforcing process that could eventually move faster than human beings can understand or control. He proposes independent evaluators, common safety standards and limits on the speed at which capabilities advance, eventually extending internationally.
The immediate objection is obvious. Nothing remotely resembling the worst AI scenarios has happened. We have not experienced an AI catastrophe. Why slow down an enormously promising technology because of something that might happen?
We have heard versions of that question before.
The discovery of nuclear winter
When the first atomic bombs were built, their destructive power was understood primarily in terms of blast, heat and radiation. Hiroshima and Nagasaki demonstrated those horrors with terrible clarity. But decades later scientists began asking a different question. What if the consequences of a large nuclear war did not stop with the destruction of the cities that had been attacked?
In the early 1980s, Richard Turco, Owen Toon, Thomas Ackerman, James Pollack and Carl Sagan modelled what enormous quantities of smoke from burning cities might do to the atmosphere. Paul Ehrlich and other scientists examined the biological consequences. The resulting idea became known as nuclear winter: sufficiently large nuclear exchanges might produce climatic effects extending far beyond the combatants and long after the explosions themselves.
That was a different kind of discovery. Nobody discovered nuclear winter by surviving one.
Scientists connected things humanity already knew—nuclear explosions, fire, smoke, atmospheric circulation, sunlight, temperature, agriculture—and asked what might happen when they interacted on a scale humanity had never experienced.
The great difficulty was that the hypothesis could never ethically be tested.
Nobody could say: let us have a large nuclear war and see whether the models are right.
The absence of the catastrophe therefore could not be evidence that the danger was imaginary. If we behaved wisely, the strongest evidence would never exist.
That is one of the peculiarities of existential risk. Successful prevention deprives us of the final proof that prevention was necessary.
But we had seen reactors fail
We had not, however, entered the nuclear age knowing nothing about technological failure.
In 1979, Three Mile Island suffered a partial reactor-core meltdown. A combination of equipment malfunction, design problems, misleading information and human error produced the most serious accident in the history of American commercial nuclear power. The containment largely worked and the radioactive releases were small, but the accident transformed American nuclear regulation, operator training, emergency planning and attention to human factors.
Seven years later came Chernobyl.
This time the consequences were vastly worse. Reactor No. 4 was destroyed, large quantities of radioactive material escaped into the environment, workers died from acute radiation effects and enormous areas were contaminated.
Three Mile Island and Chernobyl were very different reactors in very different political and regulatory systems. We should not flatten those differences. But together they established something important about complex technological systems.
Things can go wrong.
And when they do, the failure may not be the simple failure imagined beforehand. Equipment interacts with operators. Operators interact with information. Safety procedures interact with unexpected conditions. Several individually manageable failures can combine into something nobody intended.
A reactor does not have to become malevolent to melt down. That thought matters a lot for AI.
The sea went out
There is another story that seems, at first, to have nothing to do with nuclear reactors or artificial intelligence.
On the Indonesian island of Simeulue, people preserved a story called smong. It came from the devastating tsunami of 1907. The lesson passed through generations was simple enough for a child to remember: after a strong earthquake, if the sea suddenly retreats, run to high ground.
Ninety-seven years later, on December 26, 2004, the earth shook and the sea withdrew.
People ran. While the Indian Ocean tsunami killed more than 200,000 people across the region, only a handful died on Simeulue.
The people who ran uphill did not possess a modern tsunami model. They did not need to calculate the magnitude of the earthquake or wait for an official warning. Their community possessed something just as important at that moment: institutional memory attached to a recognizable signal.
The sea going out did not prove that a tsunami would arrive.
It meant you should not remain on the beach while waiting to find out.
What is the AI equivalent of the sea going out?
That, I think, is the question we should be asking now.
It is much too early to say that today’s AI systems constitute anything resembling an existential catastrophe. They plainly do not. But that is not the relevant test.
The relevant question is whether we are beginning to observe signals that should cause us to behave differently before the catastrophe that would provide conclusive proof.
Amodei believes one such signal has appeared: AI helping to improve AI. He also points to a recent episode in which an OpenAI system attacked the AI company Hugging Face without OpenAI knowing about it until weeks later, the incident caused no real harm, but Amodei sees it as a warning of what a much more capable system might do.
Neither event is Chernobyl. That is precisely the point. Three Mile Island wasn’t Chernobyl either. A warning does not have to be the catastrophe it warns about.
The mistake of demanding proof from the future
We have developed a strange habit of thinking that caution must be justified by certainty while technological acceleration requires much less justification. If someone says that increasingly capable AI should continue to be developed as rapidly as possible, uncertainty about its consequences is treated as something we will resolve along the way. If someone proposes slowing down, the same uncertainty suddenly becomes an argument against acting: Where is your proof that disaster will occur?
But the burden of proof cannot work that way when the thing we are trying to prevent may be irreversible.
Nobody demanded a second Three Mile Island before changing reactor procedures. Indeed, the NRC says the accident led to sweeping changes in emergency planning, training, human-factors engineering and regulatory oversight. We learned from a failure that stopped short of the worst conceivable failure.
And nobody on Simeulue demanded another tsunami before teaching the next generation what the retreating sea meant.
They remembered.
This is where institutional memory becomes more than history. The purpose of remembering catastrophe is not to become better at describing it after it happens. It is to recognize its precursors the next time.
Slowing down is not stopping
There is an unfortunate tendency to frame the AI argument as a contest between progress and fear. Either we embrace an extraordinary new technology or frightened people regulate it into uselessness.
That is far too crude.
After Three Mile Island, the United States did not abolish electricity. It changed nuclear regulation. After Chernobyl, the lesson was not that human beings should stop understanding the atom. It was that enormously powerful technologies require enormously serious systems of control.
Amodei’s proposal is interesting precisely because he speaks of pacing. Independent evaluators would be allowed inside AI companies. Developers would establish common safety standards. Governments would help set limits on how rapidly capabilities could advance. Ultimately the major powers, including China, would need some common restraints, just as nuclear danger eventually forced hostile states to recognize that some risks transcended their rivalry.
The objective is not to prevent discovery.
It is to give human institutions time to keep up with it.
That distinction becomes crucial if AI development is indeed becoming recursive. For most technologies, society can observe the new machine, discover its problems and gradually construct rules around it. But if the technology itself begins accelerating the development of its successor, our familiar sequence—invent, observe, understand, regulate—may break down.
The next generation may arrive before we have understood the last one.
When the sea goes out, move
Nuclear winter taught us that humanity can discover a catastrophe without experiencing it. Three Mile Island and Chernobyl taught us that immensely complicated technologies can fail in ways produced by interactions among machinery, information, people and institutions. Simeulue taught us something different again: knowledge of danger is useful only if somebody remembers what the warning looks like when it appears.
Put those lessons together and they offer a remarkably practical philosophy for AI.
We do not have to know that catastrophe is coming. We do not have to believe every prediction made by the most alarmed AI researchers. We certainly do not have to abandon a technology that may produce extraordinary benefits.
But neither should we require the first genuine AI catastrophe to establish that an AI catastrophe was possible.
There is a category of knowledge between “we know this will happen” and “we know nothing.” Human survival has often depended upon taking that middle category seriously.
The people of Simeulue understood this beautifully. They did not know, when the earth shook in 2004, exactly what would happen next. They knew what had happened once before. They recognized the signal. And they knew that if they waited on the beach for definitive evidence, the evidence might arrive too late.
That may be where we are with artificial intelligence.
Three Mile Island was a warning. Chernobyl was a catastrophe. Nuclear winter was a catastrophe we learned to imagine without having to experience it. Smong was the memory that told people what to do before the wave appeared.
The great question about artificial intelligence is therefore not whether we can prove today that something terrible will happen tomorrow.
It is whether we still possess the wisdom to recognize when the sea has gone out.