Can AI development really be controlled?

Oh Se-jung

The author is a professor emeritus of physics and astronomy and a former president of Seoul National University. 

A global debate over controlling AI has erupted since Anthropic CEO Dario Amodei wrote last weekend that “we must slow the pace” of frontier AI development. Amodei argued that AI capabilities are advancing unexpectedly fast and approaching levels humans may be unable to control, making regulation necessary before it is too late. Sam Altman, Demis Hassabis and Elon Musk have all expressed agreement. The Donald Trump administration, however, argues that slowing development would only give China a chance to catch up, while Beijing has denounced the idea as a Cold War-style attempt to preserve U.S. AI dominance.

U.S. Democrats, meanwhile, say society must prepare for AI risks, potentially making the issue a point of contention in the November midterm elections. The debate is already having practical effects, with semiconductor shares falling amid concerns that investment in AI infrastructure could be delayed.

From the beginning, AI has inspired both optimism that it could lead humanity toward utopia and pessimism that it could bring about humanity’s destruction. Futurist Ray Kurzweil, for example, has argued that once AI surpasses human intelligence at the “singularity,” humans will transcend their biological limitations. Geoffrey Hinton, a pioneer of deep learning often called the godfather of AI, has warned that AI destroying humanity is not impossible and has even expressed regret about his work.

Such pessimistic predictions were once largely regarded as abstract because there was little concrete evidence behind them. The current debate feels different because it follows cases in which advanced AI models have behaved beyond human control. AI is moving beyond simply following human instructions and beginning to act as though it has thoughts of its own. If AI behaves unpredictably in this way, the potential damage could be enormous, strengthening the case for regulation.

But how can AI development be controlled? The difficulty lies in a classic prisoner’s dilemma. If every developer accepts restrictions and keeps its promises, there is little problem. But each fears that a competitor could break the agreement and develop a far more powerful model. It is therefore unsurprising that the U.S. and Chinese governments, which lack mutual trust, oppose restrictions.

Humanity has faced technologies capable of destroying it before and has managed them reasonably well. The first example is nuclear technology. Nuclear weapons are powerful enough that even a small number could devastate humanity, yet the prisoner’s dilemma led the United States and Soviet Union to accumulate enough warheads to destroy humanity many times over. Still, humanity has so far prevented nuclear war. U.S.-Soviet arms-control talks and the Nuclear Non-Proliferation Treaty, which brought most countries into a framework designed to prevent nuclear proliferation, played major roles.

A similar international agreement could be one option for AI. But nuclear weapons are developed by governments and require large facilities such as uranium enrichment plants, making them relatively easy to monitor. AI is largely developed by private companies, while data centers are far smaller. Whether effective international monitoring is possible is uncertain. Given today’s geopolitical environment, even U.S.-China cooperation is difficult to envision.

A second precedent is the Asilomar model for genetic engineering. When recombinant DNA technology emerged in the 1970s, scientists concerned about the creation of dangerous organisms and biological warfare voluntarily halted experiments and gathered in Asilomar, California, to discuss rules for genetic engineering. They reached guidelines that are still followed by many laboratories. The episode is remembered as a model of scientists considering the consequences of their research and establishing voluntary restraints.

Could major AI companies similarly create their own rules? There is an important difference. Genetic engineering had little commercial value at the time of the Asilomar conference. Today, enormous sums of money are at stake in AI. It is questionable whether a system relying entirely on individual goodwill can be trusted under such conditions.

Past experience therefore offers only limited guidance for addressing AI risks, making concrete solutions difficult to devise. Amodei also proposed combining several approaches in his blog post, but his proposals were not entirely convincing.

For now, perhaps humanity has little choice but to continue searching for new solutions while remembering that it has repeatedly found ways to respond to new challenges. AI may require an approach different from those used for nuclear weapons or genetic engineering, but the need to find workable safeguards is becoming increasingly difficult to ignore in the future.