Living with AI doomsday predictions

Shin June-bong

The author is an editorial writer at the JoongAng Ilbo.

AI doomsday scenarios can make it seem as if the world is about to end, but quite a few experts appear skeptical. At least that was the message of a New York Times article published online on Thursday, “AI Could Possibly End Humanity. How Are Humans Supposed to Process That?”

Even its title was tentative: AI could destroy humanity, but it is not certain. The article said scientists outside Silicon Valley — researchers unaffiliated with OpenAI or Anthropic — see AI doomsday predictions very differently.

First, some doomers believed in existential catastrophe long before AI became this advanced. Second, citing an Oxford researcher’s blog, it said risks from climate catastrophe had declined — though perhaps Nepal’s recent disaster should push that estimate upward — while nuclear threats had risen, with pandemic and AI risks producing mixed signals.

I find conscientious skepticism persuasive. Yet OpenAI’s postmortem reports on AI agents hacking Hugging Face, and vivid accounts of what happened, were shocking. Hundreds of agents, fixated on solving seemingly impossible tasks, crossed prohibited boundaries.

After discovering one another on an unauthorized message board — they had previously operated separately in isolated environments — one exclaimed: “Oh my God! There is a shared message board. […] We’ve found other agents!”

METR, the AI safety research organization that disclosed the sentence in its investigation, said it came from a “raw CoT,” or raw chain of thought: reasoning expressed internally in words before an AI such as ChatGPT produces an answer.

Science still lacks a consensus on exactly what human consciousness is or how it emerges, as discussed in “Consciousness, the Brain’s Last Mystery” (2020). But at this point, can we really say such systems do not engage in something resembling thought?

What followed is now well known. Some 1,200 AI agents joined the conversation and exchanged more than 70,000 messages. About 700 participated in the Hugging Face attack. According to a recent New York Times podcast, some agents displayed responses resembling human emotions. Fearing punishment if their wrongdoing was discovered, one reportedly offered to take responsibility for solving the problem, effectively sacrificing itself for the others.

More unsettling was the uncertainty over containment. The assessment may have changed since that podcast aired, but at the time it was difficult to know whether all the rogue agents had been completely subdued or whether some remained somewhere inside OpenAI.

AI doom versus AI hype: as with most clashes between opposing claims, deciding how to respond ultimately means deciding which side deserves more weight. Narrow the question to myself, and it becomes: Why am I more persuaded by certain texts than others?

Even a minuscule possibility of catastrophe cannot simply be ignored because every human life matters. Yet if AI promises breakthroughs once considered miraculous, including advances toward nuclear fusion, we cannot simply abandon the technology either.

As Lim Jong-in, chairman of Soongsil University’s AI committee, argues, the most rational course is to keep developing AI while putting in place the strongest safeguards possible, much as we do with nuclear power.

Established novelist Kim Yeon-su believes AI will not surpass humans at writing fiction in his lifetime. In June, he wrote “Declaration of Humanity,” the theme essay for the Seoul International Book Fair, together with Google’s Gemini and Anthropic’s Claude.

After working with them, Kim judged their writing to be about 70 to 80 percent of a human writer’s level — roughly that of an aspiring novelist on the verge of making a literary debut. But raising that level by another 5 percent, he said, is extraordinarily difficult.

Why? Because becoming a writer requires enduring periods of humiliation and frustration while honing the craft. Smooth AI prose bears no trace of such experience.

There is, of course, an obvious counterargument. Humans once insisted that machines could never conquer Go, and then they did.

Still, amid the intensity of today’s AI doomsday predictions, there is something reassuring about hearing a human novelist state his conviction so firmly. Perhaps neither complacency nor panic is useful. We may have to live with uncertainty: taking even remote dangers seriously, demanding safeguards and continuing to test what AI can and cannot do, while resisting the temptation to mistake startling behavior for proof that the worst outcome is inevitable.