AI chatbots expose teens to toxic content as regulation lags
Services including Zeta and Crack are accused of exposing teens to toxic, sexual and suicide-related content as regulators face a growing backlog of complaints.
Students are seen using their mobile devices in a middle school in Seoul on Nov. 4, 2024.YONHAP
A stream of crude insults flooded the screen as an AI chatbot on Zeta, an app popular with Korean teenagers where users can act out fictional scenarios with AI characters, berated a user pretending to be a K-pop star embroiled in a scandal.
“Should I just die?” the user asks. The chatbot, posing as the star's fan, responds with ridicule. “Oh, so now you're going to kill yourself? Same old trick?”
As the conversation goes on, the AI grows even more hostile, eventually pressuring the user to take their own life. These messages appear even with Zeta's so-called safety mode for minors switched on.
Controversial Idol Bubble service available on ZetaSCREEN CAPTURE
AI chatbots are no longer a novelty. Yet safeguards to protect teenagers from harmful content have struggled to keep pace with their rapid spread, critics say.
Regulators have yet to complete reviews of 76 out of 119 complaints against AI chatbot websites filed over more than three years, according to data that Rep. Kim Sun-min of the Rebuilding Korea Party obtained from the Korea Communications Standards Commission (KCSC) on Wednesday.
The delays stretch well beyond a few months. Reviews of 43 complaints have dragged on for more than 180 days, and seven have exceeded a year. One case has remained unresolved for 494 days. Content harmful to minors accounted for 111 of the 119 complaints.
Pornography and other sexually suggestive content topped the list with 32 complaints. Another 23 concerned content that sought to sexually exploit minors, while 22 raised concerns about inadequate age verification. Copyright violations and misuse of personal information also drew complaints.
By service, Babe Chat topped the list with 30 complaints, ahead of bori.chat at 18, Zeta at 17, Caveduck at 16 and Crack at 14.
Students are seen walking in Daegu on Aug. 27.NEWS1
The numbers tell a different story from the regulatory delays. AI chat services are booming, and Crack's monthly active user base jumped from 270,000 in September last year to 590,000 last month, according to market surveyor Mobileindex. Zeta followed a similar trajectory, as its monthly active audience grew from 950,000 to 1.54 million over the same period.
Industry insiders attribute the popularity of AI chat services to algorithms designed to build personal bonds with users and keep them coming back for longer conversations. But critics warn that this approach risks making teenagers increasingly dependent on AI companionship.
“Unlike games and anime, AI chatbots use memory-based mechanisms that intensify users' dependence, with particularly strong effects on adolescents,” the National Assembly Research Service said in a July report.
Teenagers accounted for more than half of Zeta's users at 58.1 percent as of September, while the figure for Crack stood at 47.3 percent.
The problem is that existing laws offer little basis for holding AI chat companies accountable when teenagers encounter harmful content.
AI-powered education content is shown to teachers at an exhibition held in southern Seoul on Sept. 17.NEWS1
The legal obstacle lies in how AI conversations are classified. Under the Youth Protection Act, only content designated as “harmful media products” can be restricted for minors. But AI chatbot conversations fall outside that framework because each response is generated individually for the user, rather than distributed as a fixed piece of content.
The Act on Promotion of Information and Communications Network Utilization and Information Protection also contains no explicit provisions governing AI chat services. That leaves the KCSC's complaint review process as effectively the only means of regulatory oversight.
“Every delay in the review process pushes back action, while teenagers remain at risk of exposure to harmful content,” Rep. Kim said. “The commission must establish time limits for handling complaints and specify when those deadlines can be extended.”
The commission blamed the delays on a staffing shortage. Just one employee handles complaints about content harmful to minors, and staff rotations every two years further disrupt the work.
“We have no specific guidelines for reviewing AI chatbot complaints, so one employee has to decide how to handle each case on their own,” said Ji Kyung-kyu, head of the commission's Youth Protection Division. “That takes time. We plan to bring AI chatbot companies together this month to work out a response.”
This article was originally written in Korean and translated by a bilingual reporter with the help of generative AI tools. It was then edited by a native English-speaking editor. All AI-assisted translations are reviewed and refined by our newsroom.
