No precedent for this: Courts deal with AI hallucinations in legal filings

When judges at the Daejeon District Court checked the Supreme Court rulings cited by a plaintiff in a car repair cost dispute last month, they found a problem: The cases either did not exist or said something entirely different.

“The Supreme Court rulings cited are all nonexistent or entirely different from the actual holdings and appear to be AI hallucinations,” the court, led by presiding judge Hwang Seong-wook, said in its Aug. 11 ruling.

Such fabricated precedents are increasingly appearing in Korean court filings, raising concerns that AI-generated misinformation could influence judicial decisions. Judges say the problem is being driven by the growing use of generative AI in legal work, with parties and lawyers submitting AI-generated citations without first verifying their accuracy.

“I thought this only happened when parties litigate on their own, but quite a few lawyers submit them as they are,” a senior judge said.

In June, the Intellectual Property Court of Korea criticized a plaintiff in a patent invalidation suit after finding that all eight precedents cited in a filing did not exist.

“The eight precedents the plaintiff cites are nonexistent rulings, and submitting them to the court without verifying them is inappropriate,” the court said. “The plaintiff also admits to submitting some rulings extracted through AI without verifying them.”

The case was handled by a lawyer with 27 years of experience at a midsize law firm.

Judges say the spread of fabricated citations is forcing courts to spend additional time checking material that they once could generally assume was authentic. “In the past, we could trust what was written in a brief, but now we have to first check whether the precedent is real,” one judge said. “At this rate, we'd have to check every statute and precedent submitted one by one, which adds to our workload.”

A presiding judge at a high court described a similar experience. “Many judges must have experienced this. You're reading a brief, something seems odd, you look up the case number and it doesn't exist,” the judge said. “It seems the court has no adequate way to respond.”

For now, courts have few practical ways to deal with the problem beyond reprimanding lawyers or pointing out fabricated citations in rulings.

Last month, a civil panel at the Seoul High Court reprimanded a lawyer after a party submitted an unedited AI-generated brief, asking, “How can you submit this as it is?”

At the Ulsan District Court, a lawyer who cited an unrelated Supreme Court ruling admitted after being questioned by the presiding judge, “I searched for the ruling with Google Gemini and did not properly check the content.”

Some judges have gone further by explicitly criticizing such conduct in their rulings. In June, the Seoul Eastern District Court said a plaintiff had “deliberately cited false precedents and actively tried to deceive the court.”

A lawyer in Seocho District, southern Seoul, said AI use can sometimes be apparent from the content of opposing counsel's filings.

“Sometimes you can tell just by looking at opposing counsel's filing that they used AI. Sometimes they say something off-topic because of AI,” the lawyer said. “Even when AI gives an answer confidently, you shouldn't neglect verification, but some lawyers seem to.”

Warnings, however, have not always stopped the practice.

According to the National Court Administration, a lawyer appearing before the Daegu High Court cited a nonexistent Supreme Court ruling numbered “2000da57763.” After the panel asked the lawyer to check the case number, the lawyer cited another nonexistent ruling, “2012da11454,” and argued that “the precedent number may be inaccurate.”

With fabricated citations continuing to surface, the judiciary is considering stronger sanctions against parties and lawyers who submit AI-generated information without verifying it.

The National Assembly is also weighing measures to address the issue. Democratic Party Rep. Chung Chin-ook last month introduced amendments to the Civil Procedure Act and Criminal Procedure Act that would require parties to disclose the use of AI in court filings, including complaints and briefs. The legislation would also allow fines of up to 5 million won ($3,615) for parties, lawyers and other participants who cite nonexistent statutes or other legal authorities.

But drawing the line between acceptable and improper AI use remains difficult. “Polishing an argument so it sounds plausible is fine, but falsely citing objective materials needs sanctions,” an official at the National Court Administration said.

Lawyers' groups have raised concerns that requiring disclosure could be overly broad if it applies even when AI is used only to help draft a filing. The Korean Bar Association has launched an AI legal task force and is preparing guidelines for the use of AI in legal practice, while also seeking to strengthen its oversight and disciplinary measures.

The problem is not unique to Korea.

In Colorado in 2023, a lawyer who submitted false AI-generated information to a court was suspended for one year for violating the duty of candor to the court. The lawyer initially blamed an intern before acknowledging the use of ChatGPT.

He said he had not thought the technology could be deceptive because ChatGPT had given accurate answers before, and he called the incident a mistake made out of ignorance. 

BY CHOI SEO-IN, JO SU-BIN, KIM YE-JUNG [[email protected]]