Huawei, DeepSeek AI training linked to pirated books by Korean authors

Chinese companies, including Huawei and DeepSeek, were found to have used datasets containing pirated books by Korean authors — including Nobel laureate Han Kang — to train their AI models.

The Pile, a collection of books, academic papers, web documents and other materials compiled for AI training that contains approximately 800 gigabytes of data, was one of the sources of English-language text used to train Huawei's AI model Pangu Sigma, according to JoongAng Ilbo's analysis of the company's technical report published in March 2023 on arXiv, a platform for sharing scientific research papers.

A paper describing the dataset on arXiv confirms that it includes Books3, a collection of pirated e-books that Anthropic also downloaded without authorization. A list of works included in Books3, available on GitHub, shows that the collection contains English translations of at least 19 books by 10 Korean authors.

They include Han Kang's "Human Acts" (2014) and "The Vegetarian" (2007), Hwang Sok-yong's "Princess Bari" (2023), Shin Kyung-sook's "Please Look After Mom" (2008) and Kim Young-ha's "I Have the Right to Destroy Myself" (1996).

However, it remains unclear whether Huawei trained its model on all the novels included in the dataset or only some of them.

The use of Books3 became a central issue in the copyright lawsuit "Bartz v. Anthropic," which Anthropic agreed to settle by paying copyright holders $3,000 per book.

The creators of The Pile explained in the dataset's documentation that digital books were included because they were extremely valuable for research into modeling long contexts and maintaining narrative coherence.

The Pile is one of the datasets frequently used by AI researchers. Chinese AI company Zhipu AI, also known as Z.ai, and Tsinghua University identified The Pile as a training data source for their jointly developed language model GLM in a 2022 technical report. It is also unclear whether Zhipu AI trained its model on all the novels included in The Pile.

DeepSeek reproduces Han Kang's words

DeepSeek, another major Chinese AI company, has also provided information about the sources of its training data.

The company used 860,000 English-language e-books and 180,000 Chinese-language e-books obtained from Anna's Archive to train its DeepSeek-VL model, according to a technical report published in March 2024. Anna's Archive is a website that provides access to pirated books.

Anna's Archive includes English translations of works by prominent Korean authors such as Han, Hwang and Kim, as well as Chinese translations of books including Kim Ae-ran's "The Luck of Flight" (2012).

However, DeepSeek did not download every book available on the site and instead selected materials for training. It therefore remains unclear whether books by Korean authors were among those it obtained.

Researchers have nevertheless found indirect evidence suggesting that DeepSeek may have trained its models on Korean literary works.

DeepSeek 3.1 produced a sequence of 19 consecutive words matching the original text of Han's "The Vegetarian" after undergoing a process known as fine-tuning, according to a paper by researchers from Columbia University, Carnegie Mellon University and other institutions. The paper was first released in March and updated for a fifth time in September.

The reproduced words are: " […] vegetarian, I'd always thought of her as completely unremarkable in every way. To be frank, the first time […]"

Although the findings have not yet undergone peer review, the researchers interpreted the result as indirect evidence that the AI model had been pretrained on the novel.

Anna's Archive also acknowledged widespread interest from developers in its book collection following the rise of generative AI.

"Virtually all major companies building LLMs contacted us to train on our data," the website said in a blog post published in January last year.

The website added that most U.S. companies, though not all, reconsidered after learning that its operations were illegal.

"By contrast, Chinese firms have enthusiastically embraced our collection, apparently untroubled by its legality," the website wrote.

Pirated books: An AI industry practice?

The use of pirated e-books to train AI models is not limited to Anthropic and Chinese AI companies. Copyright lawsuits have also revealed that OpenAI and Meta downloaded pirated copies of books without authorization.

"The use of pirated books as AI training data is widely known within the AI industry," said Choo Jae-gul, a professor at KAIST's Kim Jaechul Graduate School of AI. "Using publicly available datasets to train AI models was standard practice, and some of those datasets included pirated books. There was a period when the AI industry was so caught up in the race to improve model performance that companies indiscriminately used copyrighted data without seriously considering the legal implications."

Choo explained that training AI models on copyrighted digital books was widespread around the world during the early stages of AI development.

"It was common enough for researchers to disclose publicly available datasets containing pirated books as sources of training data in their papers," Choo said.

"As the issue has increasingly drawn scrutiny in recent years, many major technology companies have begun keeping their training data confidential," he added.

BY YOON SUNG-MIN [[email protected]]