Up to 16.1% of Taiwan\'s academic references contain errors or may be AI

Up to 16.1% of Taiwan's academic references contain errors or may be AI-fabricated: Doublemind

A recent study found that up to 16.1 percent of references in Taiwanese academic articles are suspected to contain citation errors or artificial intelligence (AI) hallucinations, in which generative AI (GenAI) and large language models (LLMs) fabricate sources, references or citations.

The report, released today, examined academic integrity in the age of AI, scanning 26,635 references using the Paper Plagiarism Verify System, developed by the Doublemind AI Education System.

The system found that traditional plagiarism, which is defined as similarity of over 70 percent with other published works, accounted for just 5.5 percent.

Logos of artificial intelligence apps including ChatGPT, center, are displayed on a phone screen in Finland, on June 6, 2023.

Meanwhile, 16.1 percent of references were found to be incorrectly cited or were suspected to have been fabricated by AI, highlighting a new challenge for academia in the face of rapidly developing AI, it said.

A further 12.3 percent contained errors in factual claims that either conflicted with publicly available evidence or had no supporting evidence at all, it found.

Previously, systems were scanning whether documents were plagiarized from other sources, but with the widespread use of AI, the challenge is now detecting whether the sources cited actually exist, and whether the information it includes is true, Doublemind CEO Lin Cheng-hong (林政宏) said.

In academic literature, citations may provide authors, publication dates, academic journals and even digital object identifier (DOI) numbers – unique codes that identify research papers – for work that does not exist, he said.

Such fabrications can be difficult to detect with the naked eye, he added.

Doublemind would continue strengthening its verification tools while also holding a series of outreach lectures to promote education on academic ethics, he said.

The company hopes to work with universities and high schools to establish an objective, verifiable and rigorous environment of academic integrity, he added.