Tianshu Gu, Helin Feng, Minghui Li, Weikuan Gu, Guiying Wang in their paper "Alarm: Retracted articles on cancer imaging are not only continuously cited by publications but also used by ChatGPT to answer questions" hypothesized that AI may not tell the difference between the retracted literature and nonretracted publications on the imaging of cancer in a timely manner.
A search was performed in PubMed on Nov 25, 2024, to identify retracted English-language research articles on cancer imaging. Altogether, it was identified 21 retracted research articles published from 13 journals. These 21 articles were cited by authors a total of 72 times. Three months after the announcement of retractions, they were cited 26 times. For the test of the utilization of retracted articles by ChatGPT, it was found that in 5 times ChatGPT answered questions based on retracted articles. However, in 3 times, ChatGPT noticed the retraction of the article and reminded that the article had been retracted. In other two times, ChatGPT answered questions based on the retracted articles and cited the articles.
Therefore at least there is a 10% of chance that ChatGPT still answers questions based on retracted articles and does not notice the retraction status of the article. It is known that chatbot AI systems confabulate fabricated citations out of thin air. However, when ChatGPT4 answers questions with an article with a real title, it is evasive and potentially easily mislead readers.
More: https://www.sciencedirect.com/science/article/pii/S2090123225001808#b0010
