Montreal researchers put ChatGPT to the test, tell scientists to beware

Scientists who rely on artificial intelligence when they write their research papers could wind up spreading misinformation, according to a new Montreal study that looked at the quality and accuracy of information specifically from ChatGPT.

The results of the study, published in Mayo Clinic Proceedings: Digital Health, showed that the chatbot provided dubious answers and included factual errors and fabricated references.

"I was shocked by the extent of fabricated references. I wasn't expecting it to be that significant," said co-author Dr.Esli Osmanlliu, an emergency physician at the Montreal Children's Hospital.

To test the AI model, he and two colleagues at CHU Sainte-Justine — Dr. Jocelyn Gravel and Madeleine D’Amours-Gravel — asked ChatGPT 20 medical questions that came from existing studies. Then, they asked the authors of the papers to rate its answers.

Out of 20 authors, 17 participated and concluded that most of the information the chatbot returned was questionable, resulting in a median score of 60 per cent.

"Specifically, in a certain case, it was suggesting a steroid be given by injection whereas, in fact, the treatment is by mouth so that's a pretty significant difference in the way we administer a medication," Osmanlliu said.

Nearly 70 per cent of the study references provided were concocted by the AI tool and just as concerning, they looked at first glance to be authentic, which means scientists can be more easily duped.

"(ChatGPT) would be very confident and say, here are my sources, and more than 70 per cent of the time, those sources did not exist, although the authors, the titles, the journals looked like they made sense."

"So critical thinking is absolutely important," Osmanlliu said, adding that he advised his fellow scientists to fact-check the old-fashioned way so they don't end up inserting erroneous science or non-existent science into their manuscripts.

Even eight out of 18 real references spit out by ChatGPT contained errors, the study showed.

If that misinformation is published, then the false claims can spread quickly around the world. Citing inaccurate references could even theoretically lead to charges of plagiarism if other authors aren't cited accurately and given credit for their work.

Osmanlliu said ChatGPT and other AI tools do present opportunities to make work easier and faster as long as they don't distort the facts and are used as intended.

"Although sometimes it feels like we're talking to a human, feels like we're talking to someone that is sentient, that is actually intelligent, it's not always the case, right?"

"That's what's so dangerous," he said, "especially when you're trying to make high-stakes decisions."