‘I Tell People Not to Fully Trust AI for Health Concerns’

‘I Tell People Not to Fully Trust AI for Health Concerns’

A faculty member at the School of Dental Medicine shares thoughts on how patients and healthcare practitioners can use artificial intelligence responsibly

People used to talk about Dr. Google, but now it’s Dr. Chatbot. I tell people, if you want to use a chatbot for health-related concerns, go for it—but you need to use it very carefully. You must check the information with a doctor or another expert. You cannot just trust in the AI 100%.

It can be dangerous going to a chatbot to get answers related to your health because it can give you misinformation. The chatbot gathers information not only from validated medical literature, but also from online forums, social media, platforms like Discord, and other sources. You don’t know what references the chatbot used. If you ask for references, the chatbot might start to hallucinate and provide made-up references. It can be hard for the layperson to tell whether references are made up, but your doctor would be able to see that a journal name is wrong or the volume number is way too high.

We are already dealing with a trust paradox in medicine, with people placing more weight on information they get from social media than from experts. AI presents an even bigger challenge because its responses sound believable and trustworthy, while its sources remain hidden.

One reason chatbots feel trustworthy is that they have a tendency to agree with you, which is called sycophancy. People like that. Most people want someone saying, “You are right.” Human beings are always looking for empathy and AI does that very well. But you must know how to respond to that. If you’re not an expert in this field and you don’t have a lot of experience using chatbots, be careful and don’t let sycophancy override your own critical thinking skills.

There are ways to use AI that minimize the risk of misinformation. Here are my suggestions:

Providers sometimes have patients coming in with misinformation they got from AI. If the patient believes what a chatbot has told them, a provider can use AI to show them the information is wrong by using an AI tool like NotebookLM to summarize information from specific scientific papers and resources.

For providers using AI for clinical or research purposes, it’s critical to use a framework or protocol that are already validated in the literature, with the goal of reducing risks and preventing harm to patients. This is especially important in the case of large language models, since they can hallucinate and make mistakes. For that reason, at least at the present stage, human oversight by a qualified specialist remains essential to critically review and validate the information generated by artificial intelligence.

Carlos Fernando Mourão is an assistant professor in the Department of Basic & Clinical Translational Sciences at Tufts University School of Dental Medicine.