Google, ChatGPT: Inadequate Dementia Resources Despite Strengths

August 09, 2023 - A new study from the University of California, Riverside (UCR) indicated that Google and ChatGPT contained positive features such as up-to-date and nonbiased information. When used effectively, these tools can provide resources for Alzheimer’s and dementia.

“If you pick the best features of both, you can build a better system, and I think that this is what will happen in the next couple of years,” said Vagelis Hristidis, PhD, a professor of computer science and engineering at UCR’s Bourns College of Engineering, in a press release.

Speaking of Google and ChatGPT, Hristidis surmised that these resources will continue to gain a position in treating certain conditions. He reached this conclusion after testing how they interact with treating Dementia and Alzheimer’s.

Researchers selected these conditions based on their prominence and expected growth. The Centers for Disease Control and Prevention (CDC) estimated that about 5 million people aged 65 and older had dementia in 2014. The CDC estimates that by 2060, this number will increase to 14 million. million people aged 65 and older had dementia in 2014. The CDC estimates that by 2060, this number will increase to 14 million.

Hristidis and co-authors submitted 60 queries to Google and ChatGPT. These included typical submissions that would come from people with dementia. Among these queries, half sought information regarding disease processes. The other half aimed to gauge information on services to assist patients and their families.

Mixed results emerged from this research process. Researchers found that while Google provided current information, results were often biased. Specifically, they included service providers looking to gain customers.

ChatGPT, on the other hand, had impartial information. The flaws, however, were that this data was often outdated with limited sources.

“Google has more up-to-date information, and covers everything, whereas ChatGPT is trained every few months. So, it is behind. Let's say there's some new medicine that just came out last week, you will not find it on ChatGPT,” said Hristidis.

However, ChatGPT's advantage over Google was its reliability and accuracy. OpenAI creators of ChatGPT generally include reliable websites during training. its reliability and accuracy. OpenAI creators of ChatGPT generally include reliable websites during training.

Meanwhile, Google contains many inconsistencies, primarily because of its comprehensive coverage. Often, businesses pay for this platform to place their sources at the top of results pages.

Nonetheless, both platforms had low readability scores, indicating that people with lower education levels and health literacy skills may not benefit from them. and low health literacy skills may not benefit as much.

Nonetheless, Hristidis noted that these sources had potential. Moving forward, certain features of Google and ChatGPT can provide benefits.

“My prediction is that the readability is the easier thing to improve because there are already some tools, some AI methods, that can read and paraphrase text,” noted Hristidis. “In terms of improving reliability, accuracy, and so on, that's much harder. Don't forget that it took scientists many decades of AI research to build ChatGPT. It is going to be slow improvements from where we are now.”  

Prior research has also highlighted ChatGPT’s failures.

For example, a study from May indicated that two forms of ChatGPT showed a lack of ability to succeed in education in gastroenterology.

Specifically, these resources failed the 2021 and 2022 multiple-choice self-assessment tests for the American College of Gastroenterology (ACG). Between ChatGPT-3 and ChatGPT-4, neither reached a passing score of 70 percent or higher.

“Recently, there has been a lot of attention on ChatGPT and the use of AI across various industries. When it comes to medical education, there is a lack of research around this potential ground-breaking tool,” said Arvind Trindade, MD, senior author of the study and associate professor at the Feinstein Institutes’ Institute of Health System Science, in a press release discussing study findings. “Based on our research, ChatGPT should not be used for medical education in gastroenterology at this time and has a ways to go before it should be implemented into the health care field.”

Although ChatGPT contains features that show potential, validation is vital. These research pieces are clear indicators for future validation resources.