ChatGPT, Generative AI Show Promise as Complex Diagnostic Tools

June 16, 2023 - Researchers from Beth Israel Deaconess Medical Center (BIDMC) demonstrated that generative artificial intelligence (AI) technologies, such as ChatGPT, may serve as promising assistive tools for clinicians trying to solve complex diagnostic cases, according to a study published in JAMA.

Generative AI uses natural language processing (NLP) to interpret and generate human-like language. Large language models (LLMs) like ChatGPT can use this process to act as chatbots and provide in-depth responses to a user’s queries.

The recent hype around ChatGPT and other AI chatbots has led some researchers to investigate the potential utility of generative AI in health and medicine, with mixed results.

This week, a research team showed that ChatGPT performed poorly on the 2022 American Urological Association (AUA) Self-assessment Study Program exam and suggested that the tool is not ready for use in medical education. However, New York University (NYU) Langone Health has successfully deployed an LLM to predict readmissions, mortality, length of stay, comorbidities, and insurance denials.

To further test the promise of AI chatbots in healthcare, BIDMC researchers tested ChatGPT-4’s performance on complex diagnostic reasoning challenges.

“Recent advances in artificial intelligence have led to generative AI models that are capable of detailed text-based responses that score highly in standardized medical examinations,” said Adam Rodman, MD, co-director of the Innovations in Media and Education Delivery (iMED) Initiative at BIDMC and an instructor in medicine at Harvard Medical School, in a press release detailing the study. “We wanted to know if such a generative model could ‘think’ like a doctor, so we asked one to solve standardized complex diagnostic cases used for educational purposes. It did really, really well.”

The research team tasked ChatGPT-4 with evaluating 70 clinicopathological case conferences (CPCs). These CPCs presented a series of complex patient cases and corresponding clinical data, including imaging studies, laboratory results, and histopathological findings.

Using these data, the AI was asked to provide potential diagnoses. The model’s accuracy was measured in terms of how often its top diagnosis matched the final diagnosis in the CPC, and whether the final diagnosis appeared in its differential diagnoses at all.

The researchers found that ChatGPT’s top diagnosis matched the final diagnosis in 39 percent of cases, and the final CPC diagnosis was included in the tool’s differential in 64 percent of cases. This led the researchers to conclude that generative AI may prove useful as an adjunct in clinical decision-making for complex diagnoses.

“While [chatbots] cannot replace the expertise and knowledge of a trained medical professional, generative AI is a promising potential adjunct to human cognition in diagnosis,” explained first author Zahir Kanjee, MD, a hospitalist at BIDMC and assistant professor of medicine at Harvard Medical School. “It has the potential to help physicians make sense of complex medical data and broaden or refine our diagnostic thinking. We need more research on the optimal uses, benefits and limits of this technology, and a lot of privacy issues need sorting out, but these are exciting findings for the future of diagnosis and patient care.”

The research team also noted that their study had multiple limitations, including some subjectivity in the outcome measure that the researchers attempted to mitigate using a standard chat prompt for each CPC.

“Our study adds to a growing body of literature demonstrating the promising capabilities of AI technology,” said co-author Byron Crowe, MD, an internal medicine physician at BIDMC and an instructor in medicine at Harvard Medical School. “Further investigation will help us better understand how these new AI models might transform health care delivery.”