- UC Davis Health Launches AI Tool to Curb Documentation Burden
The researchers used four modalities to document common orthopedic encounters in hand surgery patient visits: an AI-based virtual scribe service run on a tablet and tasked with extracting everything said in the room, a medical scribe either physically in the office visit or participating virtually to transcribe the patient encounter, a transcription service in which the clinician records an audio file about the patient visit and sends it to a third-party company for transcription, and a voice recognition mobile (VRM) application, an EMR-integrated tool that types the words being said based on voice recognition.
During the study, three orthopedic hand surgeons evaluated 10 standardized 'patients' with prewritten clinical encounters or 'vignettes.' Clinical documentation during the encounters was initially performed using the AI-based scribe and the medical scribe and, afterward, with the VRM and transcription service.
"Our physicians who were not involved in the documentation acted out these vignettes and each scenario contained an element of distraction to determine if the AI would be thrown off by various nuances that might occur during a clinical visit — such as a parent and a minor sharing their thoughts, or a patient interjecting a story about a friend's experience with hand surgery in the middle of providing an update on their own surgery," Rivlin explained.
Using this study design, the researchers documented 118 clinical encounters, including 30 with the AI scribe, 30 with the VRM, 28 with the transcription service, and 30 with the medical scribe. Any clinical notes generated during each encounter were categorized as acceptable or unacceptable and given a letter grade of A, B, C, or F based on an eight-point scoring system. Following this process, an attorney reviewed all notes to assess medical legal risk.
The research team found that all modalities achieved similar levels of high performance but found key differences between them.
A significant limitation of the AI modality is that it scored lower than the others on the question “is the plan correct?” The AI was able to gather most of the implied and verbal elements within its documentation, but its performance was deficient compared to the human medical scribe and required a manual edit of the ‘plan’ section of the documentation.
The AI-based scribe’s need for additional correction and verification is a major drawback to its use in the clinical setting, but it has significant potential, the researchers noted.
"The AI-based virtual scribe service is a promising tool to help decrease documentation burden without significantly lowering the quality of documentation compared to transcription and voice recognition software services," said Rivlin. "While AI has some limitations, it continues to improve as the technology advances. These results create a palette of options for physicians to compare outputs should they want to explore new modalities."
By 