Researchers Develop AI Feedback Tool to Improve Surgeon Performance

April 27, 2023 - Researchers from the California Institute of Technology (Caltech) and the University of Southern California (USC) Keck School of Medicine have developed an artificial intelligence (AI) system designed to provide surgeons with feedback on the quality of their work and which of their skills need improvement. 

The system, Surgical AI System (SAIS), was introduced recently in a series of articles published concurrently in Nature Biomedical Engineering, npj Digital Medicine, and Communications Medicine. SAIS uses videos of surgical procedures to identify what type of surgery is taking place and the quality with which it was performed by the surgeon, which the researchers posited may help surgeons hone their skills. 

"In high stakes environments such as robotic surgery, it is not realistic for AI to replace human surgeons in the short term," said Anima Anandkumar, PhD, Bren Professor of Computing and Mathematical Sciences at Caltech and senior author of the studies, in the press release. "Instead, we asked how AI can safely improve surgical outcomes for the patients, and hence, our focus on making human surgeons better and more effective through AI." 

To achieve this goal, the researchers trained SAIS using large volumes of surgery videos and associated data, annotated by medical professionals. Using this information, SAIS assesses surgeon performance by evaluating individual discrete motions, like holding a needle, driving it through tissue, and withdrawing it from that tissue. 

Following training, the tool was validated using video data from multiple hospitals and procedure types. The researchers also designed SAIS to be able to justify its skill assessment, similar to how an experienced surgeon might if they were mentoring a newer surgeon, but without some of the challenges that come with surgical mentorship. 

"Human-derived surgical feedback is not presently objective nor scalable," stated Andrew Hung, MD, a urologist with Keck Medicine of USC and associate professor of urology at Keck School of Medicine of USC. "AI-derived feedback, such as what our system delivers, presents a major opportunity to provide surgeons actionable feedback." 

By informing surgeons about their level of skill and providing feedback on its rationale by referencing specific video clips, the research team indicate that SAIS’ recommendations are consistent, accurate, and scalable. 

"We were able to show that such AI-based explanations often align with explanations that surgeons would have otherwise provided," explained Dani Kiyasseh, PhD, lead author of the studies, a former postdoctoral researcher at Caltech and a senior AI engineer at Vicarious Surgical. "Reliable AI-based explanations can pave the way for providing feedback when peer surgeons are not immediately available." 

However, more work is needed to improve SAIS before it can be used by clinicians.  

According to the press release, the system exhibited unintended bias early on in testing, in which the tool would sometimes rate surgeons as more or less skilled than their experience actually indicated based only on an analysis of the surgeons’ overall movements. 

The researchers addressed this by guiding SAIS to narrow its focus to only the pertinent parts of the video. This reduced the AI’s bias but did not fully eliminate it.  

The research team is currently working to address the remaining bias in the tool, the press release states. 

These efforts reflect a larger interest across the healthcare sector in using AI to improve surgical care. 

Earlier this month, the Advocate Aurora Research Institute shared that it will leverage surgical data analytics company KelaHealth’s Surgical Intelligence Platform, with the aim of using AI and machine learning (ML) to determine the efficacy of robotic surgical techniques and enhance surgical care delivery.