Deep Learning System May Provide Automatic Surgical Skill Assessment

June 13, 2023 - Researchers have developed a deep learning (DL) model capable of recognizing standardized surgical fields and performing automatic surgical skill assessments using intraoperative videos of laparoscopic colorectal procedures, according to a recent study published in JAMA Surgery.

Surgical skill assessment is a key component of improving patient outcomes, but such assessments typically rely on manual, video-based reviews of surgeon performance. This can lead to an increased burden for human reviewers and lead to issues related to the objectivity of that reviewer, the researchers explained.

Artificial intelligence (AI) has been suggested as a potential solution to these problems, as these technologies present opportunities for advanced video analysis capabilities and increased objectivity. However, the research team points out that an AI tool for surgical skill assessment would need to be consistent with skill qualification metrics in the surgical field within which it is being used.

To address these challenges in the field of endoscopy, the researchers developed a DL model designed to recognize the standardized surgical fields in laparoscopic sigmoid colon resection and subsequently evaluate the tool’s ability to assess surgical skill.

The research team developed the model using intraoperative videos of laparoscopic colorectal procedures submitted to the Japan Society for Endoscopic Surgery between August 2016 and November 2017.

From these, videos in which surgeries were performed by surgeons with Endoscopic Surgical Skill Qualification System (ESSQS) scores higher than 75 were selected to construct and train the model. This sample was comprised of 650 videos total.

Of these, 60 were used for model development, and 60 were used for training. The remainder were used for model validation.

The model was trained to recognize a standardized surgical field, then tasked with generating an AI confidence score (AICS), which represents how similar the model’s output is to standardized surgical field development parameters.

Correlations between AICS and ESSQS were analyzed, in addition to the model’s performance in terms of AICS across score groups.

The researchers found that the Spearman rank correlation coefficient, a measure of the strength and direction of a relationship between two variables, was 0.81 between the AICS and ESSQS scores.

Additionally, the area under the receiver operating characteristic was 0.93 for the low-score group and 0.94 for the high-score group.

The strong correlation between the AICS and the ESQSS led the research team to conclude that the model may be feasible as a method of automatic skill assessment and could also potentially be used to create an automated screening system for surgical skills.

With the rise of AI in healthcare, researchers and clinicians have increasingly been looking for ways to leverage the technology to improve surgical care. Recent examples of this have included using AI and machine learning to better understand surgical care needs, leveraging clinical intelligence to bolster perioperative care, and utilizing AI to improve operating room scheduling.

However, video-based analytics of surgeon performance may provide an additional, more immediate impact on improving surgical outcomes.

Using AI for clinician skill benchmarking isn’t new, but being able to evaluate surgeon performance in near real-time is a growing possibility.

In April, researchers from the California Institute of Technology (Caltech) and the University of Southern California unveiled an AI system designed to provide surgeons with feedback on the quality of their work and indicate where their skills may need to improve.

The tool uses videos of surgeries to identify procedure type, then evaluates a surgeon’s performance by assessing discrete motions, such as holding a needle or inserting and withdrawing that needle from tissue.

The tool is also designed to justify its own assessments using video clips from the procedure to help give surgeons objective, actionable feedback.