Machine Learning Models Predict Gastrointestinal Bleeding

May 26, 2021 - Researchers effectively trained machine learning models to predict the risk of gastrointestinal bleeding (GIB) within six to twelve months of a patient being prescribed antithrombotic drugs, according to a recent study published in JAMA Network Open.

The study tested three machine learning models: regularized Cox regression (RegCox), random survival forest (RSF), and extreme gradient boosting (XGBoost). They compared these models to the effectiveness of the current standard risk model, HAS-BLED, or hypertension, abnormal kidney and liver function, stroke, bleeding, labile international normalized ratio, older age, and drug or alcohol use.

Researchers studied more than 300,000 patients over the age of 18 who were prescribed thienopyridine antiplatelet and/or oral anticoagulant agents. All patients had a history of ischemic heart disease, venous thromboembolism, or atrial fibrillation, the study stated, and the data was collected from the OptumLabs Data Warehouse. Patients were separated into a development cohort and a validation cohort.

The machine learning models were assessed using “the area under the receiver operating characteristic curve (AUC), sensitivity, specificity, positive predictive value, and prediction density plots. Relative importance scores were used to identify the variables that were more influential in the top-performing machine learning model,” the study stated.

RegCox was the top performing model, predicting GIB with an AUC of 0.67 at six months and 0.66 at 12 months. XGBoost performed identically in terms of AUCs, but the RSF model was at 0.62 at six months and 0.60 at 12 months. Meanwhile, the widely used HAS-BLED model had an AUC of 0.60 at six months and 0.59 at 12 months.

“The optimal model in this study, RegCox, performed marginally better than not only other machine learning approaches but also a version of HAS-BLED, the most commonly used existing model,” the study explained.

“Given that we modified the HAS-BLED model to use claims-based risk factors, this finding should not be interpreted as a direct comparison with the established clinical model, but the finding does suggest that machine learning approaches can improve on standard approaches if the same data are used.”

The study notes that all three models, along with HAS-BLED, are more effective at pinpointing the patients who will not have any GIB—it is not as effective for determining which patients will experience GIB. Because of this, researchers concluded that all the models mentioned are best applied to patients with a low risk of GIB.

“The moderate AUCs of these models indicate that they should be considered as supplementary to other input for clinical decision-making because they all had a limited ability to discriminate. This study’s findings should be viewed primarily as informing the development of better risk models for GIB,” the study stated.

Researchers concluded that the machine learning risk prediction models did perform well and showed slight improvement from the HAS-BLED model. Although the differences were small, the study shows that machine learning models can in fact be a useful clinical tool for assessing risk of GIB.

In an accompanying article published by JAMA Network Open, Fei Wang, PhD, of Weil Cornell Medical College, writes that the study “demonstrated on a large real-world patient claims data set that ML [machine learning] models can perform better than clinically used risk predictor tools on GIB, which implies the great potential of ML on predicting rare clinical outcomes.”

“This is a good start. Many other factors, including more comprehensive performance evaluation metrics, model interpretability, and data quantity need to be considered for assessing the potential clinical impact of these models. More importantly, efforts on prospective evaluations on clinical ML models with implementation science are critical and urgently needed.”

Predictive machine learning models have shown promise in recent studies, from anticipating pediatric blood clots to detecting deterioration in COVID-19 patients. While these models have their limits, continued research shows that machine learning has the potential to be an asset in creating predictive clinical tools that lead to better health outcomes.