- Social media may offer an untapped data resource for healthcare, but natural language processing (NLP) tools still struggle to analyze and extract meaningful insights from this information due to lack of context and ambiguous expressions, according to a study published in JAMIA.
Approximately 70 percent of the US population actively uses social media, the researchers noted, and 37 percent of those users have identified health and medicine as the most interesting topic of discussion on social sites. As a result, investigators are increasingly using social media as a data source for monitoring health trends and opinions.
The surge in social media use has also coincided with the development of NLP and other data analytics tools, the team noted.
“As the volume of health-related data in social media continues to grow, it has become imperative to introduce and evaluate NLP methods that can effectively derive knowledge from it for operational tasks,” the group said.
However, using these tools to generate actionable health insights from social media is still a challenging task. Social media messages can often contain grammatical errors and colloquial language, which makes it difficult to translate these texts into formalized data elements. While some have proposed innovative approaches in this area, the research team stated that there is still significant progress to be made.
To improve data analysis for social media, the researchers executed a community-shared task that would focus on how to progress NLP in the social media domain.
Community-shared tasks are a popular approach to enhance NLP methods on specialized tasks and have proven effective in providing clear benchmarks in evolving areas of healthcare. The group used the Social Media Mining for Health (SMM4H) shared tasks to concentrate on text classification and normalization of health-related Twitter posts.
The researchers organized three independent subtasks, including adverse drug reactions (ADRs) as subtask 1, medication consumption as subtask 2, and ADR expressions as subtask 3.
The text classification tasks involved the categorization of tweets mentioning ADRs and medication consumption, while the normalization task required systems to map ADR expressions to standard IDs.
The research group accepted 55 system runs from 13 teams for evaluation. Researchers found that the best system scores for the three subtasks were 0.435 for subtask 1, 0.693 for subtask 2, and 88.5 percent accuracy for subtask 3.
For subtasks 1 and 2, the most common reason for false negatives was the use of infrequent or creative expressions. False positives were often caused by systems mistaking ADRs for related concepts and terms. Lack of context in length-limited posts also presented significant challenges for NLP systems.
For subtask 3, NLP systems also had frequent false positives due to related concepts and terms. Lack of training data for rarely occurring concepts and terms was also a major cause of error for subtask 3, as well as lack of context.
The results show that further work will be necessary before NLP tools can generate significant insights from social media messages.
“Data imbalance and lack of context remain challenges for natural language processing of social media text. We will use the lessons learned to design future shared tasks, such as the inclusion of more contextual information along with the essential texts,” the team concluded.