Clinical Relation Extraction Toward Drug Safety Surveillance Using Electronic Health Record Narratives: Classical Learning Versus Deep Learning
UMass Chan AffiliationsDepartment of Quantitative Health Sciences
Document TypeJournal Article
Keywordsdrug-related side effects and adverse reactions
electronic health records
medical informatics applications
natural language processing
Artificial Intelligence and Robotics
Health Information Technology
Health Services Administration
Health Services Research
MetadataShow full item record
AbstractBACKGROUND: Medication and adverse drug event (ADE) information extracted from electronic health record (EHR) notes can be a rich resource for drug safety surveillance. Existing observational studies have mainly relied on structured EHR data to obtain ADE information; however, ADEs are often buried in the EHR narratives and not recorded in structured data. OBJECTIVE: To unlock ADE-related information from EHR narratives, there is a need to extract relevant entities and identify relations among them. In this study, we focus on relation identification. This study aimed to evaluate natural language processing and machine learning approaches using the expert-annotated medical entities and relations in the context of drug safety surveillance, and investigate how different learning approaches perform under different configurations. METHODS: We have manually annotated 791 EHR notes with 9 named entities (eg, medication, indication, severity, and ADEs) and 7 different types of relations (eg, medication-dosage, medication-ADE, and severity-ADE). Then, we explored 3 supervised machine learning systems for relation identification: (1) a support vector machines (SVM) system, (2) an end-to-end deep neural network system, and (3) a supervised descriptive rule induction baseline system. For the neural network system, we exploited the state-of-the-art recurrent neural network (RNN) and attention models. We report the performance by macro-averaged precision, recall, and F1-score across the relation types. RESULTS: Our results show that the SVM model achieved the best average F1-score of 89.1% on test data, outperforming the long short-term memory (LSTM) model with attention (F1-score of 65.72%) as well as the rule induction baseline system (F1-score of 7.47%) by a large margin. The bidirectional LSTM model with attention achieved the best performance among different RNN models. With the inclusion of additional features in the LSTM model, its performance can be boosted to an average F1-score of 77.35%. CONCLUSIONS: It shows that classical learning models (SVM) remains advantageous over deep learning models (RNN variants) for clinical relation identification, especially for long-distance intersentential relations. However, RNNs demonstrate a great potential of significant improvement if more training data become available. Our work is an important step toward mining EHRs to improve the efficacy of drug safety surveillance. Most importantly, the annotated data used in this study will be made publicly available, which will further promote drug safety research in the community.
JMIR Public Health Surveill. 2018 Apr 25;4(2):e29. doi: 10.2196/publichealth.9361. Link to article on publisher's site
Permanent Link to this Itemhttp://hdl.handle.net/20.500.14038/40635
RightsCopyright ©Tsendsuren Munkhdalai, Feifan Liu, Hong Yu. Originally published in JMIR Public Health and Surveillance (http://publichealth.jmir.org), 25.04.2018. This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Public Health and Surveillance, is properly cited. The complete bibliographic information, a link to the original publication on http://publichealth.jmir.org, as well as this copyright and license information must be included.
Except where otherwise noted, this item's license is described as Copyright ©Tsendsuren Munkhdalai, Feifan Liu, Hong Yu. Originally published in JMIR Public Health and Surveillance (http://publichealth.jmir.org), 25.04.2018. This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Public Health and Surveillance, is properly cited. The complete bibliographic information, a link to the original publication on http://publichealth.jmir.org, as well as this copyright and license information must be included.
Showing items related by title, author, creator and subject.
Policy Brief: Addressing Social Determinants of Health through Community Health Workers: A Call to ActionLondon, Katharine; Damio, Grace; Ferrazo, Meredith; Perez-Escamalla, Rafael; Wiggins, Noelle (2018-01-30)This technical report was compiled by the Hispanic Health Council in partnership with Southwestern AHEC and a panel of Community Health Worker Policy Research Experts which included our Katharine London from the Center for Health Law and Economics. The report offers a number of policy recommendations for community health workers for communities that might benefit from community-based services. The report offers recommendations on; payment of community health workers; community health worker caseloads; community health worker recruitment; community health worker training; reflective and trauma-informed mentoring and supportive supervision of community health workers; integration of community health workers into care teams; documenting the effect of community heal worker services on social determination of health. The Hispanic Health Council believes a service design that effectively supports community health workers would incorporate the seven areas of policy recommendation included in this report.
A Public Health Framework for the State Mental Health Authority: A Call for Action by Massachusetts Consumers and Family MembersDelman, Jonathan (2006-01-01)During the Spring of 2006, Consumer Quality Initiatives (CQI) conducted 20 focus groups across the state, 12 with adults with mental illness, 3 with parents of youth with serious emotional disorder, 2 with youth with SED, 1 with family members of adult consumers, and 2 with youth in transition. Supported by a contract with Massachusetts Department of Mental Health (DMH), the goal was to assist DMH in framing the criteria for its upcoming reprocurement. Our findings reveal a frustration with an approach to health care delivery that focuses primarily on the provision of psychiatric care (egs, medication, therapy, hospitalization). We reviewed the focus group reports to identify the most significant themes, which clustered within eight broad categories.
Making the Case for Sustainable Funding for Community Health Worker Services: Talking to Payers and ProvidersLondon, Katharine (2018-01-27)In this presentation, Katharine London of the Center for Health Law and Economics makes her case for offering sustainable funding for community health worker services. Research has shown community health workers can have a distinct impact on health systems, helping them improve population health and contain costs, while also promoting health equity and community engagement. This presentation was designed to assist CHWs and other advocates in engaging with policymakers and payers to support CHW sustainability and develop a financial plan for their CHW work. It was presented as part of a CHW Sustainability event held at the Families USA’s annual conference, Health Action 2018: Staying Strong for America’s Families, in Washington, DC. See Katharine London's blog post on payment delivery methods for community health workers here.