Transcript Medication Information Extraction - University of Arizona School of

Medication Information Extraction
-------General review of the third i2b2 Workshop on Natural Language
Processing Challenges for Clinical Records
Dongfang Xu
School of Information
Outline
• I2b2 Workshop Introduction
• Medication information task
– Overview of Medication Challenge
– Data and Materials
– Systems
• Evaluation and Analysis
– Methods
– Results and Discussion
– Conclusion
I2B2 Workshop
A workshop to Enhance the NLP tools to acquire fine grained
information from clinical records.
• Released datasets regularly
• Call for participants
– De-identification challenge, Smoking challenge,
Obesity Challenge, Medication Challenge, Relations
Challenge, Heart Disease risks Challenge
• 2016 challenge:
De-identificationa over ~1000
psychiatric evaluation records; RDoC classification: determine
symptom severity in a domain for a patient; non-specific tasks
related with mental health.
See: https://www.i2b2.org/index.html
Medication Task
Extract the following information(called field) on
Medication experienced by the patient from
discharge summary:
– Medications (m): names, brand names, generics, and collective names of
prescription substances, over the counter medications, and other biological
substances
– Dosages (do): indicating the amount of a medication
– Modes (mo): indicating the route for administering the medication
– Frequencies (f): indicating how often each dose of the medication should be
taken.
– Durations (du): indicating how long the medication is to be administered.
– Reasons (r): stating the medical reason for which the medication is given.
– List/narrative (ln): indicating whether the medication information appears in a list
structure or in narrative running text in the discharge summary.
Medication Task
Outline
• I2b2 Workshop Introduction
• Medication information task
– Overview of Medication Challenge
– Data and Materials
– Systems
• Evaluation and Analysis
– Methods
– Results and Discussion
– Conclusion
Data & Materials
1243 Discharge Summaries
Training Data
Test Data
Annotated by expert
17
-----
Annotated by community
-----
251
(based on system outputs)
Without annotation
-----
-----
Total
696
547
20 teams participated in this medication challenge.
Outline
• I2b2 Workshop Introduction
• Medication information task
– Overview of Medication Challenge
– Data and Materials
– Systems
• Evaluation and Analysis
– Methods
– Results and Discussion
– Conclusion
Systems
These20 teams were classified along three dimensions:
• External resources (marked as “Yes” or “No”)
System that used proprietary systems, data, and resources that were not available
to other teams; four were declared to have utilized external resources.
• Medical expert involvement(marked as “Yes” or “No”)
Five were declared to have benefitted from medical experts.
• Methods (marked as “rule based”, “supervised” , “hybrid”)
10 were described by their authors as rule-based, four as supervised, and six as
hybrids.
Outline
• I2b2 Workshop Introduction
• Medication information task
– Overview of Medication Challenge
– Data and Materials
– Systems
• Evaluation and Analysis
– Methods
– Results and Discussion
– Conclusion
Methods
• Two sets of evaluation matrics.
Horizontal matrics; Vertical matrics.
• Precision, recall and F1 score at phrase and token level.
Phrase level: Complete text of field values.
Token level: delimited by spaces and punctuation.
Methods
• To ran the significance test on each two system outputs:
Approximate randomization was used for testing significance.
1. Get the difference(f) of the horizontal phrase-level F-measures of two system
outputs A &B.
2. Let j be the number of entries in A, and let k be the number of entries in B, and a
combined outputs C from A and B.
3. For iterations n=1000:
Randomly select j entries without resampling from C as new A*, and let the rest
be B*, recalculate the horizontal phrase-level F-measures for both A* and B*, get
the difference f*, and count how many times there are positive differences
between f* and f, f-f*, named as k.
4. Get the p value , p=k/n
Outline
• I2b2 Workshop Introduction
• Medication information task
– Overview of Medication Challenge
– Data and Materials
– Systems
• Evaluation and Analysis
– Methods
– Results and Discussion
– Conclusion
Systems Introduction
1. These teams applied text filtering to eliminate the content
that was not related to the medications of the patient.
2. Built vocabularies from publicly available knowledge sources,
enriched these vocabularies with examples from the training
data and the annotation guidelines, and bootstrapped
examples from unlabeled i2b2 discharge summaries as well as
the web.
The top 10 teams with best performing submissions
Ra
nk
Group
external
resources,
medical
experts)
Methods
Notes
1
Usyd
N, Y
hybrid
Combined CRFs with SVMs and rules.
2
Vanderbit
Y, Y
Rule-based
MedEx system for tagging, Context free gra
3
Manchester
N, N
Rule-based
4
NLM
N, N
Rule-based
MetaMap for marking reasons
5
BMEHumboldt
N, N
Rule-based
GNU software for RE, Unstructured Infor
Manag Architecture (UIMA) as their base
6
OpenU
N, N
Rule-based
Genia Tagger for pos tagging
7
Uparis
N, N
Rule-based
Ogmios platform for linguisitic stuff
8
LIMSI
N, N
Rule-based
9
UofUtah
N, Y
hybrid
Compiled a knowledge base, Open NLP,
MetaMa, UMLS
10
UWisconsin
N, N
hybrid
CRFs and rule based for Medi, Adabosst for
paring
The top 10 teams with best performing submissions
List/narrative: indicating whether the medication information appears in a list
structure or in narrative running text in the discharge summary.
The top 10 teams with best performing submissions
• University of Wisconsin-Milwaukee’s system is statistically indiscernible
from all but two systems, including one of the top three. See red box.
•In terms of the phrase-level horizontal F-measures, the only systems to
perform significantly differently from all systems that scored below them
came from the University of Sydney and the University of Manchester.
See green and blue box.
Vertical matrices Evaluation on fields
Expert annotated
charge summaries and
community annotated
charge summaries
against final
community ground
truth (gold standard)
using Macro-averaged
F-measure.
From community
annotation experiment
paper.
Vertical matrices Evaluation on fields
Expert annotated
charge summaries
against final
community ground
truth (gold standard)
using Micro-averaged
F-measure.
From community
annotation experiment
paper.
Outline
• I2b2 Workshop Introduction
• Medication information task
– Overview of Medication Challenge
– Data and Materials
– Systems
• Evaluation and Analysis
– Methods
– Results and Discussion
– Conclusion
Conclusion
• The Third i2b2 Workshop on Natural Language
Processing Challenges for Clinical Records attracted
20 international teams and tackled a complex set of
information extraction problems.
– The state-of-the-art NLP systems perform well in
extracting medication names, dosages, modes, and
frequencies.
– Detecting duration and the reason for medication events
remains a challenge.
Reference
• Uzuner, Ö., Solti, I., & Cadag, E. (2010).
Extracting medication information from
clinical text. Journal of the American Medical
Informatics Association,17(5), 514-518.
• Uzuner, Ö., Solti, I., Xia, F., & Cadag, E. (2010).
Community annotation experiment for ground
truth generation for the i2b2 medication
challenge.Journal of the American Medical
Informatics Association, 17(5), 519-523.
Thank you!