Transcript A Methodology Specification of Clinical Guidelines

An Evaluation of a
Methodology for Specification
of Clinical Guidelines at
Multiple Representation Levels
Student :Erez Shalom
Supervisors:
Prof. Yuval Shahar
Dr. Meirav Taieb-Maymon
ISE Dep. Seminar 26/4/06
Talk Roadmap :
 Background
 Methods
 Results
 Conclusions
 Future
Directions
ISE Dep. Seminar 26/4/06
Clinical Guidelines
 Textual
documents describing “state of
the art” patient management
 A powerful method to standardize and
improve the quality of medical care
 Usually specify diagnostic and/or
therapeutic procedures
ISE Dep. Seminar 26/4/06
Subsections
Describing
patient
diagnosis and
treatment
ISE Dep. Seminar 26/4/06
The Need for Automation of Clinical
Guidelines
 Automatic support provides:
 Visual specification
 Search and retrieval
 Application of a GL
 Retrospective quality assurance
 However: Most GLs are text based and electronic
inaccessible at the point of care
ISE Dep. Seminar 26/4/06
The Required Infrastructure


A machine-comprehensible GL representation
ontology (e.g., Asbru ontology)
Runtime GL application and QA tools


A preliminary engine, namely , Spock was already
developed in our lab by [Young,2005]
Support for a gradual structuring of the GL
(from text to an executable code)
ISE Dep. Seminar 26/4/06
The Structuring Process
The Guideline as a text document
The Guideline as a tree of plans
Regimen A
P
In parallel
Cefotetan
D P
Doxycline
D P
Involves 2 main types of knowledge:
Procedural knowledge – e.g. Regimen A for administer the two
medications in parallel
Declarative knowledge - e.g. 2 g IV
ISE Dep. Seminar 26/4/06
Sample GL Modeling Methods
Method
Knowledge Acquisition tool
EON and SAGE, Prodigy,
GLIF
A Protégé-based interface
PROforma
Arrezo
GEM
GEM-Cutter
GLARE
"CG_AM" graphical interface
GUIDE
NEWGUIDE
Asbru
Asbru-View, GMT, Stepper
ISE Dep. Seminar 26/4/06
The Hybrid Representation Model
Gradually structuring the GL using increasingly formal representation levels
Expert Physician Knowledge Engineer
Expert
Physician
Collaboration
Knowledge
Engineer
DeGeL KB
Semantic
markup
Adding control
structure
Formalizing to
executable code
//Check HGB
If(HGB > 12)
{...
}
Free text
Guidelines
Semi- Structured
Level
Semi- Formal
Level
//Check HGB
If(HGB > 12)
{...
}
Formal
Level
 Implemented as part of the Digital Electronic Guideline
Library )DeGeL)
Used within the URUZ GL markup tool
ISE Dep. Seminar 26/4/06
Asbru- the Underlying GuidelineRepresentation Ontology
Includes semantic Knowledge Roles (KRs) organized in KR-Classes such as:




Conditions KR-Class (e.g., the filter condition,
and the abort condition)
Plan-body KR-Class for the GL’s Control
structures (e.g., sequential, concurrent, and
repeating combinations of actions or subguidelines),
GL’s Goals KR-Class (e.g. process and outcome
intentions),
Context KR-Class of the activities in the GL (e.g.
actors, clinical-context).
ISE Dep. Seminar 26/4/06
URUZ (I):
Specification of declarative knowledge
Expert physician
Selects “filter
condition”
knowledge role
ISE Dep. Seminar 26/4/06
URUZ (cont’d):
Specification of Procedural Knowledge
Expert physician decomposing the GL
into tree of plans
ISE Dep. Seminar 26/4/06
GL Specification: Core Issues
√
Expert Physicians (EPs) - Knowledge Engineers
(KEs) collaboration
Incremental Specification
Treatment of Multiple Ontologies
Distributed Collaboration and Sharing
Text Based Source
√
Knowledge Conversion
√
√
√
√
ISE Dep. Seminar 26/4/06
Several unresolved issues:



Definition of the necessary steps for the GL
specification process
Use optimally of EPs and KEs in the process
Evaluation is crucial for quantify the markups quality
To Achieve high quality of markups there is a Need for:
• An overall process of guideline specification
• A complete evaluation methodology
ISE Dep. Seminar 26/4/06
Talk Roadmap
Background
 Methods
 Results
 Conclusions
 Future Directions
√
ISE Dep. Seminar 26/4/06
The Overall Process of Guideline
Specification
The activities in the markup process include
three main phases :
1) Preparations before the markup activities
2) Actions during the Markup activities
3) After Markup activity
ISE Dep. Seminar 26/4/06
A Methodology Specification of
Clinical Guidelines
During Markup
Before Markup
KE
KE
KE
EP
1.
2.
KE Choose
Specification
Language
KEs
Instructing
the EPs
KE
EP
3.
Choosing
together GL
for
Specification
KE
EP
4.
Creating
together an
OntologySpecific
Consensus
KE
KE
EP
5.
EPs Training
in the
Markup Tool
After Markup
KE
EP
EP
6.
Creating
together
Gold
Standard
Markup
7.
EPs Classify
the GL
EP
8.
EPs Perform
Markup
EP
9.
Evaluating
together
the Markups
Creating a consensus is a crucial, mandatory step before markup
ISE Dep. Seminar 26/4/06
The Importance of Using an
Ontology-Specific Consensus (OSC)



An OSC is a structural document that
describes schematically the clinical
directives of the GL
Described by the semantic of the
specification ontology
Prevent disagreement and a great deal of
variability among the EPs
ISE Dep. Seminar 26/4/06
Methodology for Creation of OSC

The OSC is created in a iterative fashion by
performing the following steps
1. First, we create a preliminary structure of the clinical
pathway
2. The KE adds procedural, control structure
3. The KE adds declarative concepts for each defined
step
4. After some iteration of steps 2 and 3, an OSC is
formed
ISE Dep. Seminar 26/4/06
The second stage in forming a
consensus
Patient Treatment and evaluation
NO
is PID
severe? (*)
Hospitalization and
discharged
(sequential)
Outpatient Treatment and
evaluation (parallel)
Outpatient
Treatment
Outpatient Evaluation
and follow up
YES
See 1.7
See 1.8
Hospitalization
Post Discharged
See 1.2
See 1.5
ISE Dep. Seminar 26/4/06
Evaluation Design
Considered some specific Criteria :
Amount of Expertise
 The acquired knowledge domain
 The Ontology Specific Consensuses
 The Gold Standard markup for each GL
 The Markups for each GL
 The Evaluation of markups

ISE Dep. Seminar 26/4/06
Evaluation Design (cont’d)
 Three
GLs in different domains were
used :
Pelvic
Inflammatory Disease (PID)
Chronic Obstructive Pulmonary
Disease (COPD)
 Hypothyroidism(HypoThyrd)
ISE Dep. Seminar 26/4/06
Evaluation Design (cont’d)
KE
EP
EP
EP
KE
EP
EP
Hypo
Thyrd
COPD
PID
Hypo
Thyrd
COPD
PID
Hypo
Thyrd
COPD
PID
Hypo
Thyrd
COPD
Markup
GL
Sources
One Ontology
Specific Consensus
For each GL
COPD
PID
Markup
Markup
Hypo
Thyrd
PID
PIDGold
Markup
Standard
Gold
Standard
Markup
Markup
Two markups
for each GL
Each markup is
evaluated compare to
the Gold Standard
EP
Gold
Standard
One Gold Standard
Markup for each GL
ISE Dep. Seminar 26/4/06
Research Questions
Is markup feasible by EPs?
 Is there a difference between the EPs
editing the same GLs , and same EPs
editing difference GLs?
 Is there a difference between the KRs
across all EPs?
 Is there a difference in the amount of
errors when using different OSC?

ISE Dep. Seminar 26/4/06
Evaluation of markups
Subjective Measures Questionnaires were administered for finding the EPs
attitude regarding the specification process
Objective Measures – in two scales
(compared to the GS):
* Completeness of the markup
* Correctness of the markup
ISE Dep. Seminar 26/4/06
The Objective Measures - Completeness
GS
A
Markup
B
C
ISE Dep. Seminar 26/4/06
The Objective Measures - Correctness
* Clinical Measure (CM) – measure the clinical correctness
of the content
* Asbru Semantic Measure (ASM) - measure the semantics
correctness of the content ( Asbru semantic in our case)
ISE Dep. Seminar 26/4/06
Resolution of Measure
Mean (weighted) Quality Score (MQS) for:
 GLs - to find common trends in a GL, and in all GLs
 EPs - to find trends in between the markups of the
EPs across the same GL and between GLs
 KRs - to find trends in a specific KR type and
common trends across KRs and KR classes
across one markup, GL and in all GLs
ISE Dep. Seminar 26/4/06
The Objective Measures – Types of
Errors

General errors classified into two types, and
thus into two corresponding scales:

Clinical errors:




Clinical content not accurate
Clinical semantics not well specified
Clinical content not complete.
Asbru semantics errors:




Asbru semantics content not accurate
Asbru semantics content not well specified
The content does not includes mappings to standard terms
The necessary knowledge is not defined in the guideline
knowledge when it should be.
ISE Dep. Seminar 26/4/06
The Objective Measures – Types of
Errors

Specific errors for each KR Type a specific
error, for example :

Conditions /Intentions KRs:


Simple Action Plan-Body Type:



There are no And/Or operators between the different criteria.
Has no text content describing the plan
Has no single atomic action semantics with clear specification
and description for the action to be performed.
Plan Activation Plan-Body Type:


Plan name is not defined
Defined plan does not exist in DeGeL.
ISE Dep. Seminar 26/4/06
The Markup-Evaluation Tool
ISE Dep. Seminar 26/4/06
The Markup-Evaluation Tool (Cont’d)
ISE Dep. Seminar 26/4/06
The Markup-Evaluation Tool (Cont’d)
ISE Dep. Seminar 26/4/06
Talk Roadmap
Background
√ Methods
 Results
 Conclusions
 Future Directions
√
ISE Dep. Seminar 26/4/06
Results – Subjective Measures
Purpose
Stage
Result
1
Finding the Aspects that
most helped the EPs'
when creating an OSC
after creating
the OSC
Using their medial knowledge
and their understanding of the
specification ontology (Asbru, in
our case) Vs. specification Tools
2
Finding the Aspects that
most helped the EPs'
making a markup
after markup
Specification Tools is considered
as more helpful
3
Finding how well The
EPs Understand Asbru
KRs
before markup
Declarative KRs are more easy to
understand (such as filter
condition)
4
Finding what were the
difficulties of the EPs' in
structuring the Asbru
KRs
after markup
Procedural KRs are more easy to
structure (such periodic plan)
5
System Usability Scale
(SUS) for URUZ
after markup
SUS=47 ; Not Usable!
• Non significant correlation between results 1 and 2
• Significant correlation between results 3 and 4
ISE Dep. Seminar 26/4/06
Results – Objective measures
Number of specified plans:
Measures Summary:
• Mean Completeness for all markups of EPs of 91%
•All markups of EPs has significant (P<0.05) proportion of
scores of 1 higher than 0.33 (some even higher then 0.75)
Markup is feasible by EPs
ISE Dep. Seminar 26/4/06
Results –Difference between EPs
Issue
1
Difference between the proportions of
completeness measure between EPs editing
the same GLs
2
Difference in correctness measure between
EPs editing the same GLs in most GLs
(except the Hypo)
Difference in Correlation measure between
EPs editing the same GLs in most GLs
(except the PID)
3
4
Difference in correctness measure between
different GLs editing the same EPs
significant
(P<0.05)
nonsignificant
(P>0.05)
√
√
√
√
•Any EP can perform markup with high completeness
•There is wide variability between the EPs in the correctness
measure with a range of [0.13,0.58] on a scale of [-1,1]
ISE Dep. Seminar 26/4/06
Results – Difference between KRs
There was significant difference (P<0.05) between
homogenous groups of KRs
EPs has difficulty to structure procedural KRs than
declarative ones
ISE Dep. Seminar 26/4/06
Results – Types of errors
Markup Error Rate(MER) per KR, per EP
199
360
0.6
0.55
112
360
0.5
87
0.31 360
0.24
MER
0.4
0.3
79
194
61
0.41
194
0.31
18
194
0.09
0.2
0.1
12
9
98
3
98
0.12
0.09 98
0.03
0
PID
Total
Ontological
COPD
GLs
Clinical
HypoThyrd
No. of errors in GLi
= MERi.
No. of KRs in GLi * 2
The differences in total between the three GLs were
highly significant in a proportion test (P<0.001)
The more detailed and structured the OSC was, the lower
the total number of errors committed by the EPs for each KR
ISE Dep. Seminar 26/4/06
Talk Roadmap
Background
√ Methods
√ Results
 Conclusions
 Future Directions
√
ISE Dep. Seminar 26/4/06
Four main aspects:
Creation of an Ontology-Specific Consensus
(OSC)
 The essential aspects needed to learn to
support the specification process by EPs
 The medical and computational qualifications
needed for specification
 The characteristics of the KA tool needed for
this kind of specification

ISE Dep. Seminar 26/4/06
Creation of an Ontology-Specific
Consensus (OSC)
Should be made as detailed as possible,
including all relevant procedural and
declarative concepts
 The OSC is independent of the
specification tool
 Saving the OSCs in an appropriate digital
library for re-using and sharing

ISE Dep. Seminar 26/4/06
The Essential Aspects Needed to
Learn to Support the Specification
Process by EPs




creating an OSC and performing the markups are two
different tasks which require teaching two different
aspects
Teaching the “difficult” KRs in particular, the procedural
KRs
Short test should be administered before the EPs
perform markups
A help manual and a small simulation of marking up a
GL should be included in the teaching session
ISE Dep. Seminar 26/4/06
The Medical and Computational
Qualifications Needed for Specification



Senior EPs and KEs together should work on the
tasks of selecting a GL for specification and making
the OSC
Any EP (senior, non-senior or a general physician)
can structure the GL's knowledge in a semiformal
representation completely
To specify it correctly, a more available EP should
be selected, perhaps from among residents, interns
or even students
ISE Dep. Seminar 26/4/06
The Characteristics of the KA Tool
Needed for This Kind of Specification



A robust, graphical, highly usable framework is
needed
More intuitive, graphic, user friendly interfaces
should be used for acquiring the “difficult” KRs ,
especially the procedural ones
Need to bridge the gap between the initial
structuring of the EP and the full semantics of
the specification language
GESHER - A Graphical Framework for Specification of Clinical
Guidelines at Multiple Representation Levels
ISE Dep. Seminar 26/4/06
Limitations and Advantages of
the research


Small of the number of EPs and GLs, But, in
fact, 196 sub-plans and 326 KRs in total were
structured by all of the EPs together in all
markups
Lack of careful measurement of the required
time , but, obtain more realistic results, since
the interaction with most of the EPs took place in
their own "playground"
ISE Dep. Seminar 26/4/06
Talk Roadmap
Background
√ Methods
√ Results
√ Conclusions
 Future Directions
√
ISE Dep. Seminar 26/4/06
The GESHER’s Main Features
 User friendly graphical client application
 Support specification at multiple
representation levels
 Support to multiple specification languages
(GL ontologies)
 Access centralized resources such as
DeGeL and a knowledge base
ISE Dep. Seminar 26/4/06
GESHER: Semi-Structured Level
The Hybrid
Ontology Tree
showing KRs
at all
representation
levels
ISE Dep. Seminar 26/4/06
GESHER(II) :Semi-Formal Widgets
ISE Dep. Seminar 26/4/06
Talk Roadmap
√
√
√
√
√
Background
Methods
Results
Conclusions
Future Directions
ISE Dep. Seminar 26/4/06
Summary
 The need for gradual GL specification
 Making an ontology-specific consensus as first step
Use a well defined methodology for the overall process
 Markup is feasible by EPs
 Any EP can perform markup with high completeness
 We should use methodology for increase quality of markups
 Use GESHER as the new framework for specification
 Ongoing new research is being conducted (Pre-Eclampsia GL)
based on this research results
ISE Dep. Seminar 26/4/06
Research Publications






2005 ‫ הכנס הישראלי למ"מ רפואיות‬,‫סביבת עבודה גראפית במס' רמות ייצוג‬
Shalom, E. and Shahar Y. (2005). A Graphical Framework for Specification
of Clinical Guidelines at Multiple Representation Levels. AMIA Annual Fall
Symposium, Washington DC, USA
2006 ‫האם רופאים מסוגלים להבנות ידע רפואי? הכנס הישראלי למ"מ רפואיות‬
Shalom E, Shahar Y, Young O, Bar G, Taieb-Maimon M, Yarkoni A,
B.Martins S, Vaszar L, K.Goldstein M, Liel Y, Leibowitz A, Marom T, and
Lunenfeld E. (2006) A Methodology for Evaluation of A Markup-Based
Specification of Clinical Guidelines Submitted to AMIA , Washington DC,
USA
Shalom E, Shahar Y, Young O, Bar G, Taieb-Maimon M, Yarkoni A,
B.Martins S, Vaszar L, K.Goldstein M, Liel Y, Leibowitz A, Marom T, and
Lunenfeld E.(2006) The Importance of Creating an Ontology-Specific
Consensus Before a Markup-Based Specification of Clinical Guidelines ,
Submitted to ECAI06 ,Tronto, Italy
JAMIA journal paper is in preparation
ISE Dep. Seminar 26/4/06
Acknowledgments







Prof. Yuval Shahar and Dr. Meirav Taib-Maymon
All Medical informatics research center members
Our colleagues at Soroka’s university medical center :
Prof. Eitan Lunenfeld, Dr. Avi Yarkoni, Dr. Guy Bar,
Prof. Yair Liel and Dr. Tal Marom
Our colleagues at Stanford and VA hospital: Drs. Mary
Goldstein, Susana Martins, Lawrence Basso, Herbert
Kaizer, Laszlo Tudor Vaszar
NLM award No LM-06806
Contact info : [email protected]
Visit our web site : http://medinfo.ise.bgu.ac.il/medlab/
ISE Dep. Seminar 26/4/06
Questions?
ISE Dep. Seminar 26/4/06
Methodology for creating an
Ontology-Specific Consensus
EPs create a Clinical consensus :
Give Doxycycline 100 mg orally or IV every 12 hours Plus
Metronidazole 500 mg IV every 8 hours
EPs and KE adds procedural knowledge :
Order :parallel
Doxycycline
Metronidazole
EP+KE add
declarative
knowledge :
Ontology- specific
consensus
Filter condition for drug :
Is patient not sensitive to Doxycycline and
the drug available?
ISE Dep. Seminar 26/4/06
Textual Source
Parenteral Regimen A
Cefotetan 2 g IV every 12 hours
OR
Cefoxitin 2 g IV every 6 hours
PLUS
Doxycycline 100 mg orally or IV every 12 hours.
Parenteral Regimen B
Clindamycin 900 mg IV every 8 hours
PLUS
Gentamicin loading dose IV or IM (2 mg/kg of body weight) followed by a
maintenance dose (1.5 mg/kg) every 8 hours. Single daily dosing may be
substituted.
Alternative Parenteral Regimens
Ofloxacin 400 mg IV every 12 hours
OR
Levofloxacin 500 mg IV once daily
WITH or WITHOUT
Metronidazole 500 mg IV every 8 hours
OR
Ampicillin/Sulbactam 3 g IV every 6 hours
PLUS
Doxycycline 100 mg orally or IV every 12 hours.
ISE Dep. Seminar 26/4/06
The first stage in forming a
consensus
Inpatient treatment Give one of the following regimens
Parenteral Regimen A
Parenteral Regimen B
Alternative Parenteral Regimens
Cefotetan 2 g IV every 12
hours
OR
Cefoxitin 2 g IV every 6
hours
PLUS
Doxycycline 100 mg orally
or IV every 12 hours.
Clindamycin 900 mg IV every
8 hours
PLUS
Gentamicin loading dose IV or
IM (2 mg/kg of body weight)
followed by a maintenance
dose (1.5 mg/kg) every 8 hours.
Single daily dosing may be
substituted.
Ofloxacin 400 mg IV every 12
hours
OR
Levofloxacin 500 mg IV once
daily
WITH or WITHOUT
Metronidazole 500 mg IV every
8 hours
OR
Ampicillin/Sulbactam 3 g IV
every 6 hours
PLUS
Doxycycline 100 mg orally or IV
every 12 hours.
ISE Dep. Seminar 26/4/06
The second stage in forming a
consensus
Inpatient treatment (sequential)
note :give any one of the regimens
OR
OR
OR
Parenteral Regimen A
(parallel)
Cefotetan or Cefoxitin
Cefotetan 2 g IV every
12 hours
OR
Cefoxitin 2 g IV every
6 hours
PLUS
Doxycycline 100 mg orally
or IV every 12 hours.
Parenteral Regimen B (parallel)
Clindamycin 900 mg IV every 8
hours
PLUS
Gentamicin loading dose IV or
IM (2 mg/kg of body weight)
followed by a maintenance dose
(1.5 mg/kg) every 8 hours. Single
daily dosing may be substituted.
Alternative Parenteral Regimens
(parallel)
(Ofloxacin or Levofloxacin) +/(Metronidazole or Ampicillin/
Sulbactam)
Ofloxacin 400 mg IV every 12
hours
OR
Levofloxacin 500 mg IV once daily
WITH or WITHOUT
Metronidazole 500 mg IV every 8
hours
OR
Ampicillin/Sulbactam 3 g IV
every 6 hours
PLUS
Doxycycline 100 mg orally or IV every 12
hours.
ISE Dep. Seminar 26/4/06
The third stage in forming a
consensus
ISE Dep. Seminar 26/4/06
Results – Subjective measures
ISE Dep. Seminar 26/4/06
ISE Dep. Seminar 26/4/06
ISE Dep. Seminar 26/4/06
Results – Objective measures
• Mean Completeness for all markups of EPs of 91%
•All markups of EPs has significant (P<0.05) proportion of
scores of 1 higher than 0.33 (some even higher then 0.75
Markup is feasible by EPs
ISE Dep. Seminar 26/4/06