Transcript Use of Online Resources While Using a Clinical Information

Personalized Prediction and Resolution of
Clinician Information Needs
James J. Cimino, M.D.
Departments of Biomedical Informatics and Medicine
Columbia University
Covell et al.
Information Needs
1984
1986
1988
1990
1992
1994
1996
1998
2000
2002
2004
Studying Information Needs
Covell DG, Uman GC, Manning PR. Information
needs in office practice: are they being met? Ann
Intern Med. 1985 Oct;103(4):596-9.
Findings of Observational Studies
• Information needs occur often
• They are often unresolved
• Computer-based resources are underused:
– Lack of knowledge of existence
– Lack of access
– Lack of navigational skills
– Perceived lack of time
Information Needs in Clinical Care
?
Clinical Information for Decision Support
Health Knowledge for Decision Support
Automated Clinical Decision Support
Alerting and
Reminder
System
Clinical Decision Support Systems
Modes of Decision Support
Decision
Support
Mode
User
Recognizes
Need?
Patient Info
Yes
Knowledge
Yes
Alerting
?
Retrieval
Uses Clinical
Data?
Maybe
Retrieval
Uses
Knowledge?
What is the
Information
Retrieved?
No
Clinical Data
No
No
Knowledge
No
Yes
Yes
Knowledge
Yes
Yes
Yes
Clinical Data
or Knowledge
Infobuttons
Anticipate
Need and
Provide
Queries
i
Information Needs of CIS Users
• Common tasks may have common needs
• System knows:
– Who the user is
– Who the patient is
– What the user is doing
– What information the user is looking at
• So: We may be able to predict the specific need
• User is sitting at a computer!
• So: We may be able to get an answer automatically
Covel et al.
Information Needs
UMLS
Project
1984
1986
1988
1990
1992
1994
1996
1998
2000
2002
2004
Unified Medical Language System
The purpose of the UMLS is to improve the ability
of computer programs to “understand” the
biomedical meaning in user inquiries and to use
this understanding to retrieve and integrate
relevant machine-readable information for users.
- Donald A.B. Lindberg 1986/1993
Covel et al.
Information Needs
First Version
of UMLS
UMLS
Project
1984
1986
1988
1990
ICD9
MeSH
1992
Medline
Button
1994
1996
1998
2000
2002
2004
First Attempt: The Medline Button
•
•
•
•
CIS (WebCIS’s predecessor) on mainframe
BRS/Colleague (Medline) on same mainframe
Get them to talk to each other
Search using patient diagnoses and procedures
Admission Profile
----------------------------------------------------------------------------Admission Date: 01/03/95 Discharge Date: 02/16/95
Location:
Doctor: CIMINO, JAMES J
Discharge Summary: N
Primary Diagnosis: 410.71
ACUTE MI,SUBENDO INFARC, INITI
M6GS
Select Terms You Are Interested in:
X
X
_
410.71
780.3
507.0
426.0
415.1
453.8
428.0
Diseases:
ACUTE MI, SUBENDO INFARC, INITI
CONVULSIONS
FOOD/VOMIT PNEUMONITIS
ATRIOVENT BLOCK COMPLETE
PULMON EMBOLISM/INFARCT
VENOUS THROMBOSIS NEC
CONGESTIVE HEART FAILURE
F8 = for more information
----------------------------------------------------------------------------Help=F1
Search MEDLINE=ENTER
MEDLINE Queries from Admission Profile
----------------------------------------------------------------------------Select a question:
_
1.
Does Myocardial Infarction cause Convulsions?
2.
Is Myocardial Infarction caused by Convulsions?
3.
Does Myocardial Infarction occur with Convulsions?
----------------------------------------------------------------------------Help=F1
Search MEDLINE=ENTER
BRS Query from Admission Profile
----------------------------------------------------------------------------(Myocardial Infarction WITH (ET OR SC)) AND (Convulsions WITH CO)
----------------------------------------------------------------------------Help=F1
MEDLINE Queries=ENTER/F3
Session
Edit
Commands
Options
Help
^F File ^E Edit ^A Search ^L Limit ^V View
1
^T Tools
^O Options
Myocardial infarction/et,sc and convulsions/co
Ovid – Medline <1973-1983>
[To select option hold Ctrl and letter indicated.
Enter subject, then press <Enter>
_: _
Press ^Y for Help.]
^Y Help
1
First Attempt: The Medline Button
•
•
•
•
•
•
CIS (WebCIS’s predecessor) on mainframe
BRS/Colleague (Medline) on same mainframe
Get them to talk to each other
Search using patient diagnoses and procedures
Technical success
Practical failure
Covel et al.
Information Needs
First Version
of UMLS
PubMED
Mosaic
UMLS
Project
1984
1986
1988
1990
1992
1994
1996
WebCIS
ICD9
MeSH
Medline
Button
Web-based
Generic Queries
1998
2000
2002
2004
Covel et al.
Information Needs
First Version
of UMLS
PubMED
G.O. Barnett
DXplain
1984
1986
Mosaic
UMLS
Project
1988
1990
Web DXplain
1992
1994
1996
1998
WebCIS
ICD9
MeSH
Medline
Button
DXplain
Button
Web-based
Generic Queries
2000
2002
2004
Covel et al.
Information Needs
First Version
of UMLS
PubMED
G.O. Barnett
DXplain
1984
1986
Mosaic
UMLS
Project
1988
1990
Web DXplain
1992
1994
1996
1998
WebCIS
ICD9
MeSH
Medline
Button
2000
Infobuttons
DXplain
Button
Web-based
Generic Queries
2002
2004
“Just in Time” Education
MRSA
“Just in Time” Education
MRSA
1
Understand
Information
Needs
“Just in Time” Education
2
Get Information
From EMR
MRSA
1
Understand
Information
Needs
“Just in Time” Education
2
Get Information
From EMR
MRSA
1
Understand
Information
Needs
3
Resource
Selection
“Just in Time” Education
4
2
Get Information
From EMR
Resource
Terminology
MRSA
1
Understand
Information
Needs
3
Resource
Selection
“Just in Time” Education
4
Resource
Terminology
5
2
Automated
Translation
Get Information
From EMR
MRSA
1
Understand
Information
Needs
3
Resource
Selection
“Just in Time” Education
4
Resource
Terminology
6
5
2
Automated
Translation
Get Information
From EMR
MRSA
1
Querying
Understand
Information
Needs
3
Resource
Selection
“Just in Time” Education
4
Resource
Terminology
5
2
Automated
Translation
Get Information
From EMR
6
MRSA
1
Querying
Understand
Information
Needs
3
Resource
Selection
7
Presentation
Research Issues
• What are the information needs?
User’s
Workstation
75 foot cable
Portable Usability LabVideo Monitor
Converter Controller
Microphone
Video Converter
Cassette Recorder
Headphones
VCR
Morae screen
capture
Research Issues
• What are the information needs?
Information Resource Use According to Log Files
12% IBs
10% 1% 1%
6%
51%
6%
4%
4%
4%
3%
7% 3%
Micromedex
Pharmacy IB
Sensitivity IB
Culture IB
Ovid Medline
Online Formulary
Harrison's
PubMed
Journals
MedlinePlus
ICD9-CM
Other Resources
Research Issues
• What are the information needs?
• Which context information is important?
Context-Specific Resource Use
100%
Sensitivity IB
Culture IB
80%
Pharmacy IB
Medlineplus
60%
ICD9-CM
Journals
PubMed
40%
Harrison's
Formulary
20%
Ovid Medline
Micromedex
Ph
ar
m
ac
y
t
it
Li
s
S
ge
ch
ar
D
is
Vi
s
um
.
gy
ho
lo
Pa
t
ol
og
y
C
ar
di
C
lin
ic
gy
R
ad
io
lo
La
bo
ra
t
or
y
0%
Context-Dependent Information Needs
?
Age
Sex
Role Training
!
Task
Context
Data
Institution
Research Issues
•
•
•
•
What are the information needs?
Which context information is important
What resources can satisfy needs?
How can retrieval be automated?
– What context data are used?
– How are the data translated?
– How are the data transmitted?
Covel et al.
Information Needs
First Version
of UMLS
PubMED
G.O. Barnett
DXplain
1984
1986
Mosaic
UMLS
Project
1988
1990
Web DXplain
1992
1994
1996
1998
WebCIS
ICD9
MeSH
Medline
Button
2000
2002
Infobuttons
DXplain
Button
Web-based
Generic Queries
Infobutton
Manager
2004
Infobuttons vs. Infobutton Manager
Resource s
Clinical System
Infobutton
Query
Knowledge
Base
Context
Infobutton
Manager
Page
of
Hyperlinks
Covel et al.
Information Needs
First Version
of UMLS
PubMED
G.O. Barnett
DXplain
1984
1986
UMLS
Project
1988
1990
Web DXplain
1992
1994
1996
1998
WebCIS
ICD9
MeSH
Infobutton
Manager
Standard
Mosaic
Medline
Button
2000
2002
Infobuttons
DXplain
Button
Web-based
Generic Queries
Infobutton
Manager
2004
Infobutton Managers have been Deployed
• New York Presbyterian Hospital’s WebCIS
• New York Presbyterian Hospital’s Eclipsys
• LDS Hospital’s HELP system
• New York State Psychiatric Institute’s PSYCKES
• Regenstrief Medical Record System
• Partners Healthcare System’s Knowledgeliink
• Vanderbilt University’s PC-POETS
Evaluation of the Columbia University’s
Infobutton Manager at NYPH
• Use: who, what, when, where, and why?
• How usable is it?
• How useful is it?
• What impact is it having?
Evaluation Modalities
• System log files
• Pop-up surveys
• On-line feedback
• E-mail surveys
Evaluation
• Log file analysis:
analysis 32 months of data
• On-line survey
• Popup questionnaires
• E-mail survey
Results: System Log Files
• March 2004-October 2006:
– IM: 3,009 users, 29,541 times
Infobutton Manager (IM) Usage
1000
900
800
700
Housestaff
600
Attending
Nurse
500
Student
Other
400
300
200
100
Feb-06
Nov-05
Aug-05
May-05
Feb-05
Nov-04
Aug-04
May-04
Feb-04
Nov-03
Aug-03
May-03
Feb-03
Nov-02
Aug-02
0
Resource Use by Context
3000
Laboratory Results Review
Inpatient Drug Order Review
1000
2000
500
1000
0
0
Microbiology Sensitivity Results
Diagnosis List
300
100
200
50
100
0
120
0
Inpatient Lab Order Entry
200
Inpatient Drug Order Entry
100
80
150
60
100
40
50
20
0
0
Health Resources
Infobutton Manager
Results: System Log Files
• March 2004 - October 2006:
– IM: 3,009 users, 29,541 times
– HR: 3,609 users, 155,718 times
• October 2006:
– IM: 281 users, 1,022 times (33/day)
– HR: 708 users, 5,744 times (185/day)
Resource Use by Context - October
2500
2000
1500
Infobuttons
1000
Health
Resources
500
0
Lab
Results
Inpatient Outpatient
Drugs
Drugs
Order
Entry
Finding a Resource or Question
• Users chose questions 48.7% of the time
Diagnosis list:
10.5-24.3%
Outpatient drug order review: 23.0-63.3%
Inpatient drug order review: 63.3-78.7%
• Users chose at least one resource from HR page
92.4% of the time (82.6-93.7%, depending on
user type and context)
Evaluation
• Log file analysis:
analysis 32 months of data
survey 108 comments
• On-line survey:
• Popup questionnaires
• E-mail survey
Results: On-line Feedback
• 108 questions or topics submitted by users
– 54 (50%) in Laboratory Results Review context
• 30 were about test interpretation
– 22 (20%) in Inpatient Drug Order Review
• 15 were about side effects, dosage, cost,
trade names, and interactions
Evaluation
• Log file analysis: 32 months of data
• On-line survey: 108 comments
• Popup questionnaires: 195/~3,642 (5.4%)
• E-mail survey
Pop-up Survey Responses
100%
80%
Strongly Disagree
60%
Disagree
Neutral
Agree
40%
Strongly Agree
20%
Q1
Q2
Q3
Other
Student
Housestaff
Attending
Nurse
Other
Student
Housestaff
Attending
Nurse
Other
Student
Housestaff
Attending
Nurse
0%
Q1: Easy to use
Q2: Got answer
Q3: Helpful
Evaluation
• Log file analysis: 32 months of data
• On-line survey: 108 comments
• Popup questionnaires: 195/~3,642 (5.4%)
• E-mail survey
survey: 73/1,228 (5.8%)
E-mail Survey Responses
100%
80%
5
4
60%
3
2
40%
1
20%
Q2
Q3
Q4
Q5
Q2: Infobuttons are easy to use
Q3: Presence of question on the list
Q4: Speed of answer
Q5: Helpful answer
Q6: Positive effect on patient care decisions
1 (strongly positive) to 5 (strongly negative)
Attending
Housestaff
Nurse
Student
Other
Attending
Housestaff
Nurse
Student
Other
Attending
Housestaff
Nurse
Student
Other
Attending
Housestaff
Nurse
Student
Other
Attending
Housestaff
Nurse
Student
Other
0%
Q6
Results: Pop-Up and E-Mail Surveys
Pop-Up
83%
E-Mail
92%
89%
Easy to use
Question on list >50% of time
Answered question
69%
Useful
77%
Helpful >50% of time
90%
Positive effect on care
74%
Specific cases of improved care
14
(40% of negative responses due to Lab Manual)
(Students were the least positive)
Summary of Impact
• Over 29,000 additional accesses to resources (19%)
• Positive impact at least half the time: 74%
• 14 respondents identified one or more specific
situations in which patient care improved
Lessons Learned
• Use of the IM has been more gradual than desired
• User fails to select a question half the time
– Is the IM failing to anticipate the information need?
– Example - questions sugested during lab review:
Housestaff did not select a question 57.8% of time
Online survey: 4/11 dealt with reference ranges
IM almost always provides a link to lab manual
– Is the user unable to find questions on the IM page?
• Solutions:
– Education
– Add questions
– User interface
Conclusions
1. Context-specific access to health knowledge
resources has been successfully accomplished
2. Impact on patient care decisions has been positive
3. Need to improve user interface for navigation
4. Increased resource use should result in clinicians
making better informed patient care decisions
Acknowledgments
• This work is supported by NLM grant R01LM07593
• Evlauation plan:
– Vimla Patel
– Sue Bakken
– Leanne Currie
– Beth Friedman
• Programming: Jianhua Li
• Log files: Rick Gallagher