Transcript Anticipating and Satisfying Clinician Information Needs

Anticipating and Satisfying
Clinician Information Needs:
The Infobutton Manager Project
James J. Cimino, M.D.
Department of (Bio)Medical Informatics
Weekly Seminar
March 6, 2003
Studying Clinician Information
• Covell DG, Uman GC, Manning PR. Information
needs in office practice: are they being met? Ann
Intern Med 1985 Oct;103(4):596-9
• Other studies: Osheroff, Forsythe, Gorman, Ely…
• Types of needs
– Patient-specific
– General knowledge
– Institutional
• Resources
– Medical record
– Colleagues
– Journals
– Books
Do Clinicians Satisfy their Needs?
•
•
•
•
•
•
Needs are underestimated
Information needs may not be recognized
Resources are outdated or unavailable
Computers not used
Resolution may be deferred
Unsatisfied information needs are a potential
source for errors
Needs Related to CIS Use?
• Very small part of the clinician’s day
• Very important part of the day
– New information stimulates needs
– Decision-making moments
• Opportunity to resolve needs (using a computer)
• Context might predict needs
• Context might help resolve needs
• Needs in this setting have not been studied
Context-Dependent Information Needs
?
Age
Sex
Role Training
!
Task
Context
Data
Institution
Hypothesis:
• Information needs are stereotypical
• Information needs can be predicted based on context
• Appropriate resource can be identified (automatically
or in advance)
• Resource can be queried automatically
• Information from context can be used for query
• “Infobuttons”
A Brief History of Infobuttons
• 1991: “Medline Button” translated ICD9 to MeSH
and performed Medline searches
• 1994: World Wide Web
• 1994: First CPMC Web-based CIS
• 1996: WebCIS
• 1996: Infobuttons link microbiology and medication
data to PubMed and Micromedex
• 1997: Infobuttons for patients (PatCIS)
• 2002: Handheld, wireless infobuttons (PalmCIS)
Infobutton Manager Project
•
•
•
•
•
•
Study information needs of CIS users
Characterize needs based on context
Develop way to anticipate needs
Develop ways to resolve needs
Deploy anticipation/solution
Evaluate
Infobutton Manager Team Members
•
•
•
•
•
•
•
•
•
Jim Cimino
Jianhua Li
Sue Bakken
Vimla Patel
Mark Graham
Leanne Currie
Mureen Allen
Anne-Marie Ramierez
Your name here?
Methods
• Determine information needs
– WebCIS log files
– WebCIS user questionnaires
– Observational studies
• Characterize information needs
• Build Infobutton Manager (IM)
• Build infobuttons
• Populate IM database
• Integrate IM with WebCIS
• Evaluation
Insert WebCIS Demo Here
WebCIS Log File Methods
• Health Resources page in WebCIS
• Infobuttons in WebCIS
• Problems:
– Which user selected which resource?
– What was the user was doing?
• Solution: look at previous activity at same IP address
WebCIS Log File Records
Mar 2 16:16:46 webcis3-i syslog:
|WebCIS|ciminoj|156.111.145.61|3131313
|lab^2003-03-02-05.19.00.819650|view
Mar 2 16:18:55 webcis3-i syslog:
|WebCIS|ciminoj|156.111.145.61|3131313
|healthresource|view
Mar 2 16:18:58 webcis3-i syslog:
|Resources|(nobody)|156.111.145.61||
|url^http://salk.cpmc.columbia.edu:70/mdxdocs
WebCIS Log Files Results
•
•
•
•
Six months of log files
2,607 users (1,034 per month) used a resource
2931 IPs (2008 “156.x.x.x”)
User types:
Housestaff: 51%
Attendings: 34%
Nurses:
5%
Other:
10%
WebCIS Log Files Results
38,763 uses of resources
14,036 outside WebCIS (2339/month)
24,727 within WebCIS (4,104/month; 4/user)
19,913 uses of 24 resources (3,319/month)
4,814 uses of 3 infobuttons (802/month)
WebCIS Log Files Results
Popular resources:
Micromedex:
Ovid:
NYP Formulary
Harrison’s
PubMed
Health Journals
Medline Plus
ICD9-CM
Infobuttons
Pharmacy/Micromedex
Microbiology/PubMed
Pharmacy/PubMed
Inside
56.5%
7.4%
7.2%
5.1%
5.1%
5.0%
3.3%
3.3%
Outside
51.7%
8.2%
7.1%
5.1%
5.1%
8.2%
3.6%
2.9%
92.9%
6.1%
1.0%
-
WebCIS Log Files Results
Health Resources: 19,913 times in 105 contexts:
Lab results:
52.1%
Radiology results:
9.0%
Clinical notes:
7.4%
Pathology results:
2.8%
80.4%
Visit list:
2.1%
Pharmacy:
2.0%
Discharge summary: 1.9%
Infobuttons: 4,814 times in 2 contexts:
Pharmacy:
93.9%
100%
Lab results (Micro): 6.1%
WebCIS Log Files Results
Context:
MDX
Ovid
Formul.
Harrison
PubMed
Journals
Med+
ICD9
Infobuttons
MDX
PubMed
Lab Rad
53% 9%
49% 9%
52% 8%
49% 10%
48% 10%
43% 8%
50% 8%
37% 12%
86%
-
Note
3%
6%
6%
5%
7%
7%
6%
10%
Card
3%
2%
3%
3%
3%
6%
3%
2%
-
-
Path Vis Phar DSum
3% 2% 2%
2%
3% 1% 1%
1%
3% 1% 3%
2%
4% 1% 1%
3%
5% 1% 3%
1%
5% 2% 2%
1%
2% 4% 2%
1%
3% 2% 1%
2%
-
- 100%
- 14%
-
WebCIS Log Files Results
Context:
Lab Rad Note Card Path Vis Phar DSum
MDX
56% 57% 61% 59% 49% 42% 9% 62%
Ovid
7% 7% 8% 5% 8% 4% 1% 6%
Formulary 7% 6% 6% 6% 7% 3% 1% 7%
Harrisons 5% 6% 4% 5% 6% 3% 1% 7%
PubMed 5% 5% 5% 4% 8% 2% 1% 3%
Journals 4% 4% 5% 9% 8% 3% 1% 3%
Med+
3% 3% 2% 3% 3% 32% 1% 2%
ICD9
1% 4% 4% 2% 3% 2% .2% 3%
Infobuttons
MDX
- 84% PubMed 3% 2% -
WebCIS Log Files Results
User:
Housestaff Attending
MDX
62%
61%
Ovid
6%
7%
Formulary
6%
5%
Harrison
4%
3%
PubMed
4%
5%
Journals
3%
0%
Med+
2%
2%
ICD9
3%
5%
Statref
1%
1%
PIER
0%
2%
Infobuttons
MDX
5%
6%
PubMed
2%
1%
Nurse
42%
>1%
3%
4%
1%
3%
3%
1%
1%
3%
Other
27%
1%
5%
3%
1%
1%
2%
1%
3%
1%
42%
2%
53%
1%
WebCIS Usage Logs Results
• Take home messages:
– Micromedex is a popular resource
– Ovid edges out PubMed
– Lab tests are a popular context for resources
– No correlation between context and resource
– Nurses and pharmacists like the Micromeex
infobutton for microbiology
• Still needed:
– Context/resource/user type interactions
– Concept-oriented contexts
• Questions:
– How much is based on knowledge of resources?
– What are the questions in each context?
WebCIS User Questionnaires
•
•
•
•
Health resources page outside vs. inside WebCIS
Resources used, how often, how useful, anecdotes
Infobuttons used, how often, how useful, anecdotes
Other resources used
WebCIS User Questionnaire Results
1055 users in January
300 randomly selected
250 with valid e-mail addresses surveyed
42 responded
21 attendings
19 housestaff
1 nurse
1 pharmacist
WebCIS User Questionnaire Results
R
e
s
o
u
r
c
e
I
n
f
o
b
u
t
How often do you access:
Never <Month Monthly Weekly Daily
Attend
2
1
3
5
9
House
3
11
5
Nurse
1
Pharm
1
Attend
House
Nurse
Pharm
11
14
1
8
3
1
2
1
1
WebCIS User Questionnaire Results
R
e
s
o
u
r
c
e
I
n
f
o
b
u
t
Why do you access:
Patient Qs Genl Know Other
Attend
20
12
3
House
18
10
1
Nurse
1
Pharm
1
1
Attend
House
Nurse
Pharm
10
4
1
WebCIS User Questionnaire Results
R
e
s
o
u
r
c
e
How do you access:
None WC-Login WC-Menu Both Other
Attend
2
4
7
8
12
House
0
8
8
3
17
Nurse
1
Pharm
1
1
WebCIS User Questionnaire Results
What resources and how useful:
Very
Somewhat
A H
A
H
MMDX
17 17
0
2
Ovid
10 11
1
2
PubMed
12 7
2
3
Medline+
4 4
0
1
FullText
8 7
2
2
Harrisons
7 4
3
9
PIER
2 4
3
1
Formulary
2 3
2
1
Health J
6 9
0
1
Science J
3
4
0
1
STATref
1
2
6
0
ICD9
1
2
5
0
Not _
A
H
1
0
0
0
0
0
1
0
0
0
1
0
1
0
1
0
0
1
1
0
1
1
0
1
WebCIS User Questionnaire Results
How useful are infobuttons:
Very
Somewhat
A H
A
H
Micro Org
3
3
5
1
Micro Sens
7
3
2
3
Pharmacy
6
1
0
2
A
2
0
0
Not _
H
0
0
0
WebCIS User Questionnaire Results
How does self-reporting correlate with log files?
AA
A
100
80
A
H
A
60
40
A
A
A
20
0
A
Never
A
HA
A
H
H
HA
H
H
HH
H N
HHH
A H
A
A
A
A
A
H
HH
P
H
A
H
<Monthly
Monthly
Weekly
Health Resources
Daily
WebCIS User Questionnaire Results
How does self-reporting correlate with log files?
25
20
A
15
H
A
AP
A
10
5
0
H
H A
A H
H AHHHHHH
A H
AA
AAAAHAAH N
Never
HA
A
A
AH A
<Monthly
H
H
Monthly
Infobuttons
Weekly
Daily
WebCIS User Questionnaire Results
• Anecdotes:
– Avoid errors (drug interactions)
– Identify correct doses
– Identify pills
– Prepare publications
– Teaching
– Understand abbreviations (sensitivity tests)
• Other desired resources:
– Up To Date
– NEJM
– MDConsult
– Books
WebCIS User Questionnaire Results
• Resources are used and liked
• Resources are used within WebCIS
• Resources are used to answer patient-specific
questions
• Use of resources may be overestimated (can’t tell
for sure)
• Infobuttons are used and liked
• Additional resources requested
• Questions:
– How knowledgeable are users about resources?
– What are the context-specific questions
Observational Studies
•
•
•
•
•
•
Connect portable usability lab to clinical workstation
Ask users to think aloud about information needs
Videotape screen and audiotape users
Transcribe tapes
Code transcripts, using tapes
Analysis of coding
User’s
Workstation
75 foot cable
Portable Usability LabVideo Monitor
Converter Controller
Microphone
Video Converter
Cassette Recorder
Headphones
VCR
Observational Results
• Sites:
– Medical /Surgical
– CCU
– Ambulatory Care
• Number of Days:
– 7 different days
– Varying days of the week
– Varying times of day
• Total hours of tape: 15 hrs 29 minutes
• Total Number of events: 154
Observational Results
Total Number of events: 154
Medical/Surgical
71 (46%)
CCU
35 (23%)
Ambulatory Care
48 (31%)
Observational Results
Total Number by User Type
Nurse
17 (11%)
House Officer
103 (67%)
Attending/Fellow 18 (12%)
Student
7 (5%)
Event Coding
• Types of Events:
– Foreground/Background
– Implicit/Explicit
– Patient/Institution/Administrative/Domain
• Types of Resources
– Human/Computer/Paper
• Types of Outcomes
– Success/Deferred/Failed
Coding Event Type
• Foreground
– Specific information about patient management
– Problem, Intervention, Comparison, Outcome
• Background
– Root question: who, what, where,when, why, how?
– Related to health disorder or aspect of a disorder
Coding Event Type
Foreground or Background
1
2
Total
Frequency
89
Percent
57.8
65
42.2
154
100.0
Coding Event Type
• Explicit: Subject expresses an information need
• Implicit: Subject uses a resource other than WebCIS
Coding Event Type
Explicit or Implicit
1
2
Total
Frequency
131
Percent
85.1
23
14.9
154
100.0
Coding Event Type
• Patient
– A patient related need
• Institution
– An information need that is related to the
infrastructure of the institution
• Administrative
– An institutional information need that is
related to a specific patient
• Domain
– A domain related information need that is
related to a specific patient
Coding Event Type
Patient or Institution
Valid
1
2
Total
Missing
System
Total
Frequency
127
Percent
82.5
26
16.9
153
99.4
1
.6
154
100.0
Administrative or Domain
Valid
1
2
Total
Frequency
98
Percent
63.6
56
154
36.4
100.0
Coding Resources Used
• Human
– The subject sought information from a human
interaction in person or via telephone
• Computer
– The subject sought information from a
computer interaction either webCIS or other
internet resource
• Paper
– The subject sought information from a paper
resource – patient chart or text book
Coding Event Type
Human, Computer or Paper
1
Percent
36.4
2
77
50.0
3
20
13.0
153
99.4
1
.6
154
100.0
Total
Missin
g
Total
Frequency
56
System
Coding Outcomes
• Success
– The subject sought and found the desired
information
• Deferred
– The subject did not seek an answer
• Failed
– The subject sought, but did not find desired
information
Coding Outcomes
Success, Deferred or Failed
1
Frequency
75
Percent
48.7
2
45
29.2
3
34
22.1
154
100.0
Total
Coding Events
• Need to correlate across event types
• Especially need to look at types of needs that
were deferred or that failed
• Good news:
– Information needs exist and can be identified
– Needed resources are computer-based
• Bad news (good news for us):
– Many needs failed or deferred
– Solutions will require integration of patient
data with information resources
• Questions:
– What are the triggers?
– What resources can help?
Classification of Information Needs
Classification
Definition
Navigation
A question about anything in WebCIS (either currently present or could be
added at a later date to WebCIS) that the user is unable to access or use.
The IB link would take the user to a resource that tells him how to access
the needed information.
Functionality
A response or question which alludes to the need to add an item to
WebCIS or give the user access to another resource which answers the
need. The IB would access the resource that would solve the problem
and present the solution to the user.
Laboratory
Information needs that are institution or patient specific related to the
laboratory. The IB would provide specific information about a laboratory
finding.
Pharmacy
Information needs that are institution or patient specific related to the
pharmacy. The IB would provide information about specific medications.
Differential
Diagnosis
Questions related to alternate causes of a symptom or laboratory or
radiology finding. IB would provide context-specific differential diagnoses
Definition/
Information
Questions related to the meaning of words. In addition, general
information on a topic. IB would provide a definition or general information
about a topic/word.
Miscellany
Those needs that do not readily translate into patient/institution needs.
Methods: Classification Scheme
General question: What is the normal value for this
calcium result?
Generic question: What is the normal value of this
laboratory test?
Classification: Navigation: because normal laboratory
values are available in WebCIS, this IB link would
take the user to another page informing the users
on how to access the information needed.
Methods: Classification Scheme
General question: Since the CT scan of the liver is
abnormal what are his liver function test results?
Generic question: What laboratory tests are
associated with this radiology report?
Classification: Functionality group since it will require
designing an IB that would bring information to the
user from another area in WebCIS (or some other
resource) while he is at a particular site.
Observational Studies Results
Classification Code
Navigation
Functionality
Laboratory
Pharmacy
Differential
Diag.
Definition/info.
Miscellany
Total
Count
Percent
48
27
19
31
31.2
17.5
12.3
20.1
9
5.8
15
5
154
9.7
3.2
100.0
Observational Studies Results
•
•
•
•
•
Wide variety of information needs
Many needs are stereotypical
Many can be answered with simple resources
Some complex solutions will be needed
Next steps:
– Finish the characterizations of needs
– Propose ways to solve each need
– Build infobuttons (project opportunities!)
The Infobutton Manager
•
•
•
•
•
•
CGI script using “get” or “post”
Context parameters as input
Uses context table and infobutton table
Returns a list of questions that are links
Log files for calls to IM and selecting links
Infobutton Manager Manager for maintenance
Matching Context to Need
I
n
f
o
b
u
t
t
o
n
s
Infobutton Manager
Context Table
Question
Selection
Instantiation
Context
Matching
Question Table
C
o
n
t
e
x
t
Context Parameters
• Patient: age, gender
• User: physician, nurse, student, patient, …
• Setting: lab reports, medications, …
• Concept of interest: lab test, drug, …
Infobutton Parameters
• Question type: fixed, fill-in-the-blank
• Source type: simple query, additional data
• URL:
http://www.guideline.gov/SEARCH/search_results.asp?
sSearch_string=<>&bSort_by_date=&query_id=&
bDebug=&results=10&view=search_results
• Question:
What are guidelines for evaluation of <>?
Infobutton Manager Manager Demo
Insert Infobutton Manager Demo Here
Insert Infobutton Manager Manager Demo Here
Evaluation Plan
• Baseline observation
– Creation of Infobutton Manager tables
– Resources used
– Unmet information needs
• Log of access to online resources
• Introduction of infobuttons
– Usability studies
• Log of access to online resources (infobuttons)
• Post-intervention observation
– Use of infobuttons and other resources
– Unmet information needs
Discussion
•
•
•
•
•
•
•
•
Information needs occur and are detectable
Information needs are unmet
Information needs are predictable based on context
Solutions are at hand
Solutions are institution independent
Integration into clinical systems is easy
Will infobuttons reduce unmet information needs?
Next project: show that this reduces errors!
Acknowledgments
• National Library of Medicine
grant R01-LM07593
• Rick Gallagher
• The team: Jianhua Li, Sue
Bakken, Vimla Patel, Mark
Graham, Leanne Currie, Mureen
Allen, Anne-Marie Ramierez
<a href="Javascript:get_info('35702')"><img src="info.gif"></a>
function get_info(mc){ var infowin = window.open("","INFO");
infowin.focus(); document.INFO.info_med.value=mc;
document.INFO.submit(); }
<form method="POST" name=“INFO”
action="wc_infomanage.cgi">
<input type="HIDDEN" name="info_med" value="">
<input type="HIDDEN" name="info_context" value="LabDetail">
<input type="HIDDEN" name="info_institute" value="CPMC">
<input type="HIDDEN" name="DOB" value="1951-05-26">
<input type="HIDDEN" name="SEX" value="F">
<input type="HIDDEN" name="USER" value="ciminoj">
</form>
Context Parameters
• Patient: age, gender
• User: physician, nurse, student, patient, …
• Setting: lab reports, medications, …
• Concept of interest: lab test, drug, …
• Institution: CPMC, RMRS, LDS, …
<form method="POST" action="wc_infomanage.cgi">
<input type="HIDDEN" name="info_med" value=“35702">
<input type="HIDDEN" name="info_context" value="LabDetail">
<input type="HIDDEN" name="info_institute" value="CPMC">
<input type="HIDDEN" name="DOB" value="1951-05-26">
<input type="HIDDEN" name="SEX" value="F">
<input type="HIDDEN" name="USER" value="ciminoj">
</form>
<form method="POST" action="wc_infomanage.cgi">
<input type="HIDDEN" name="info_med" value=“DIGOXIN|..">
<input type="HIDDEN" name="info_context" value="LabDetail">
<input type="HIDDEN" name="info_institute" value=“RMRS">
<input type="HIDDEN" name="DOB" value="1951-05-26">
<input type="HIDDEN" name="SEX" value="F">
<input type="HIDDEN" name="USER" value="ciminoj">
</form>