Transcript 2008-AMIA-Leading a Horse to Water - People

Leading a Horse to Water:
Using Automated Reminders to Increase
Use of Online Decision Support
James J. Cimino and Dmitriy Borovtsov
NIH Clinical Center
and
Department of Biomedical Informatics
Columbia University College of Physicians and Surgeons
Knowledge Resources are Underused
Clinicians (physicians) with questions in clinical practice:
– sought answers only 45% of the time
– found answers only 34% of the time
– used online resources 18% of the time
* Ely JW, Osheroff JA, Maviglia SM, Rosenbaum ME. Patient-care questions
that physicians are unable to answer. J Am Med Inform Assoc. 2007 JulAug;14(4):407-14.
Health Resources
• Available from main menu
• Static list of resources
• No assistance with retrieval
Infobuttons
• Links inserted next to clinical information
• Resources selected based on likely
information need
• Links are customized to assist with retrieval
Log File of HR vs IB 2006
7000
M
o 6000
n
t 5000
h
l 4000
y
3000
U
s 2000
a
g 1000
e
0
Jan
Feb Mar
Apr May Jun
Jul
Aug Sep Oct
Nov Dec
2006
Health Resources
Infobuttons
Log File of HR vs IB 2006, 2007
7000
M
o 6000
n
t 5000
h
l 4000
y
3000
U
s 2000
a
g 1000
e
0
Jan
Feb Mar
Apr May Jun
Jul
Aug Sep Oct
Nov Dec
2006
Health Resources
Infobuttons
2007
Educational Interventions
• Orientation for new housestaff
• Orientation for new medical students
• Infobutton exercises for new medical students
Educational Interventions
M
o
n
t
h
l
y
450
U
s
a
g
e
200
400
350
300
250
150
100
50
0
Jun
Jul
Aug
Sep
Oct
Nov
2006
Medical Students
Housestaff
Dec
Educational Interventions
M
o
n
t
h
l
y
450
U
s
a
g
e
200
400
350
300
250
150
100
50
0
Jun
Jul
Aug
Sep
Oct
Nov
2006
Medical Students
Housestaff
Dec
2007
Usage of IB vs HR in Inpatient Meds
M
o
n
t
h
l
y
U
s
a
g
e
1400
1200
1000
800
600
400
200
0
Jan
Feb Mar
Apr May Jun
Jul
Aug Sep Oct
Health Resources
Infobuttons
Nov Dec
Usage of IB in Outpatient Meds
M
o
n
t
h
l
y
U
s
a
g
e
800
700
600
500
400
300
200
100
0
Jan
Feb Mar
Apr May Jun
Jul
Aug Sep Oct
Health Resources
Infobuttons
Nov Dec
Hypothesis
Users of the Health Resources page in
a particular context will be especially
receptive to a suggestion to use
Infobuttons in the same context.
Intervention
HR used
in OM
IB Used
in past
2 months?
Yes
Stop
No
Reminder
Sent in past
2 months?
No
HR = Health Resources Page
OM = Outpatient Medications
Send
Reminder
Yes
Stop
E-Mail
Date: Fri, 2 Mar 2007 06:00:11 -0500 (EST)
From: James Cimino <[email protected]>
To: [email protected]
Subject: Getting Drug Information while Using WebCIS
Dear WebCIS User:
I am writing to let you know about a handy feature in WebCIS called the
"infobutton". You will see it in places like the Outpatient Medication list - it is
a little purple circle with a white letter "i". If you click on it, it will give you a
list of topics that you can select to get more information about the drug next
to the infobutton.
I hope you'll find it useful.
--Jim Cimino, Infobutton Project Manager
Reminders Sent
• 522 messages
• 371 users
– 80 attending physicians
– 189 housestaff
– 29 nurses
– 24 students
– 49 other/unknown
• 173 users received two messages
• 1 user received three messages
• Average 89.9 days between messages
Usage of Infobuttons
by Reminder Recipients
• 371 recipients of 552 reminders
• 111 (20.1%) eventually used IBs in OM
• 52 (9.4%) “early responders”
– 26 housestaff
– 18 attending physicians
– 8 other/unknown
• 201 used either IB or HR within one month
• 25.9% “early responders with opportunity”
• 9 early responders used IBs 8 to 97 times in
subsequent months
Intervention versus Control
% 10%
U
s
e
o
f
I
B
s
9%
8%
7%
6%
5%
4%
3%
2%
1%
0%
P<0.05 by Day 4
1
3
5
7
9
11
13
15
17
19
21
23
25
27
29
31
Days to First Use of Infobuttons after First Use of Health Resources Page
552 Intervention
525 Control
Effect of delay
400
300
Reminders
Sent
Early
Responders
200
100
0
1-2
3-5
>5
Days Days Days
Adverse Effects
• 522 reminders
• 12 replies
– All were positive
– Most were thanking us for infobuttons
Discussion
• Traditional training method appears to
have limited effect
• Simple, spam-like reminder appears to
have an effect on a substantial fraction
(26%) of users
• Sustained effect in some (17%) of the
early responders
Limitations
• Impersonal message
• Timing of reminder less-than ideal
• However: Logfile+e-mail approach is
application independent
• Quasi-experimental design
• Convenience sample of controls
• However: Strong temporal association
between intervention and outcome
Conclusions
• Automated, context-specific reminders are
technically feasible, requiring few resources
• Reminders to use online information
resources appear to be effective
• Timing of message is critical to success
Acknowledgments
This work was supported in part by:
National Library of Medicine Grant R01LM07593
NIH Clinical Center intramural research funds
National Library of Medicine intramural research funds
The authors also thank:
Dr. Jianhua Li for technical support
Dr. Rick Gallagher for log files
Dr. Krystl Haerian for statistical support