Transcript Abstract Presentation (167424 bytes)

Assessing the efficacy of
EBM teaching in a
clinical setting
Gardois P., Grillo G., Lingua C.,
Jourdan A., Fronteddu S., Piga A.
Turin – Italy
24/9/2004
Santander, 9. EAHIL Conference
1
Background: evaluating IR and EBM
skills
► Growing
popularity of EBM practice
►  Growing number of EBM courses
► Many methods for teaching and learning
evaluation
► But... only few existing data on the
application of the EBM skills in the clinical
practice, mainly when:
 the working environment is not EBM-oriented
 there is an interest in long-term results
24/9/2004
Santander, 9. EAHIL Conference
2
Objectives
► To
assess the long term effect of an
educational program for health
professionals, based mainly on information
retrieval and EBM principles
► Secondary goals:
 to understand the role of EBM methods and
instruments in clinical practice
 to assess users’ perception of the main
barriers to EBM practice
24/9/2004
Santander, 9. EAHIL Conference
3
Our educational program…
►
►
Institution: Pediatrics and Gynecology Departments of
Turin University (Italy)
Number of courses: 6 (5 with CME credits).
 4 focused on medical information retrieval:
► theory of databases
► basics of information retrieval
► basic and advanced use of medical
► bibliography
formatting software
databases and search engines
 2 specifically EBM oriented:
► advanced IR skills
► basic statistics
► study design
► critical appraisal
►
Time span: 2001-03 to 2003-06
24/9/2004
Santander, 9. EAHIL Conference
4
…Our educational program
► Average
duration: 16.5 hours (min 11, max 28)
► 7 teachers and 6 tutors (librarians, clinicians,
statisticians, epidemiologists)
► 126 unique users (140 total); categories:




MDs, postgraduating doctors
Nurses, midwives
PhDs (chemistry, biology, …)
Other technical staff
► average
5.3)
24/9/2004
learners/facilitators rate: 3.9 (min 2.7 max
Santander, 9. EAHIL Conference
5
Methods: 1. IR skills durability
assessment
► The
multiple choice questionnaire
administered after the course was
administered again in may 2004 (11 to 38
months after)
  assessment of the Delta (Δ) between the two
results
24/9/2004
Santander, 9. EAHIL Conference
6
Methods: 2. EBM skills durability
assessment
► The
first part of the Fresno Test (validated in
2002; see Ramos – BMJ 2003) was administered
to test EBM skills in 4 areas:




24/9/2004
Clinical question
Selection of bibliographic sources
Selection of study design
MEDLINE search strategy
Santander, 9. EAHIL Conference
7
Methods: 3. EBM practice
assessment
► Qualitative
questionnaire with the following
main categories of items:
 Ideal and real percentage of clinical activity
devoted to EBM
 Relative frequency of use of different
bibliographic sources
 Experience of barriers and facilitators for EBM
practice
24/9/2004
Santander, 9. EAHIL Conference
8
Methods: users invitation and
questionnaires completion
► Users
approached by letter and – after 1
week – by e-mail, explaining the study
purpose and methods
► Up to 5 phone calls during 3 weeks to make
an appointment for questionnaire
completion
► Questionnaires administered individually in
the library, without the use of computers.
► 45 minutes for completion
24/9/2004
Santander, 9. EAHIL Conference
9
Methods: statistical analysis
► Software
Windows
24/9/2004
used: StatSoft Statistica 6.1 for MS
Santander, 9. EAHIL Conference
10
Results
►1
Basic statistics
 1.1 Response rate
 1.2 Demographics (gender, age, role)
► 2.
Descriptive statistics
 2.1 Δ first – second multiple choice questionnaire
 2.2 Fresno test results
 2.3 Average responses for any variable in the EBM
practice questionnaire
►3
Correlations between:
 1.2 + 2.3 vs 2.1 + 2.2
24/9/2004
Santander, 9. EAHIL Conference
11
Basic statistics: response rate
► 70
/ 126 users  55,6% response rate
► N = 70
 Based on reported opinions the questionnaire
was considered “difficult”.
 A higher response rate might have been
reached by accepting “remotely completed
questionnaires”, but the data reliability would
have been lower
24/9/2004
Santander, 9. EAHIL Conference
12
Demographic data
► Gender:
F 71%, M 29%
► Age: Mean = 41,03, StdDv = 8,70, Max =
62, Min = 27
► Role: 72% doctors, 28% other roles
24/9/2004
Santander, 9. EAHIL Conference
13
Delta between 1st and 2nd
completion of multiple choice
learning evaluation questionnaire:
► delta%:
N = 70; Mean = -19,43;
StdDv = 19,02
24/9/2004
Santander, 9. EAHIL Conference
14
Barriers
► The
main barriers affecting the results of
the 2nd completion of the multiple choice
questionnaire are:
 Finding contradictory results in the literature
 Knowledge of English
 Insufficient number of PCs
24/9/2004
Santander, 9. EAHIL Conference
15
Barriers
► The
main barriers affecting the Δ pct
between 1st and 2nd completion of the
multiple choice questionnaire is insufficient
access to PCs
24/9/2004
Santander, 9. EAHIL Conference
16
Results: Fresno test
The average total score was 54,99 (valid N
46, Std.Dev. 24,24).
24/9/2004
Santander, 9. EAHIL Conference
17
Barriers
► The
main barriers affecting the Fresno test
total score are:




24/9/2004
Lack of institutional support
Lack of statistics skills
Insufficient search strategy building abilities
Poor knowledge of English language
Santander, 9. EAHIL Conference
18
Results: EBM practice ideal and real
With a valid N of 66, the average pct of
time that our professional would devote
to EBM practice was 23,06 (20,10
Std.dev), the real pct of time they devote
is 6,29 (7,56 std.dev).
24/9/2004
Santander, 9. EAHIL Conference
19
EBM practice: ideal and real
► The
real and ideal practice of EBM are
correlated. Users who consider more
important EBM practice tend to have a
higher percentage of real EBM practice
24/9/2004
Santander, 9. EAHIL Conference
20
Comment
► Time
is an important barrier as to practice
EBM skills. There is a significant correlation
between the two variables
24/9/2004
Santander, 9. EAHIL Conference
21
Main bibliographic sources searched
(pct. total weekly research hours, valid N = 67)
► MEDLINE
31,00
► Proceedings 16,40
► Journals
16,30
► Colleagues 10,07
24/9/2004
(25,78 std.dev)I
(18,22 std.dev)I
(13,38 std.dev)I
(16,62 std.dev)I
Santander, 9. EAHIL Conference
22
Main barriers to EB practice
AVG points on 1-5 Likert scale:
“totally disagree” to “totally agree”
► Time
3,4 (1,2 Std.Dev.)
► Isolation
3,2 (1,1 Std.Dev.)
► Lack of institutional support 3,2
Std.Dev.)
24/9/2004
Santander, 9. EAHIL Conference
(1,3
23
Main facilitators for EB practice
AVG points on 1-5 Likert scale:
“totally disagree” to “totally agree”
► Time
4,0 (0,8 Std.Dev.)
► Critical appraisal skills
4,0 (0,9
Std.Dev.)
24/9/2004
Santander, 9. EAHIL Conference
24
Other remarks
► Users
who had good 1st questionnaire
results had a greater probability to have
also good 2nd questionnaire results
► Users who had good 2nd questionnaire
results had a greater probability to have
also good Fresno test total score results
►  Good internal coherence of the
assessment methods of the study
24/9/2004
Santander, 9. EAHIL Conference
25
Conclusions
► EBM
skills tend to be lost after some time
► Our users tend to consider ideal EBM
practice very important, but the real
practice of EBM is significantly lower than
the ideal one
► The main barriers to EBM practice, and the
more related to the results of our tests are:
time, access to PCs, isolation and lack of
institutional support
24/9/2004
Santander, 9. EAHIL Conference
26
Further research
► We
need larger, multicenter studies to reach
more reliable and relevant results
► We need to test if – for the practicing
professional – regular recalls of EBM
concepts are useful
► We need to study more extensively the
influence of barriers vs EBM practice and to
take consequent action
24/9/2004
Santander, 9. EAHIL Conference
27