Transcript Ottawa 2012 KL Cushing Cardiff April 19th

Domain Based Communication
Skills Assessment Pilot For EU
Doctors
Annie M Cushing1, Jean S Ker2, Paul Kinnersley3,
Anthony N Warrens1, Olwyn M R Westwood1 ,
Pascal McKeown4
European Union Law:
Article 53 of Directive 2005/36/EC
‘Persons benefitting from the recognition
of professional qualification shall have a
knowledge of languages necessary for
practicing the profession in the host
Member State’
Assessment is NOT a test of
Clinical Knowledge
Legal Framework:
‘Proportionate and necessary’ :
1. Applied in a non-discriminatory manner
2. Justified by overriding reasons based on the general
interest
3. Suitable for securing the attainment of the objective
which they pursue and
4. Not go beyond what is necessary to obtain that
objective
Mis-communication is multi-factorial
• Language Issues
• Cultural differences in concept of roles, systems
and professionalism
• Consultation style and Communication skills in
doctor-patient relationship
• Cultural aspects of healthcare and team
relationships
Pilotti LS 2007, Watt D, 2007,
Woodward-Kron R 2009, Trafford P 2010 (personal communications)
Development Process for Test of Applicants to
the UK Foundation Programme
Validity
• Tasks (GMC Tomorrow’s Doctors 2009, Scottish Doctor)
• Domain-based mark scheme (Communication)
• Setting the Standard – Pass mark
Reliability
• Training: Assessors Clinicians and
Simulated Patients
• Information for Candidates
BLUEPRINT TEMPLATE FOR POPULATING EXAM STATIONS
CONTEXTS
TASKS
History
Routine
Oral
Emotional
Challenge
Station 1
Station 15
Diversity culture,
interpreter/
impairment
Telephone
Station 13
Reading/
Writing
Partnership
Station 2
Acute Setting
Station 1
Drug History
Explain Condition
Station 5
Explain Treatment /
Procedure
Station 4
Examination/Procedure
Station 15
Station 14
Station 3
Station 2
Station 7
Write Notes/Drug Chart
Read Notes/Drug Chart
Station 7
Team/Handover
Station 6
Station 7
3rd Party/Interpreter
Station 6 &12
Station 7
Team Receive Instructions
Bad News/Sensitive Issues/
Error/Complaint
Station 7
Station 14
Station 5
Station 8
Station 13
Shared Decision Making
Station 9
Consent / Ethical Issues
Station 10
DOMAINS
A = English language skills
1. Listening comprehension: understands patient/colleague
2. Spoken English comprehension: patient/colleague understands
3. Reading comprehension: demonstrates understanding of written input
4. Written comprehension: writing understood to enable appropriate action
B = Clinical communication
1. Opening the consultation: establishes relationship with patient
2. Professional /humanistic quality: continues to maintain the relationship
3. Information gathering: gathers enough information to understand the problem(s)
4. Examining the patient: patient is enabled to co-operate with examination
5. Explaining/negotiating/consenting: information is understood with appropriate
patient involvement
6. Communication with colleagues/team members
C = Patient ratings:
Patient satisfied with doctor-patient relationship and understanding
OSCE Format
15 Station Exam , 10 Minutes each = 2hrs 30mins
5th year Medical Students (n= 15 at each site)
Results (n = 58)
Raw Percentage Mean
Standard Deviation
Cronbach’s alpha
Standard Error of Measurement
= 81.4
= 6.7
= 0.78
= 3.11
Standard Setting by the Borderline Regression Method
Cut Score (pass mark)
= 62.44
Cut Score + 1 SEm
= 65.62
Cut Score + 2 SEm
= 68.80
Cumulative Marks’ Curve (integrated frequency
distribution) combined sites scores
Cumulative Marks' Curve, Cut Score and Standard Error of Measurement (SEM)
Cumulative Marks' Curve
Cut score
Cut score + 1 SEm
Cut score + 2 SEms
Percentage of candidates
100
90
80
70
60
50
40
30
20
10
0
50
60
70
80
90
Raw marks, %
100
Overall Marks vs no. of stations failed, overall cut
score, confidence intervals and stations’ criterion
Figure 1: Marks scored and stations failed
ACCS Pilot BL-Dund-Card-Belf
Cut Score + 1 SEm
Stations' Criterion
Cut Score
Cut Score + 2 SEm
Linear (ACCS Pilot BL-Dund-Card-Belf)
Candidates' Scores (raw%)
100
95
90
R² = 0.5987
85
80
75
70
65
60
55
50
-1
1
3
5
No of Stations failed
7
9
Means of Domains & Variance across All Stations
A1-4 = Language, B1-5 = Consultation, B6 = Team, C= Patient Rating
Next Steps
Pilot with Doctors qualifying from
non-English Speaking Countries
Acknowledgements
Sam Leinster
Pascal McKeown
John Patterson
Lynne Rustecki
Katie Petty-Saphon
Siobhan Fitzpatrick
Jonathan Silverman
Penny Trafford
Tim Swanick