Transcript Carol O`Byrne

“Performance Reports for
Failing Candidates ”
Carol O’Byrne
Pharmacy Examining Board of Canada
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
What failing candidates want to know
• How close was I to passing?
• What did I do wrong? What did I miss?
• How many such errors and omissions lead to a
failing result?
• In which area(s) do I need to improve?
• What does PEBC expect in these areas?
• Why am I expected to perform at a higher level
than what I see some pharmacists doing?
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
PEBC rationale for providing
feedback to candidates
• Supports PEBC’s mandate: to certify candidates who
demonstrate that they have the knowledge, skills,
abilities and attitudes required for practice
• Increases candidates’ awareness of practice
requirements
• Supports the cooperative but arms length relationship
between credentialing bodies and training bodies
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Rationale…
Benefits all parties:
• Assists candidates to recognize and address their
weaknesses
• Improves efficiency of PEBC processes and lessens
potential threat on exam security by reducing the number
of retakes
• Benefits the profession and the public by supporting
further development of qualifications of those preparing to
enter practice
• Addresses manpower needs - guides remediation and
bridging efforts, facilitating earlier entry to the profession
of those who may not yet have received adequate training
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Why only to failing candidates?
• No demand from passing candidates
• Resource issues
– Issuance of reports
– Failing candidates often retake the exam
without appropriate preparation and plug the
system
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
PEBC Qualifying Examination
• Based on national
competencies and
standards
• Offered in English
and French
• Must be PEBC
certified to license in
9/10 provinces
• Mobility enabled by
mutual recognition (if
PEBC certified)
Part I (MCQ) – 200 scored
items
Part II (OSCE) – 15 scored
stations
12 SP/HP interactions
+ 3 non-client stations
• 7 minutes/station
• 1 assessor/station
• 2 sets of scores/station
Analytical checklist
Holistic scales
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Competencies assessed
Competencies
Weights
%
1. Practise pharmaceutical care
29
2: Assume ethical, legal and professional
responsibilities
9
3: Access, retrieve, evaluate and disseminate
relevant information
5
4: Communicate and educate effectively
43
5: Manage drug distribution
9
6: Apply practice management knowledge and skills
5
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Test format
• Interactive client stations
– Standardized patients
– Standardized health professionals
• Non-client stations
– Technical, e.g.:
• Screening prescriptions for appropriateness
• Checking dispensed prescriptions
– Written short answer, e.g.:
• Responding to drug information requests - evaluating and
interpreting drug information from several / conflicting sources
• Medication management - reviewing patient data and
recommending therapeutic options, along with a rationale
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Assessor scoring sheet - ratings
Three 4-point scales
1. Communications – generic scale
–
–
–
Rapport
Organization and flexibility (adaptive to the client/situation)
Verbal and nonverbal skills (including language proficiency)
2. Outcome (problem solving) – station specific scale
–
Based on critical checklist items
3. Overall Performance – inclusive, global scale
–
–
–
–
Communications and outcome
Process quality and thoroughness (critical and noncritical items)
Accuracy (vs misinformation)
Risk (occurrence, degree)
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Assessor scoring sheet - checklist
• ‘Critical’ items ()
– essential to solve the problem & meet station objective/s
– each linked to a competency assessed in the station
• ‘Noncritical’ items
– represent good practice & contribute to effective
outcome(s)
– each linked to a competency
• Risk and misinformation
• Unique response (UR) - for scoring & QA purposes
• Comment boxes - to record evidence to support
scores (used for QA purposes)
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Scoring the examination
Analytical scores
• Each checklist item relates
to one competency
• Competency sub-scores =
percent of items related to
each competency to which
candidate responds
• Frequency of risk and
misinformation tabulated
Holistic scores
• Each scale 1 to 4 points
– 12 points per client station
(Comm, Outc, Perf)
x 12 stations
– 8 points per nonclient
station (Outc, Perf)
x 3 stations
• Raw score = sum of all
stations holistic scale scores
• Holistic cut score set for
each scale in each station
• Cut score = sum of all
stations holistic cut scores
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Mean scores & alphas
Coefficient 
.83
12 stns - competency 4
2. Outcome
.66
15 stns – all competencies
3. Performance
Holistic Scales
4
Scale Value
Holistic Scales
1. Communications
3
2.5
2
1.5
1
.73
1
2
Holistic Scale Means
15 stns – all competencies
3
Analytical Scores
Analytical Scores n items Coefficient
1

0.8
Pharm care
107
.80
0.6
Ethics
7
.43
0.4
0.2
Drug information 8
.24
0
Communications 14
.33
1
2
3
4
5
6
Competency Means
Drug distribution 8
.57
Management Presented
8 at the 2005 CLEAR
.55 Annual Conference
Percent
1.
2.
3.
4.
5.
6.
3.5
September 15-17
Phoenix, Arizona
Factors affecting competency subscore reliabilities
• Candidate variability (or lack of)
• Number and context of stations in which
the competency was assessed
• Number of non-critical items vs critical
items (importance of their performance
to the task at hand)
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Reports to candidates
• Results: pass-fail status (all candidates)
• Feedback (for failing candidates, on request):
– Individual score breakdown
• by major skill – mean Communication, Outcome and
Performance ratings – aggregated across all stations
• by competency – mean percent scores – aggregated across all
stations in which the competency was assessed
• by critical incident – frequency of risk, misinformation
– Comparative data
• ‘Reference group’ mean scores and frequencies for score
comparison with a stable population
• to show where performance needs to improve
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
QUALIFYING EXAMINATION - Part II (OSCE) -
revised 15Mar04
Training Station - Cancer Pain Control (Video Performance)
Station Checklist
Location:
CASE #____
0
1
2
3
4
5
6
7
0
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
Shift :__Candidate:______Start Stn:__
_____________
8
A
9
Track:
1
B
2
C
D
E
3
4
5
Assessor:_____
8
0
0
F
6
G
H
I
J
7
8
9
0
Client_____
0
6
0
0
0
ATTACH BAR CODE HERE
Be sure to attach candidate id barcode label. Also, ensure that your assessor id barcode or client (sp
id) barcode are correct. If not correct, please mark an X through the barcode box and enter the correct
number in the spaces (blank lines) provided above the box.
Practise Pharmaceutical Care:
Communicate and Educate Effectively:
1.
Asks about nature of the pain (e.g. severity, frequency).
2.
Asks patient about side affects (e.g. constipation).
Unique Response (if any)
Advises / informs:
*
*
*
*
If another response is given
which you are unsure is
appropriate - or which
influences your grading below please shade in the bubble
and
note details
3.
It is safe to take both tablets and liquid.
4.
Liquid morphine is for immediate relief of occasional pain; tablets are for
maintaining pain control.
5.
Should stop taking Tylenol.
6.
Liquid morphine will act quickly to relieve your pain.
7.
Continue taking one morphine tablet twice daily.
8.
Take 5 mL every 4 hours if needed / if pain recurs before you are to take your next morphine tablet.
9.
You may experience more drowsiness / sleep (if you take both liquid and tablets).
10. For constipation, increase fluid and fibre intake, exercise regularly and/or could use a stimulant laxative.
11. Contact your doctor if you get breakthrough pain often OR need the liquid often.
12. If you need liquid morphine regularly it may indicate that your tablets are not strong enough / that you
need a dose adjustment.
13. Do NOT take both tablets and liquid / unsafe. (incorrect response)
COMMENT (if rating less than Acceptable/Marginal or Solved/Marginal) : use back of sheet if needed
Fill in one bubble for each rating scale below - to score the candidate's performance.
Communications
Acceptable
Acceptable/ Marginal
Unacceptable/ Marginal
Unacceptable
Outcome
Problem Solved
Solved/ Marginal
Uncertain
Unsolved
Performance
Acceptable
Acceptable/ Marginal
Unacceptable/ Marginal
Unacceptable
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Misinfo
Risk
Assessor scoring sheet
STATION __
PEBC QUALIFYING EXAMINATION - PART II (OSCE)
EXAMINATION FEEDBACK REPORT
OSCE feedback report
Candidate ID
Table 1
Status: Fail
RATINGS
Communication
Your average
3.10
Group average 3.67
Rating
4=Acceptable
3=Marginally acceptable
2=Marginally unacceptable
1=Unacceptable
Outcomes
Your average
2.33
Group average 2.90
Rating
# of Stations (of 15)
4=Problem solved
2
3=Solved marginally
5
2=Uncertain/marginally unsolved
4
1=Problem unsolved
4
Overall Performance
Your average
2.51
Group average 3.01
Rating
4=Acceptable
3=Marginally acceptable
2=Marginally unacceptable
1=Unacceptable
Table 2
# of Stations (of 15)
3
5
4
3
MISINFORMATION AND RISK/INEFFECTIVE THERAPY
Misinformation
Risk or ineffective therapy
Table 3
# of Stations (of 12)
4
6
1
1
# of Stations/Instances (of 15)
3
2
Group Average
1.31
0.63
COMPETENCY SCORES
Competency
Your Score (%)
1 Practise pharmaceutical care
47
2 Assume ethical, legal and professional responsibilities
39
3 Access, retrieve, evaluate, disseminate relevant information
44
4 Communicate and educate effectively
35
5 Manage drug distribution
80
6 Apply practice management knowledge and skills
50
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Group Average (%)
59
56
47
53
74
63
Candidate findings
• Most candidates understand the information provided
but want more guidance (content information – where
they went wrong)
• Some do not accept the exam results and feedback
information – may request hand-scoring
• Failing candidates generally score low in
Communications (rating scale and competency 4)
and/or Pharmaceutical Care (competency 1 – clinical
role)
• Many failing candidates lack clinical training in
Canada (or the US) - though many have some
technical training / experience (as a pharmacy
technician)
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Are we really helping candidates?
• Anecdotally, yes – some do not know where
to start, what to focus on
• Skills scores and competency sub-scores are
consistent enough to be meaningful in areas
that are weighted more heavily
• All candidates who fail show weaknesses in
one or more of these areas (low scores
relative to the reference group)
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
What questions do (can) we
answer?
 What area(s) do I need to improve?
 What does PEBC expect in these areas?
 What did I do wrong? What did I miss?
Why am I expected to perform at a higher level
than what I see some pharmacists doing?
 How many errors and omissions lead to a failing
result?
 How close was I to passing?
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
What other strategies are (may be)
helpful?
• Provide information about training and/or
remedial resources, e.g.:
– Clear expressions, including visual exemplars, of good
practice in each competency area
– Recognized training programs and resources
– Practice exams (e.g. ‘mock OSCE’s’) for format
familiarization
• Provide general tips, e.g.:
– Typical performance errors/deficits in each competency
– Competency-related descriptions of candidates who are
clearly qualified, borderline qualified and unqualified
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona
Contact information
Carol O’Byrne
Pharmacy Examining Board of Canada
415 Yonge Street, Suite 601
Toronto, ON M5B
T: 416-979-2431, ext 226
Email: [email protected]
Website: www.pebc.ca
Presented at the 2005 CLEAR Annual Conference
September 15-17
Phoenix, Arizona