Transcript Developing High Quality Clinical Skills Assessments

Developing High Quality
Clinical Skills Assessments
University of North Carolina – Chapel
Hill School of Medicine
November 10, 2011
Ann Jobe, MD, MSN
Clinical Skills Evaluation Collaboration
Philadelphia, Pennsylvania

The most important consideration in
developing a clinical skills
assessment is to keep in focus the
purpose of the assessment
Formative or Summative?


Formative: to provide feedback for
improvement of performance, usually
midway through a course of study.
Summative: to distinguish between
those who are competent, and those
who aren’t, usually at the end of a
course of study.
Test Design

Begin at the beginning…
• What are you trying to assess?
• What is the level of the examinee?
• How much time will the examinee have?
Can they complete the task in the time
allotted?
Test Design
• What are you trying to teach and
assess?

Objectives for assessment – history taking
skills, PE skills, communication skills, data
interpretation skills?
• What is the level of the examinee?

Focus and length of checklists will be
different for 1st year medical students and
individuals seeking board certification
USMLE Step 2 Clinical Skills
Purpose is to assess ability to:
 Gather information from patients
 Correctly perform physical
examination maneuvers
 Synthesize and communicate
findings to patients and colleagues
Examinees are moving to supervised
patient care (PGY-1)
Development Of An
Examination Blueprint


A test blueprint defines the requirements
for each examination, regardless of where
or when it is administered.
Commonly seen cases.
Blueprint (continued)

Identify the criteria used to define an
exam blueprint
• Examination length
• Case content
• Examinee tasks (e.g. history, physical,
communication)
• Setting: inpatient or outpatient
• Patient gender, age
Case Selection

Documentation of commonly seen
cases
• if specialty focused assessment, cases
will be those common to that specialty


Most common presenting complaints
in clinical setting
Health Department population
statistics – especially for region,
state or local area
Case Pool Size


Dependent on blueprint criteria
Security concerns (more is better!)
Case Content
Cardiovascular
Respiratory
Gastrointestinal
Musculoskeletal
Constitutional
Neurological
Psychiatric
Genitourinary
Women’s health
Other
Case Acuity
Acute
Subacute/Chronic
Form
Patient age
Age less than 18
Age 18 – 44
Age 45 – 64
Age 65 +
Patient Gender
Male
Female
Blueprint Criteria Define Case Needs

This information is provided to faculty
to guide their case development
• Case content: Gastrointestinal
• Case acuity: Acute
• Patient age: 18-44
• Patient gender: Female

Complete a “Medically Relevant Case
Details” sheet
Case Development Committee

Physicians (medical school faculty):
content experts

SP trainers: training experts

SPs: portrayal experts

Process evolved from experience


Usually break into several groups
Each group focuses on developing
several cases to fit blueprint needs
Case Development Process

Chief Complaint

“Medically Relevant” case details

Patient Personality Profile

Case checklist

Examinee instructions

Patient note development
Chief Complaint/
Differential Diagnoses



Brainstorming process
List all of the possibilities
Then narrow the choices as
development proceeds
Medically Relevant Case Details



Patient /Case Name
Race
• Any
• African American
• Caucasian
• Other _________________
BMI
•
Any
•
Weight proportionate to height
•
Overweight: > 24
•
Other
Medically Relevant Case Details

Gender
•

Male
Female
Either
Age Range
• 10 year age range – example 30 years old = 2535




Acuity
• Acute
• Subacute/Chronic
Category (Blueprint)
Exclusionary Medical conditions/scars
Differential Diagnosis
Medically Relevant Case Details

Communication Tasks
•
•
•
•
•



Fostering the Relationship
Gathering Information
Providing Information
Making Decisions
Supporting Emotions
Overview of Case Stimuli for Communication
Tasks
Thread of Encounter
Primary MD
Personality Profile


Giving life and individuality to the
patients who are being seen – more
like “real patients”
Not verbatim statements from a
script – more general ideas of each
patient’s perspective
Personality Profile





Who am I and how would I describe
myself?
Why am I here?
What made me come in today?
What do I think is going on and why?
What are my expectations for the
visit and how will I react if my
expectations are not met?
Personality Profile






What questions do I have?
What concerns/anxiety/fears do I
have?
How has this illness impacted my
life?
What do I bring with me?
What do I look like?
How do I behave/what’s my
communication style (mood/attitude
related to my illness)?
Personality Profile


What is my level of trust of the
medical profession?
What type of communication style do
I appreciate in a health care
provider? How will I respond to
different styles of communication?
How much information do I want the
doctor to provide?
Personality Profile




What words won’t I understand
(jargon)?
What are my own words I use to
describe what is going on?
How motivated am I to accept
medical advice?
How do I feel about the use of
complementary/alternative
medicine?
Case Checklist





Recording instrument
Checklist length
One concept per item
Evidence-based items
Use lay language
Example: Onset
(“I’ve been coughing for about a
month.”)
Refining the Checklist



Brainstorm a large number of items
Role play
Checklist revised
• A – essential
• B – important
• C – relevant
• F - delete
Examinee Instructions






Name
Age
Setting
Presentation problem
Vital signs
Examinee Tasks
• Refers to the skills that the
examination assesses
Role Play with SP



A physician from another group
“sees” the patient
Following the encounter, the
physician provides his/her differential
diagnoses (see if it matches what the
group selected): face validity
Revise the checklist based on
observing the physician
Role Play with SP

This allows an SP to provide
feedback on any difficulties that arise
in portrayal of the case – especially if
an SP needs to repeat the case
several times in an assessment
session
Development of Patient Note

Key essentials and case-based
scoring guidelines
Questions?