Transcript Part 2 - Med-Peds Program Directors Association

Observable Practice Activities (OPAs)
Content-Based (COPA)
Mapping
Reporting
Milestones
External Reporting
NAS report
Process-Based (POPA)
Mid/End of Rotation
Feedback (formative)
Entrustment (1-5)
Curricular
Milestones
Mapping
End-of-training
EPAs
Longitudinal Assessment
(summative)
Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C.
Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of
print] PubMed PMID: 24557518
Mapping
• We mapped with the following in mind:
– when assessing a trainee on a given OPA, would the
evaluator feel as if they were also assessing the mapped
milestones?
– We removed all soft calls
Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C.
Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of
print] PubMed PMID: 24557518
OPA: Minimize unfamiliar terms during patient encounters
Milestones
ICS-A2 - Effectively use verbal and nonverbal skills to create
rapport with patients/families
ICS-D1 - Deliver appropriate, succinct, hypothesis-driven
oral presentations
SBP-B2 - Work effectively as a member within the interprofessional team to ensure safe patient care
GIM
PGY-1
Wards
COPA: Identify causes of delirium
Reporting Milestones
PC-1 - Gathers and synthesizes essential and accurate
information to define each patient’s clinical problem(s).
MK-1 Clinical knowledge
COPA: Manage exacerbations of
obstructive lung disease
Reporting Milestones
PC-2 Develops and achieves comprehensive management plan
for each patient.
PC-3 Manages patients with progressive responsibility and
independence.
MK-1 Clinical knowledge
MK-2 Knowledge of diagnostic testing and procedures
COPA: Initiate cost-effective workup for
anemia
Reporting Milestones
PC-1 Gathers and synthesizes essential and accurate
information to define each patient’s clinical problem(s)
MK-1 Clinical knowledge
MK-2 Knowledge of diagnostic testing and procedures
SBP-3 Identifies forces that impact the cost of health care, and
advocates for, and practices cost-effective care
COPA: Assess and treat pain
Reporting Milestones
PC-2 Develops and achieves comprehensive management
plan for each patient
PC-3 Manages patients with progressive responsibility and
independence
MK-1 Clinical knowledge
PROF-3 Responds to each patient’s unique characteristics and
needs
POPA: Demonstrate shared decisionmaking with the patient
Reporting Milestones
PC-2 Develops and achieves comprehensive management
plan for each patient
PROF-1 - Has professional and respectful interactions with
patients, caregivers and members of the interprofessional
team (e.g. peers, consultants, nursing, ancillary
professionals and support personnel)
PROF-3 Responds to each patient’s unique characteristics and
needs
ICS-1 - Communicates effectively with patients and
caregivers.
POPA: Use feedback to improve
performance
Reporting Milestones
PBLI-3 - Learns and improves via feedback
PROF-4 - Exhibits integrity and ethical behavior in professional
conduct.
Mapping OPAs to Milestones
Rotation 1
OPA 1
OPA 2
OPA 3
OPA 4
OPA 5
OPA 6
OPA 7
OPA 8
Score
3
3
2
3
2
2
3
2
Rotation 2
OPA 9
OPA 10
OPA 11
OPA 12
OPA 13
OPA 14
OPA 15
OPA 16
3
3
3
4
2
2
3
2
Patient Care
Medical
Knowledge
Practice Based
Learning and
Improvement
Interpersonal and
Systems
Communication
Based
Skills
Professionalism Practice
A8
A4, A5, A6, A8
A4, A5, D1
A3
F2
C3
F4
A2
C4, E2, F6, F7,
A6,
Patient Care
Medical
Knowledge
F5, F6,
F5, F6,
C2, C3, C4
C4
C3, F5, F6
F5, F6,
F5, F6,
F5, F6,
A4, A6, A8,
A4, A6, A8,
B1, B2
A4, A6, A8, B2
A4, A6, A8, B2
A4, A6, A8, B2
A4, A6, A8,
A4, A6, A8,
Practice Based
Learning and
Improvement
B1
E2, E3
Interpersonal and
Systems
Communication
Based
Skills
Professionalism Practice
E2, E3
Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C.
Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of
print] PubMed PMID: 24557518
Mapping OPAs to Milestones
Rotation 1
OPA 1
OPA 2
OPA 3
OPA 4
OPA 5
OPA 6
OPA 7
OPA 8
Score
3
3
2
3
2
2
3
2
Rotation 2
OPA 9
OPA 10
OPA 11
OPA 12
OPA 13
OPA 14
OPA 15
OPA 16
3
3
3
4
2
2
3
2
Patient Care
Medical
Knowledge
Practice Based
Learning and
Improvement
Interpersonal and
Systems
Communication
Based
Skills
Professionalism Practice
A8
A4, A5, A6, A8
A4, A5, D1
A3
F2
C3
F4
A2
C4, E2, F6, F7,
A6,
Patient Care
Medical
Knowledge
F5, F6,
F5, F6,
C2, C3, C4
C4
C3, F5, F6
F5, F6,
F5, F6,
F5, F6,
A4, A6, A8,
A4, A6, A8,
B1, B2
A4, A6, A8, B2
A4, A6, A8, B2
A4, A6, A8, B2
A4, A6, A8,
A4, A6, A8,
Practice Based
Learning and
Improvement
B1
E2, E3
Interpersonal and
Systems
Communication
Based
Skills
Professionalism Practice
E2, E3
Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C.
Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of
print] PubMed PMID: 24557518
Mapping OPAs to Milestones
Rotation 1
OPA 1
OPA 2
OPA 3
OPA 4
OPA 5
OPA 6
OPA 7
OPA 8
Score
3
3
2
3
2
2
3
2
Rotation 2
OPA 9
OPA 10
OPA 11
OPA 12
OPA 13
OPA 14
OPA 15
OPA 16
3
3
3
4
2
2
3
2
Patient Care
Medical
Knowledge
Practice Based
Learning and
Improvement
Interpersonal and
Systems
Communication
Based
Skills
Professionalism Practice
A8
A4, A5, A6, A8
A4, A5, D1
A3
F2
C3
F4
A2
C4, E2, F6, F7,
A6,
Patient Care
Medical
Knowledge
F5, F6,
F5, F6,
C2, C3, C4
C4
C3, F5, F6
F5, F6,
F5, F6,
F5, F6,
A4, A6, A8,
A4, A6, A8,
B1, B2
A4, A6, A8, B2
A4, A6, A8, B2
A4, A6, A8, B2
A4, A6, A8,
A4, A6, A8,
Practice Based
Learning and
Improvement
B1
E2, E3
Interpersonal and
Systems
Communication
Based
Skills
Professionalism Practice
E2, E3
Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C.
Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of
print] PubMed PMID: 24557518
Mapping OPAs to Milestones
Rotation 1
OPA 1
OPA 2
OPA 3
OPA 4
OPA 5
OPA 6
OPA 7
OPA 8
Score
3
3
2
3
2
2
3
2
Rotation 2
OPA 9
OPA 10
OPA 11
OPA 12
OPA 13
OPA 14
OPA 15
OPA 16
3
3
3
4
2
2
3
2
Patient Care
Medical
Knowledge
Practice Based
Learning and
Improvement
Interpersonal and
Systems
Communication
Based
Skills
Professionalism Practice
3
3
2
3
2
2
2
3
2
2
Patient Care
Medical
Knowledge
3
3
3
4
2
2
3
2
3
3
3
4
2
2
3
2
Practice Based
Learning and
Improvement
3
3
Interpersonal and
Systems
Communication
Based
Skills
Professionalism Practice
3
Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C.
Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of
print] PubMed PMID: 24557518
Mapping OPAs to Milestones
Rotation 1
OPA 1
OPA 2
OPA 3
OPA 4
OPA 5
OPA 6
OPA 7
OPA 8
Score
3
3
2
3
2
2
3
2
Patient Care
Practice Based
Learning and
Improvement
Interpersonal and
Systems
Communication
Based
Skills
Professionalism Practice
3
3
2
3
2
2
2
3
Rotation 2
OPA 9
OPA 10
OPA 11
OPA 12
OPA 13
OPA 14
OPA 15
OPA 16
Medical
Knowledge
3
3
3
4
2
2
3
2
AVG
2
2
Patient Care
Medical
Knowledge
3
3
3
4
2
2
3
2
2.67
3
3
3
4
2
2
3
2
2.67
Practice Based
Learning and
Improvement
3
Interpersonal and
Systems
Communication
Based
Skills
Professionalism Practice
3
3.00
3
2.67
2.50
3.00
Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C.
Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of
print] PubMed PMID: 24557518
Patient Care
Patient Care
Traditional Competency-Based Evaluations
Titrate Insulin
Manage Vent
Read EKG
OPA Based Evaluations
Diagnose Hepatitis
Titrate Insulin Manage Vent Read EKG
The Milestones are the
apples to be compared
Diagnose Hepatitis
MK
A1
1. Understand the relevant
pathophysiology and basic science
for common medical conditions
PC
E2
2. Make appropriate clinical
decision based upon the results of
diagnostic tests
PBLI
1. Identify learning needs
B1
(clinical questions) as they emerge
in patient care activities
Approximately
3000
Assessments
Reporting Milestones
Resident A
Resident B
Resident C
Reporting Milestones
Resident D
Resident E
Reporting Milestones
Resident F
Resident F
6 nurses
Reporting Milestones (ACGME)
Example: PGY-3 30 Month NAS Report
1
1.5
2
2.5
3
3.5
4
4.5
5
PGY-3
Single
PGY-1
NOT
Reviewed
Entire
PGY-1s
PGY-2
Depth
Residency
byCompetency
CompetencyCommittee
Committee
Single
PGY-1
– Reviewed
ininDepth
by
717
982 Assessments
Assessments
Accepts
Acceptsresponsibility
responsibilityand
andfollows
follows
through
on
tasks.
Has professional
respectful
Accepts
responsibility
and follows
through and
on tasks.
Works effectively
within
Transitions
patients
effectively
within in
Recognizes
systemand
error
andanadvocates
Exhibits integrity
ethical
behavior
Skill
interactions
with patients,
Skillin
inperforming
performingprocedures.
procedures.
through
on tasks.caregivers…
Skill
in
performing
procedures.
interprofessional
team
(e.g.
peers,…
and
across
health
delivery
systems.
for
system
improvement.
Identifies
forcesof
that
impact the
costand
of
Exhibits
integrity
and
ethical
behavior
in within
Knowledge
diagnostic
testing
Learns and
improves
atpatients
the point
of respectful
professional
conduct.
Knowledge
Transitions
of diagnostic
testing
and
Works
effectively
within
an within
Has professional
and
Knowledge
diagnostic
testingand
and
Transitions
patientseffectively
effectively
Knowledge
ofofdiagnostic
testing
health
care,
and
advocates
for,
and…
professional
conduct.
procedures.
care.
procedures.
and
across
health
delivery
systems.
interprofessional
team
(e.g.
peers,…
interactions
with
patients,
caregivers
3.8
procedures.
andCommunicates
across
health
delivery
systems.
3.8
procedures.
Identifies
forces
that
impact
costofof
Requests
and
provides
consultative
Learns and
improves
at thethe
point
effectively
in
and members of the interprofessional…
3.8
3.8
health
care,
and
advocates
for,
and…
care.
care.
interprofessional
teams
(e.g.
peers,…
Gathers
and
synthesizes
essential
and
Exhibits
integrity
and
ethical
in
Identifies
forces
that
impact
the
cost
of
Identifiesforces
forcesthat
thatimpact
impactthe
thecost
costofof
Exhibits
integrity andeffectively
ethicalbehavior
behavior
in
Identifies
Communicates
in
professional
conduct.
accurate
information
to
define
each
…
health
care,
and
advocates
for,
and…
healthcare,
care,and
andadvocates
advocatesfor,
for,and
and…
professional
conduct.
health
interprofessional
teams
(e.g.
peers,
3.3
3.3
Gathers
andof
synthesizes
and
Learns and
improves via
Knowledge
diagnosticessential
testing
Accepts
responsibility
andperformance
follows
practices
cost-effective
care.and
consultants,
nursing,
ancillary…
3.3
3.3
audit.
accurate information
to
define
each
…
procedures.
through
on
tasks.
Appropriate
utilization
andessential
completion
Communicates
in
Gathers
synthesizes
and
Gathersand
and
synthesizes
essential
and
Has
professionaleffectively
and respectful
2.8
of
health
records.
interprofessional
(e.g.
peers,…
Manages
patients
with
progressive
Transitions patients
effectively
within
accurate
information
totodefine
each
2.8
interactions
with teams
patients,
caregivers…
accurate
information
define
each……
2.8
responsibility
and independence.
and
across healtheffectively
delivery systems.
Monitors practice
with a goal for
Communicates
in
2.8
Appropriate utilization and completion
Communicates effectively with patients
improvement.
interprofessional teams (e.g. peers,…
of
health
records.
and caregivers.
2.3
2.3
Has
professional
and
respectful
Monitorspatients
practicewith
withprogressive
a goal for
Manages
Works
effectively
within
an
Clinical knowledge
2.3
Works
effectively
an
interactions
with patients,
caregivers…
improvement.
2.3
responsibility
and independence.
interprofessional
teamwithin
(e.g.
peers,…
Clinical knowledge
interprofessional team (e.g. peers,
Appropriate utilization and completion
1.8
consultants,
nursing,
ancillary…
Clinical
knowledge
1.8
Develops and achieves
comprehensive
Responds
toofeach
patient’s
health
records. unique
1.8
1.8
management
plan for each
patient.
characteristics
and needs.
Learns and improves
at the
point of
Communicates
effectively
withinpatients
Communicates
effectively
Clinicalessential
knowledge
Gathers and synthesizes
and
care.
and
caregivers.
interprofessional
teams (e.g. peers,…
Responds
to each patient’s
unique
accurate information to define each
characteristics
and needs.
Transitions
patients effectively
within
patient’s
clinical problem(s).
Skill
in synthesizes
performing
procedures.
Gathers
and
essential
and
and
across
health
delivery
systems.
Learns
andtosystem
improves
via and
feedback.
Identifiesinformation
forces that impact
theeach
cost…of
Recognizes
error
advocates
Responds
each
patient’s
unique
Communicates
effectively
with
patients
accurate
tothe
define
Clinical
knowledge
Learns
andcare,
improves
at
point
care.
health
and advocates
for,of
and…
for system
improvement.
characteristics
and needs.
and
caregivers.
Communicates effectively with patients
Requests and provides consultative care.
andeffectively
caregivers.
Works effectively within an
Communicates
with patients
Monitors
practice
aprogressive
goal
for for
Worksto
effectively
within anunique
Manages
patients
with
Responds
each
patient’s
Monitors
practice
with
a goal
Responds
to
eachand
patient’s
unique
interprofessional
teamwith
(e.g.
peers,…
caregivers.
Learns
and improves
via (e.g.
feedback.
improvement.
interprofessional
team
peers,…
responsibility
and
independence.
characteristics
and
needs.
improvement.
characteristics
and
needs.
Develops and achieves comprehensive
Learns and improves via feedback.
management
planachieves
for
each
patient.
Has
professional
and
respectful
Monitors
practice
with a goal for
Develops
and
comprehensive
Appropriate
utilization
and via
completion
Transitions
patients
effectively
within
Learns
and improves
feedback.
Learns
and
via
Requests with
and provides
consultative
care.
Learns
andimproves
improves
viafeedback.
feedback.
Clinical
knowledge
interactions
patients,
improvement.
management
plan
forcaregivers…
each
patient.
of
health
records.
Appropriate
utilization
and
completion
and across health delivery systems.
Learns and
improves
at
the
point
of
care.
Learns and
improves
viacomprehensive
performance
Develops
and
achieves
comprehensive
Recognizes
system
and
advocates
Develops
and
achieves
Appropriate
utilization
and
completion
oferror
health
records.
Responds
toand
eachimproves
patient’s
unique
Develops
and
achieves
comprehensive
Learns
via
performance
Requests
and provides
consultative
Monitors
practice
with a goal for
Learns
and improves
via
performance
audit.
management
plan
for
each patient.
for system
improvement.
management plan
for each patient.
of health
records.
management
plan
for
characteristics
and
needs.
audit.
care.
Requests
and
provides
improvement.
andin
improves
viaconsultative
performance
Exhibits integrity
and
ethical Learns
behavior
Manages
patientsaudit.
with
progressive each patient.
Manages patients
with
progressive
care.
audit. responsibility and independence.
professional conduct.
responsibility and independence.
NAS report Jul-Dec 2013
Assist colleagues in the provision of duties.
central lines.
Demonstrate
accurate,
Initiate
antibiotic
therapy
for
neutropenic…
Respond
tosafe,
pages
inshared
timelyand
and
courteous…
Write
a prescription
for Perform
the
appropriate…
Provides
regular
feedback
tocomplete…
other…
Implement
a
diagnostic
work-up
Demonstrate
decision-making
Takesfor…
leadership role of teaching…
Managefor
and
writechest
ventilator
orders
Perform
bedside
presentations
thatwith…
engage…
Initiate evaluation
acute
syndrome…
Demonstrate
empathy,
compassion,
Improve
care
via
Plan-Do-Study-Act
(PDSA)…
Was
this
resident
receptive
to
the
unique…
Manage
heart
failure
(acute,
chronic,…
Conduct
a
comprehensive
psychiatric…
Collaboratively
assess
and
refine
office…
Manage
alcohol
withdrawal
Interpret
pulmonary
function
testsand a…
3.4
Manage
COPD
Interpret
chest
x-rays
for
common
lung…
Construct
a
comprehensive
psychiatric…
Demonstrate
appropriate
use
of blood… for…
Initiate cost effective workup of venous…
Initiate
appropriate
antibiotic(s)
Manage
diabetes
Manage
hyperglycemia
Initiate
appropriate
antibiotic(s)
for…
Initiate
basal bolus
insulin
therapy and…
Demonstrate
respectful
professional…
Begin
initial
management
plan
for basic…
Diagnose
spinal
cord
compression
aand
transudative
from an…
RespondDifferentiate
to pages in timely
and courteous…
Determine
the
appropriate
initial…
3.2
Refer patients for hospice
and
palliative
care
Initiate
fall
precaution
in patients at…
Demonstrate proper ethics in orders
medical…
Identify causes of delrium
InitiateSelect
cost-effective
workup
for
anemia
Distinguish
between
cirrhotic
and non-…
Manage
patients
care
in
a
safe
manner.
type, dose, and duration of…
4
InitiateDiagnose
antibiotic
therapy
forloss
neutropenic…
Interpret
x-raysworkup
for common
lung…
the
cause of
of consciousness…
3
Initiate
costchest
effective
of venous…
Manage
chronic sinusitis
Manage
in the…
Distinguish
between hepatitic, cholestatic,…
Initiateparathyroid
work-up ofabnormalities
diabetic foot ulcer
Manage
benzodiazepines
patient with…
Initiateantibiotic
antibiotic(s)
for skin
soft tissue…
Initiate
CIWA protocolinina patients
at risk for…
Select
therapy
andand
duration
for…
2.8
pain
complaints overnight
Formulate
a cost-effective
approach to the…
Recognize
the need
for urgent/emergent…
Refer for Manage
right heart
catheterization
when…
3.5
Recognize delirium
andinidentify
potential…
Initiate
workup
for hemoptysis.
Explore
anddiagnostic
respond to
patient/family…
Demonstrate
proficiency
endotracheal…
2.6
Manage
hyperkalemia
in
the
renal
patient
Formulate
cost-effective
approach
to the…
Formulate
antibiotic
discharge
planning…
Initiate antibiotic(s) for pneumonia
Initiate workupEvaluate
and management
of
fever
Initiate
venous
thromboembolism
prophylaxis
Manage
peri-operative
hypertension
2.4
hypogonadism
Manage hyperkalemia during tumor lysis…
Gather subtle,
sensitive,
and complicated…
Recognize
and
manage
impending…
3
Manage derangements of potassium
Work-up lung nodule or mass
Interpret
serologic testing
for available
hepatitis to…
A,…
2.2
Describe
community
resources
Manage hyperkalemiaManage
during tumor
lysis…
hyperglycemia
Manage common sequelae of cirrhosis
Demonstrate empathy, compassion, and a…
Recommend acute
dialysis
for selected…
Manage
acute
renal failure
Manage derangements of potassium
2
Perform
pre-operative
risk…
2.5
Titrate insulin
based
on glucose
readings
Manage
extremes
of blood
pressure
Assess risk
factors for cardiac
acute hepatitis,…
Evaluate
memory
and…
1.8
Minimize
termscatheterization
during patient…
Referunfamiliar
for right heart
when…
Recognize
the riskloss/dementia
factors and clinical…
Manage
hyperuricemia
during
tumor
lysis…
Deliver
appropriate
goal-directed
therapy…
Refer
appropriate
patients
with
acute
and…
Manage hyperkalemia in the renal patient
Recognize
and
manage
conflicts
of
interest
Document
cross-cover
care
Communicate
effectively
with
the…
2
Recommend bronchoscopy for various…
Diagnose
the cause
of loss of
consciousness…
Identify
salient of
features
of fulminant…
Managethe
extremes
blood pressure
Develop
prioritized
differential
diagnoses
Initiate appropriate antibiotic(s) for skin and…
Identify
the
indications and…
Manage
peri-operative
NSTEMI
Initiate
antibiotics
pneumonia
Identify
how tofor
change
a ventilator to…
Was this resident receptive to the unique…
Manage a patient with perimenopausal…
Interpret
pulmonary
function
tests of renal…
Dose drugs appropriate to the level
Demonstrates respectful behavior to all…
Initiate
non-pharmacological
Initiate workup
forderangements
disorders of sodium
Manage
of electrolytes
Communicate
with primary caretreatments…
physicians
Manage
pancreatitis
Initiate
work-up
of
diabetic
foot
ulcer
Minimize
unfamiliar
terms during patient…
Describe the benefits and limitations…
Recommend
bronchoscopy
for various…
Acquire
accurate
relevant history
Titrate
cardiacand
medications
Order diagnostic
forand
neurologic…
Utilizetesting
oximetry
arterial blood gas…
Standard
of Workproficiency
Assessmentin the placement…
Demonstrate
Manage direct
thrombin
inhibitors
for…
Initiate
diagnostic
testing
for venous…
Demonstrate accurate medication…
Evaluate
functional abilities of geriatric…
Adjusta short
and long
actingfor…
narcotics for…
Autonomy
Assessment
Implement
diagnostic
work-up
Diagnose
neutropenic
fever
Perform
an accurate
physicalorders
exam on geriatric…
Write
initial admission
Differentiate
between
various
types
of…
Initiate workup of non neutropenic fever.
Provide
accurate,
complete,
and timely…
Interpret
basic
and
advanced
EKGs
obstructive lung…
Identify appropriate circumstances to use…
Refer patients forManage
hospiceexacerbations
and palliativeof
care
Overalland
Knowledge
Assessment
IntegrateEvaluate
clinical evidence
cause
of into
falls decision…
and suggest initial…
Provide accurate,
complete,
timely…
Identify
clinical
questions
as
they
emerge
in…
Initiate
antibiotics
for
pneumonia
Use
teach-back
method
withAssessment
patients…
when
it is appropriate
to intubate a…
Manage
arrythmias
Manage
acute
renal
failure
Coping
with
Complexity
UseIdentify
feedback
to
improve
performance
Develop
prioritized
differential
diagnosesand…
Assess
and
treat
pain
Identify
patients
whocare…
may benefit from…
Develop
anand
evidence-based
diagnostic
Modify
the
differential
diagnosis
and
Distinguish between
cirrhotic
non-…
Manage
elevated
blood
pressure
Identify
causes
ofManage
delrium
Recognize
the
scope
of
his/her
abilities
and…
Minimize
unnecessary
care
including
tests
Identify
the
appropriate
clinical
question
for…
Initiate
antibiotic(s)
for
pneumonia
acute chest syndrome in sickle…
Manage anticoagulation of venous…
Provide anticipatory guidance during…
Perform a musculoskeletal examination
Perform medication reconciliation on post…
Initiate evaluation for acute chest…
Use
feedback
to improve
Describe resources
available
to
hospice…
Manage
bowelperformance
disease
Manage
individuals
with rhinitis
Select
initialinflammatory
therapy
for patients…
Initiate end
of life discussions
with…
Evaluate
disorders
of
hemostasis
Demonstrate
pre-procedure
evaluation
for…
Perform
a
breast
exam.
Initiate
workup
for
polycystic
ovarian…
Interpretscreening
results offor
a graded
exercise…
Perform work-up of suspected lung… Determine
viral hepatitis.
Single PGY-1 - OPAs from Jul 1st to Feb 28th
Entire Residency – All OPAs from Jul 1st to Feb 28th
Trend line of all PGY-1 Milestone Assessments By Month n = 105721
Standard Deviation of all PGY-1 Milestones Assessments by Month
Validity?
• Validity – how well one can trust the results of a test as
interpreted for a specific purpose1
– Not a property of the instrument, but of interpretation
1Cook
DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and
application. Am J Med. 2006 Feb;119(2):166.e7-16. Review. PubMed PMID: 16443422
Validity?
• Content: do instrument items represent the construct?
• Response Process: are the construct and the thought processes of
observers the same?
• Reliability – internal consistency, temporal stability, inter-rater reliability,
generalizability
• Consequences: do scores really make a difference?
• Relations to other variables
1Cook
DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and
application. Am J Med. 2006 Feb;119(2):166.e7-16. Review. PubMed PMID: 16443422
R3 Class Milestone Average Jul 1st to Feb 1st
Selected
to be
Chief
Resident
Had issues
brought
before
competency
committee
1
4.07
1
4.42
1
4.39
2
4.05
2
4.28
2
4.29
3
3.91
3
4.27
3
4.28
4
3.89
4
4.14
4
4.24
5
3.79
5
4.13
5
4.24
6
3.77
6
4.13
6
4.20
7
3.77
7
4.09
7
4.15
8
3.71
8
4.08
8
4.13
9
3.60
9
4.02
9
4.10
10
3.57
10
3.98
10
4.10
11
3.55
11
3.87
11
3.99
12
3.49
12
3.87
12
3.91
13
3.44
13
3.84
13
3.90
14
3.40
14
3.81
14
3.88
15
3.32
15
3.71
15
3.77
16
3.32
16
3.64
16
3.76
17
3.27
17
3.64
17
3.71
18
3.26
18
3.60
18
3.69
19
3.19
19
3.49
19
3.55
20
3.07
20
3.42
20
3.45
21
3.05
21
3.35
21
3.43
22
3.05
22
3.24
22
3.33
23
3.00
23
3.08
23
3.29
24
2.91
24
3.02
24
3.10
Patient Care
Communication
Professionalism
2012
Name
A
B
C
D
E
F
G
H
I
J
K
L
M
N
O
P
Q
R
S
T
U
2012 ITE
8
4
10
11
21
6
14
7
18
9
12
1
22
2
19
13
17
5
15
16
20
2013
MK Milestones
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
17
18
19
20
21
22
Name
2013 ITE
MK Milestones
A
20
1
B
15
2
C
4
3
D
1
4
E
19
5
F
7
6
G
2
7
H
5
8
I
11
9
J
21
10
K
18
11
L
12
12
M
6
13
N
16
14
O
14
15
P
22
16
Q
23
17
R
8
18
S
9
19
T
13
20
U
17
21
V
10
22
W
3
23
Research Directions
• Study the progression of assessment entrustment decisions made
over time for internal medicine residents by faculty physicians, peer
physicians, and allied health providers
• Identify assessment biases present in these entrustment decisions,
including bias associated with age, sex, resident year of training,
attending experience level, or job description of assessor
• Determine validity of assessment entrustment decisions made over
time for internal medicine residents in relation to other variables
such as standardized test scores, and other clinical assessments
OPAs
Level Of Entrustment
12345
Rides in
Driveway
Reporting
Milestones
Narratives
Rides Bicycle
Safely
Falls off Bike
Rides 1000
feet without
Stopping
Rides on
Sidewalk
Rides on
Quiet Street
Rides in
Traffic
Rides to work
on time
Rides in
Triathlon
Level Of Entrustment
12345
Rides on
Quiet Road
Rides in Rush
Hour Traffic
Rides Tour de
France
OPAs
Level Of Entrustment
12345
Titrate cardiac
medications
Manage
pancreatitis
Reporting
Milestones/EPAs
Develops and achieves
comprehensive
management plan for
each patient. (PC2)
Titrate insulin based
on glucose readings
Adjust short and
long acting
narcotics for cancer
and sickle cell
patients
Manage parathyroid
abnormalities in the
renal patient
Deliver appropriate
goal-directed
therapy for severe
sepsis
Level Of Entrustment
12345
Narratives
Care plans are consistently
inappropriate or inaccurate
Inconsistently
develops an
appropriate care plan
Consistently develops
appropriate care plan
Appropriately modifies care
plans based on patient’s
clinical course, additional
data, and patient
preferences
Role models and
teaches complex and
patient-centered care
Observable Practice Activities (OPAs)
Content-Based (COPA)
Mapping
Reporting
Milestones
External Reporting
NAS report
Process-Based (POPA)
Mid/End of Rotation
Feedback (formative)
Entrustment (1-5)
Curricular
Milestones
Mapping
End-of-training
EPAs
Longitudinal Assessment
(summative)
Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C.
Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of
print] PubMed PMID: 24557518
TEAM EFFORT
Residency Program Staff
Dell
Diers
Warm
Lee
Mathis
Ashbrook
Fichtenbaum
Held
Mueller
O’Toole
Lee
Pai
Wood
Schauer
Tolentino
Wexler
Residency Education Coordinators
Khan
Robertson
Houk
Falciglia
Cohen
Chaudhary
Lenz
Schoch
Kamath
Discussion questions from the survey
• Is it possible to have a generic map that can
be used for both Medicine and Peds?
– You could design an excel solution which Eric
initially did before medhub figured out how to do
it for us
– Other ideas?
Discussion questions from the survey
• Specific directions for Med-Peds reporting
– Will be straightforward with the same forms
– ACGME sent me a copy of the reporting form
when I called them and asked them to; it is
literally just the 2 sets put together into 1 form
What about similar milestones?
• Some of this is subject to opinion, but about
8-10 are largely the same
• What do I mean?
• This is split into PC, MK and SBP as one slide
and PBLI, Prof, and ICS as second slide
• ALERT: COMPLICATED VISUALS!!!
PC, MK, SBP
IM
Gathers and synthesizes essential and
accurate information to define each
PC 1 patient’s clinical problem(s)
Develops and achieves comprehensive
PC 2 management plan for each patient
Manages patients with progressive
PC 3 responsibility and independence
Peds
PC 1
PC 2
PC 3
PC 4 Skill in performing procedures
Requests and provides consultative
PC 5 care
PC 4
MK 1 Clinical knowledge
Knowledge of diagnostic testing and
MK 2 procedures
Works effectively within an
SBP 1 interprofessional team
Recognizes system error and advocates
SBP 2 for system improvement
Identifies forces that impact the cost of
health care, and advocates for, and
SBP3 practices cost-effective care
Transitions patients effectively within
SBP 4 and across health delivery systems
MK 1
PC 5
Gather essential and accurate information about
the patient
Organize and prioritize responsibilities to provide
patient care that is safe, effective and efficient
Provide transfer of care that ensures seamless
transitions
Make informed diagnostic and therapeutic
decisions that result in optimal clinical judgment
Develop and carry out management plans
Locate, appraise, and assimilate evidence from
scientific studies related to their patients’ health
problems (EBM)
SBP 2
Coordinate patient care within the health care
system relevant to their clinical specialty
Advocate for quality patient care and optimal
patient care systems
SBP3
Work in inter-professional teams to enhance
patient safety and improve patient care quality
SBP 1
IM
Gathers and synthesizes essential and
accurate information to define each
PC 1 patient’s clinical problem(s)
Develops and achieves comprehensive
PC 2 management plan for each patient
Manages patients with progressive
PC 3 responsibility and independence
Peds
PC 1
PC 2
PC 3
PC 4 Skill in performing procedures
PC 4
PC 5 Requests and provides consultative care
PC 5
MK 1 Clinical knowledge
Knowledge of diagnostic testing and
MK 2 procedures
Works effectively within an
SBP 1 interprofessional team
Recognizes system error and advocates
SBP 2 for system improvement
Identifies forces that impact the cost of
health care, and advocates for, and
SBP3 practices cost-effective care
Transitions patients effectively within and
SBP 4 across health delivery systems
MK 1
Gather essential and accurate information about
the patient
Organize and prioritize responsibilities to provide
patient care that is safe, effective and efficient
Provide transfer of care that ensures seamless
transitions
Make informed diagnostic and therapeutic
decisions that result in optimal clinical judgment
Develop and carry out management plans
Locate, appraise, and assimilate evidence from
scientific studies related to their patients’ health
problems
SBP 2
Coordinate patient care within the health care
system relevant to their clinical specialty
Advocate for quality patient care and optimal
patient care systems
SBP3
Work in inter-professional teams to enhance
patient safety and improve patient care quality
SBP 1
IM
Gathers and synthesizes essential and
accurate information to define each
PC 1 patient’s clinical problem(s)
Develops and achieves comprehensive
PC 2 management plan for each patient
Knowledge of diagnostic testing and
MK 2 procedures
Works effectively within an
SBP 1 interprofessional team
Recognizes system error and advocates
SBP 2 for system improvement
Transitions patients effectively within
SBP 4 and across health delivery systems
Peds
PC 1
PC 5
Gather essential and accurate information about
the patient
SBP 2
Develop and carry out management plans
Make informed diagnostic and therapeutic
decisions that result in optimal clinical judgment
Work in inter-professional teams to enhance
patient safety and improve patient care quality
Advocate for quality patient care and optimal
patient care systems
PC 3
Provide transfer of care that ensures seamless
transitions
PC 4
SBP3
Peds
IM
PBLI 1
PBLI2
PBLI3
Monitors practice with a goal for
improvement
Learns and improves via performance
audit
ICS2
Learns and improves via feedback
Learns and improves at the point of
care
Has professional and respectful
interactions with patients, caregivers
and members of the interprofessional
team
Accepts responsibility and follows
through on tasks
Responds to each patient’s unique
characteristics and needs
Exhibits integrity and ethical behavior
in professional conduct
Communicates effectively with patients
and caregivers
Communicates effectively in
interprofessional
teams
ICS3
Appropriate utilization and completion
of health records
PBLI 4
Prof 1
Prof 2
Prof 3
Prof 4
ICS1
PBLI3
Identify strengths, deficiencies, and limits in one’s knowledge and
expertise
Identify and perform appropriate learning activities to guide personal and
professional Development
Systematically analyze practice using quality improvement methods, and
implement changes with the goal of practice improvement
PBLI 4
Incorporate formative evaluation feedback into daily practice
PBLI 1
PBLI2
Prof 1
Prof 2
Prof 3
Prof 4
Prof 5
Prof 6
ICS1
Humanism, compassion, integrity, and respect for others; based on the
characteristics of an empathetic practitioner
Professionalization: A sense of duty and accountability to patients,
society, and the Profession
Professional Conduct: High standards of ethical behavior which includes
maintaining appropriate professional boundaries
Self-awareness of one’s own knowledge, skill, and emotional limitations
that leads to appropriate help-seeking behaviors
Trustworthiness that makes colleagues feel secure when one is
responsible for the care of patients
The capacity to accept that ambiguity is part of clinical medicine and to
recognize the need for and to utilize appropriate resources in dealing with
uncertainty
Communicate effectively with patients, families, and the public, as
appropriate, across a broad range of socioeconomic and cultural
Backgrounds
Demonstrate the insight and understanding into emotion and human
response to emotion that allows one to appropriately develop and
manage human interactions
ICS2
Peds
IM
PBLI 1
PBLI2
PBLI3
Monitors practice with a goal for
improvement
Learns and improves via performance
audit
PBLI3
Identify strengths, deficiencies, and limits in one’s knowledge and
expertise
Identify and perform appropriate learning activities to guide personal
and professional Development
Systematically analyze practice using quality improvement methods,
and implement changes with the goal of practice improvement
PBLI 4
Incorporate formative evaluation feedback into daily practice
PBLI 1
PBLI2
ICS1
Learns and improves via feedback
Learns and improves at the point of
care
Has professional and respectful
interactions with patients, caregivers
and members of the interprofessional
team
Accepts responsibility and follows
through on tasks
Responds to each patient’s unique
characteristics and needs
Exhibits integrity and ethical behavior
in professional conduct
Communicates effectively with patients
and caregivers
ICS2
Communicates effectively in
interprofessional teams
Prof 6
ICS3
Appropriate utilization and completion
of health records
ICS1
PBLI 4
Prof 1
Prof 2
Prof 3
Prof 4
Prof 1
Prof 2
Prof 3
Prof 4
Prof 5
ICS2
Humanism, compassion, integrity, and respect for others; based on the
characteristics of an empathetic practitioner
A sense of duty and accountability to patients, society, and the
Profession
High standards of ethical behavior which includes maintaining
appropriate professional boundaries
Self-awareness of one’s own knowledge, skill, and emotional
limitations that leads to appropriate help-seeking behaviors
Trustworthiness that makes colleagues feel secure when one is
responsible for the care of patients
The capacity to accept that ambiguity is part of clinical medicine and
to recognize the need for and to utilize appropriate resources in
dealing with uncertainty
Communicate effectively with patients, families, and the public, as
appropriate, across a broad range of socioeconomic and cultural
Backgrounds
Demonstrate the insight and understanding into emotion and
human response to emotion that allows one to appropriately
develop and manage human interactions
Peds
IM
Learns and improves via
PBLI3 feedback
Exhibits integrity and ethical
behavior in professional
Prof 4 conduct
ICS1
Communicates effectively
with patients and caregivers
Incorporate formative evaluation feedback into daily
PBLI 4 practice
High standards of ethical behavior which includes
Prof 3 maintaining appropriate professional boundaries
Communicate effectively with patients, families, and
the public, as appropriate, across a broad range of
socioeconomic and cultural backgrounds
ICS1
Bottom line
•
•
•
•
22 IM milestones
21 Peds milestones
43 total
Of these, 9 have significant overlap, so there
are actually 43 minus 9 or 34 SEPARATE
milestones to report
Discussion questions from the survey
• What about overlapping milestones?
– Since the goal is to establish whether residents are
progressing and competent, is there a reason not
to use the benefit of evaluations from both sides
together?
Questions?