Transcript Portfolio Assessment in Clerkship

Portfolio Assessment in
Clerkship
Michelle Gibson Geriatrics
(thanks to Chris Frank and Melissa
Andrew too)
Objective

Participants will discuss the use of
portfolios in assessing medical trainees
(after a brief review of some basic
principles…)
Background

Change to the Internal Medicine clerkship
in January 2007


4 weeks to 6 weeks
Opportune time to review the assessment
process in geriatrics

Previously, 8 assessment cards filled out by
MD supervisors
Challenges with old system
Cards were often filled out the last week
of the rotation
 Depended solely on supervisor
impression
 No direct observation was incorporated
 Students were assessed using only one
modality

Reasons for change
Desire for more formative assessment
opportunities
 Desire for more robust assessment
 (Need to meet course requirements!)

Formative assessment
Assessment that occurs during instruction
to provide feedback to teachers and
students
 Describes needs for future learning
 Assessment for learning

Summative Assessment
Assessment that occurs at the end of an
instructional unit to document student
learning
 Certifies learning
 Assessment of learning


Is this portfolio formative or summative?
High Quality Assessment








Clear & appropriate learning targets
Appropriateness of assessment methods*
Validity
Reliability
Fairness
Positive consequences*
Alignment*
Practicality & efficiency*
Reliability & Validity
Reliability: concerned with the
consistency, stability, and dependability of
the scores
 Validity: the appropriateness of the
inferences, uses, and consequences of
the assessment


Is it assessing what you want it to assess?
Portfolio
A systematic collection of student
products to assess progress
 Structured portfolio


The assessor determines the mandatory
content of the portfolio
Literature review

Assessment of housestaff in clinical
rotations
Portfolio literature reviewed
 Specific instruments examined


Portfolio components were selected to
provide multiple methods of assessment
for each clerkship objective
Components by Methodology
Methodology
Portfolio components
Foundational evaluations
Assessment cards - 6
Direct observations
Mini-CEX - 2
Practice- & Data-based
learning
Multi-source evaluations
Medication Review
Self-assessment of case
write-up
TAB - 2
Miscellaneous (optional)
Student’s choice
Mini-CEX
TAB
Card
Medication Review - written

Based on an actual patient, preferably
one for whom the student has
responsibility
brief summary of patient,
 meds he/she is taking including whether or
not they are appropriate
 suggestion of other medications he/she
should be taking

Self-assessment of write-up
H & P from early in rotation
 Students asked to read and identify

2 things done well
 2 things that could be improved


Reviewed with undergrad coordinator on
the last day
Components & CPE Categories
CPE Category
EC
MC TAB
Knowledge Base
X
X
Motivation to Learn
X
X
Team Functioning
X
Communication
X
X
X
Relationship with patients/families
X
X
X
Resource Management
X
X
MR
Case
X
X
X
X
X
X
X
X
X
X
Process
Orientation with students on Day 1
 Team, faculty, & residents are oriented to
the portfolio
 Students meet with undergrad coordinator
at mid-term, review progress, identify any
gaps, etc.
 Last day - review portfolio with students,
completion of CPE

Evaluation of Pilot

Focus on acceptability

Students and assessors
Student evaluations ongoing - not seen by
the undergrad coordinator until collated in
April 2008
 Will redo faculty portion in December
2007

Challenges noted - 1st time

First rotation ever for clerks
Time spent learning how to be a clerk
 Couldn’t assemble half the portfolio in 3
weeks

Exam stress “detracted from our learning”
 Supervisors were absent more than usual

*all except exam stress have resolved
Strengths noted
Multidisciplinary team members provided
high-quality feedback
 Direct observation occurred
 Forms were felt to be user-friendly,
acceptable
 Students appreciated discussing their
medication review and being observed

Assessor ratings of forms

TAB


5/5 for ease of use; 5/5 for usefulness
Mini-CEX
5/5 for ease of use; 4/5 for usefulness
 Time spent by assessor: 13.5 minutes


Encounter card
4.5/5 for ease of use; 4.5/5 for usefulness
 Time spent by assessor: 15 minutes

My time?

5-10 minutes on day 1 for orientation


Plan to develop “talking power point”
15 min per student at the end of week 3
Mid-term review
 Could be any faculty


15 min per student on the last day
Final evaluation
 Could be any faculty

My *impressions*
Students receive more feedback in a
timely fashion
 Direct observation doesn’t take long, and
is very valuable
 TAB form is very useful, and team
members are pleased to be filling them
out
 Mid-term meetings are very useful

Future directions
Continue with student and assessor
feedback
 Incorporate other tools?

Peer assessment?
 Patient assessment?

Study it more
 Implement it for residents

Factors to note
Small unit, discrete group of nursing and
other staff
 Small division
 Very workable for us … but for others?

Questions to consider
Is this high quality assessment? How
could it be improved?
 Can it be extended into other areas?
 Is there a role for a “whole clerkship”
portfolio? “Pre-clerkship” portfolio?
 Residents & non-MDs as assessors?
 Electronic?

Comments about our students
“ .. Is very attentive… he listens well to pt.
& staff, & then goes off & evaluates what
he’s been told”
 “excellent verbal communication observed
with all team members”
 “better H & P than most residents”
