Transcript Designing Grading Rubrics to Assess Student Learning: The DIY

Designing Grading
Rubrics to Assess Student
Learning
The DIY (Do It Yourself) Workshop
Mary T. Blackinton PT, MS, EdD
Director Transition DPT Program
Chair, PT Outcomes Committee
Nova Southeastern University
Agenda

Introduction to Grading



Analyzing Grading Rubrics


Small group discussion
Developing Rubric



Myths & Principles of Effective Grading
Choosing an Assignment for a Rubric
Definitions
Primary Trait Analysis
Creating & Using Rubrics


Design
Pitfalls
Dr. Mary Blackinton, 01-19-08
Objectives

Upon completion of the workshop, faculty will be able to:

Describe the role of grading rubrics in:








Clarifying teacher expectations
Evaluating student learning
Providing detailed feedback
Improving instruction
Performing outcome assessment
Evaluate the strengths and weaknesses of grading rubrics as a
student assessment tool in health professions education
Integrate the principles of primary trait analysis (PTA) into
designing a rubric for an assignment or performance skill
Create a grading rubric for an assignment, project, or exam
Dr. Mary Blackinton, 01-19-08
Grading Challenge # 214:
Student’s Perceptions
Dr. Mary Blackinton, 01-19-08
Why do we Grade?

Grading1

The process used by faculty to assess student
learning via assignments and exams, including:
 Relating
test items or assignments to learning objectives
 Establishing criteria/standards (SAFETY)
 Helping students acquire needed knowledge/skills
 Facilitating student motivation
 Giving feedback about performance
 Communicating about what has been learned
 Using results to influence teaching and curriculum
1
BE Walvoord, VJ Anderson (1998). Effective grading: A tool for
learning and assessment. Jossey-Bass: San Francisco
Dr. Mary Blackinton, 01-19-08
Student Assessment in Health Professions

Faculty charged to assess competency in patient care



Professionals integrate 3 domains of learning




Responsibility to protect public
Other stakeholders: accrediting bodies, profession, students, parents…
Cognitive-thinking/reasoning
Affective-feeling, valuing, ethics
Psychomotor-doing, manipulating, performing, administering
Assessment strategies reflect demands professional ed


Must include more than multiple choice assessment to determine learning
Often have multiple faculty grading student performance
Dr. Mary Blackinton, 01-19-08
Myths About Grading

Grading:








Is a necessary EVIL
Must be objective
Should be easy
Should not take detract from teaching
Use knowledge of experts to evaluate novices
Is assessment only, doesn’t impact learning
With clear expectations means you are “spoon-feeding”
Cannot be used in program assessment
Dr. Mary Blackinton, 01-19-08
Rubrics as Assessment of Program
Outcomes
Learning
Outcome
Direct Measure
Assessment
Method
Students will be able
to select valid/
reliable
tests/measures to
assess patient
outcomes
Assignment 2,
Neuromuscular I:
Given case, students
select tests/measures
& provide rationale
based on patient
needs & psychometric properties
Rubric identifies
ability to ascertain
validity & reliability
of tests and
measures by reading
relevant literature,
and applying to
specific needs of a
patient (see sample)
Dr. Mary Blackinton, 01-19-08
Principles of Effective
Appreciate the complexity of grading
Substitute judgment for objectivity
Distribute time effectively
Be open to change
Listen & observe
Communicate & collaborate with students
Integrate grading with other key processes
Seize the teachable moment (feedback)
Make student learning the primary goal
Be a teacher FIRST, gatekeeper LAST
Encourage learning-centered motivation
Emphasize student involvement
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
1
Walvoord &Anderson (1998)
Dr. Mary Blackinton, 01-19-08
1
Grading
Types of Grading in Health Professions
Student Assessment Methods





StudentConstructed
Responses







2LC
Multiple choice examination
Short answer questions on tests
Essay questions
Annotated bibliographies
Literature reviews
Case Studies
Oral Examinations
Practical/Performance Examinations
Clinical Assessments
Journals
Portfolios
Lab Conclusions
Jacobs, CI Chase (1992). Developing and using tests effectively. Jossey-Bass: San Francisco
Dr. Mary Blackinton, 01-19-08
Methods of Grading Student-Constructed
Responses
Method
Advantage
Norm Referenced:
categorize work into
A, B, C, D
Little work up front,
recognizes faculty as
‘expert’, flexible
Checklist: list of
criteria to include
(introduction,
research question…)
Assignment directions
match checklist, not
difficult to prepare
Disadvantage
-Potential for bias
-Less opportunity for
learning, vague
-Usually lacks
descriptions
-Lists + traits or
behaviors, no negative
Grading Rubric:
Explicit expectations,
-Takes time to
criterion referenced, better feedback, greater construct, may need to
describe performance inter-rater reliability,
modify after 1st run
expectations &
links to performance
weighting
Dr. Mary Blackinton, 01-19-08
What are Assessment Rubrics?
Small Group Discussion

In small group, look at the sample rubric packet.
Then, answer the following:
Discussion Questions
Which rubrics seem the MOST clear to you? Why?
 Which rubrics are not clear? Why?
 What was common among the rubrics you viewed as
clear?
 Based on samples, how do you define assessment
rubrics?

Dr. Mary Blackinton, 01-19-08
What is a Grading Rubric?



Method of “articulating expectations for an assignment by
listing the criteria, or what counts, & describing levels of
quality from excellent to poor”3
Type of assessment that specifies gradations of quality
from excellent to poor4
A criterion-referenced method of grading using highly
specific grading criteria that are linked to objectives
3
HG Andrade, Y Du (2005). Students perspectives on rubric-referenced assessment. Practical
Assessment, Research & Evaluation, Vol 10 (3).
4
HG Andrade ((2005). Teaching with rubrics: The good, the bad, and the ugly. Coll Teaching, 53 (1)
Dr. Mary Blackinton, 01-19-08
What is a Grading Rubric?
Gradation: excellent-poor
Categories important to
the teacher/class
Performance Performance Performance
Level 3
Level 2
Level 1
Dimension 1
3 pts: describe
2 pts
1 pt
Dimension 2
4 pts
2.5 pts
1 pt
Dimension 3
3 pts
2 pts
1 pt
Dimension 4
2 pts
1 pts
0 pts
Weighted Points
Dimensions also called criteria
Dr. Mary Blackinton, 01-19-08
Criteria 1
Criteria 2
Criteria 3
Level 1
3 pts: describe
2 pts
1 pt
Level 2
4 pts
2.5 pts
1 pt
Level 3
3 pts
2 pts
1 pt
Level 4
2 pts
1 pts
0 pts
Dr. Mary Blackinton, 01-19-08
How do Students Perceive Rubrics?

Andrad & Yu, 2005

Investigated how students use grading rubrics


Students reported that they used rubrics to





Focus group / qualitative design
Help them determine faculty expectations
To plan an approach to assignment
Check/revise work before handing in
Help reflect on their learning-see strengths/weaknesses clearly
Perceived results of rubric use



Better, fairer grades
Improvements in quality across classes
Less anxiety
Dr. Mary Blackinton, 01-19-08
Using Primary Trait Analysis1,4
To Develop a Grading Rubric
Rationale

Analyze traits / characteristics of student learning
and then clearly articulate them, leading to:
Assignment specific
 Explicit criteria
 Adds objectivity & consistency to holistic scoring
 Lets students know in ADVANCE how will be graded

1 Walvoord
&Anderson (1998)
4R
Lloyd-Jones (1977). Primary Trait Scoring in C. Cooper & L. Odell (Eds), Evaluating Writing:
Describing, measuring, judging. Urbana IL: National Council of Teachers
Dr. Mary Blackinton, 01-19-08
Constructing a Primary Trait Analysis Scale1
1.
2.
Choose assignment/test that matches course objective
Identify all traits that will count for scoring in assessment
1.
3.
Build scale for scoring performance, gradations of skill
1.
2.
3.
4.
5.
4.
Body Language vs Thoroughness vs Accuracy
Scale usually ranges from 2-5 points
Include what should be demonstrated and what should be avoided
Build a range that discriminates A from A- from B+
Run scale by colleague, graduate, teaching assistant
Weight items: content > spelling; accuracy > efficiency, etc
Evaluate performance against criteria
1.
Try scale with sample & revise as needed
Dr. Mary Blackinton, 01-19-08
Example: Designing a Rubric

Select assignment appropriate for PTA


Taking a Patient History (clinical performance)
Identify ALL traits you look for in patient history
Body language & eye contact (X-1)
 Thoroughness (X-2)
 Data gathering (X 1.5)


Weight them each trait
Dr. Mary Blackinton, 01-19-08
Building a Scale: Body Language

Level 3-Excellent






Level 2-Good



At least 4 of the above behaviors noted
OR, all 5 qualities demonstrated but not consistently
Gives students
specific
behaviors to
demonstrate!
Identifies
behaviors to avoid!
Level 1-Fair


Consistently had good eye contact with patient
Communicated at patient’s eye-level
Maintained an open posture, leaning toward patient
Facial expression interested, non-judgmental
Confident and relaxed, not arrogant
Inconsistent in several (2-3) areas, for example, confident through some
portions of history but not others; or, eye contact missing at end
Level 0-Poor

Arrogance, avoidance, lack of confidence detracted from relationship
Dr. Mary Blackinton, 01-19-08
Building a Scale: Thoroughness

4: Extremely Thorough


3: Thorough


All items (1-9) included but may have missed some sub-categories (ie
missed 2 body systems); or, only missed 1-2 of the 9 core areas above
2: Somewhat Thorough


History includes all the following core areas: 1. all body systems; 2. history
current illness/problem; 3. family history; 4. medications; 5. educational
background / learning style; 6. lifestyle (diet, exercise, habits); 7. living/
work/social environment; 8. assistive/adaptive devices (glasses, cane, etc); 9.
prior level of function
Missing 3-4 of the 9 core areas, or, missing only 1-2 areas but was also
superficial in review of body systems
1: Incomplete

Less than half of the 9 core areas were covered; incomplete data
Dr. Mary Blackinton, 01-19-08
Building a Scale: Data Gathering

Excellent-4


Good-3


Demonstrates at least 4 of 6 skills noted above throughout the history, or,
used all 5 skills but inconsistently.
Fair-2


1. Uses open-ended, broad questions; 2. encourages patient to tell story
chronologically; 3. allows patient to talk without interrupting; 4. encourages
by using phrases such as ‘tell me more’ or ‘what else?’; 5. summarizes what
was heard; 6. Approach was organized
Demonstrates 3 of the core probing skills, or, 3-4 skills used however did
so inconsistently; multiple ‘missed’ opportunities to probe; disorganized at
times
Poor-1

Did not consistently demonstrate probing skills, questions were narrow and
rarely open-ended, interrupted or cut patient off, rarely encouraged more
information with phrases such as ‘tell me more’.
Dr. Mary Blackinton, 01-19-08
DIY: Build Your Own PTA
Assignment or Test:____________________________________________________
Related Class Objectives: ________________________________________________
Traits
Gradations
Trait 1:
Trait 2:
Excellent
Good
Fair
Poor
Dr. Mary Blackinton, 01-19-08
Trait 3:
Step 4: Evaluate Performance
Using Criteria

Test the Scale


Use it with an past paper, videotape, case report, etc.
What changes are needed in the rubric?
Were the criteria easy to follow?
 Did the overall grade reflect performance?


Are any changes needed in description or in the
point allotment?


Did an A performance score better than a B?
Is it easy? Hard? Too cumbersome?
Dr. Mary Blackinton, 01-19-08
Applying the Rubric
Distribute ( or post) rubric to students in advance


Have conversations about expectations
Ask students to attach rubric to assignment


Helps students pay attention!
Use rubric to grade


Match written comments to phrases in rubric
Revise after use


Make changes soon after grading for next time
Answer Curriculum Questions




Did the students learn? To what degree were objectives met?
If not, was it the teaching? The assignment? Background skills?
What, if any, changes should be made in the class or curriculum? Rubric?
Dr. Mary Blackinton, 01-19-08
Advantages of Grading









1
Rubrics
Save time in grading process
Makes process of grading reliable/fair
Clarifies expectations for students
Reinforces key concepts - help faculty relate to objectives
Students are participants as expectations are known
Student peer review
Works well if team-teaching
Share across courses or over curriculum
Basis for departmental/program assessment
Which of these advantages pertain to your teaching in the health
professions?
1 Walvoord & Anderson, 1998
Dr. Mary Blackinton, 01-19-08
CAUTION: Common Rubric Pitfalls

Rubric does not correspond with class or program outcomes


Scale does not have enough gradations or levels


Grammar = Content
Too broad, not enough content described



Not distinguishing the A’s from the B+’s
All traits are given equal weight regardless of complexity


Example: Entire rubric focused on writing quality not content
Words like ‘breadth’ and ‘depth’ used in lieu of specifics
Students still not sure, other grading faculty still not clear
Too long/too complicated

Faculty + students get lost in the rubric
Dr. Mary Blackinton, 01-19-08
Classroom Assessment Technique

Write down ONE thing you will commit to incorporate
regarding the use of grading rubrics

Write down ONE unanswered question
Contact: [email protected]
Dr. Mary Blackinton, 01-19-08