Transcript Document

Developing and Applying Tools:
Rubrics and Exam Questions
Linda Dickens, Ph.D
Dawn Zimmaro, Ph.D.
John Kucsera, M.A.
Agenda
1.
2.
3.
4.
Overview of SACS
Mapping Assessments to Program Outcomes
Writing Critical Thinking Exam Questions
Creating Rubrics
•Academic units
•Administrative units
Agenda
1.
2.
3.
4.
Overview of SACS
Mapping Assessments to Program Outcomes
Writing Critical Thinking Exam Questions
Creating Rubrics
Mapping Assessments to Program Outcomes
At what level should you collect data to measure your
outcomes?
• Classroom
• Program
Issues to consider
• What existing course assignments, exams, projects, etc.
can you use to measure your learning outcomes?
• What program level assessments can you identify or
develop?
• What is unique about your program that is measurable?
Classroom level
•
•
•
•
•
Exam
Assignment
Presentation
Paper
Project
Program level
•
•
•
•
•
•
Exit exam
Thesis/dissertation
Capstone course paper or project
Portfolio
Licensure/professional exam
Exit survey
English Department B.A.
Example learning outcome:
Students will be able to construct arguments and
communicate them effectively in writing.
Classroom level assessment
Read description of E 370W – Major Authors (Emerson)
Assignment
• Writing assignment that asks students to develop an
argument about an issue
– Emerson’s efforts to examine the authority and effects of
religious and educational institutions help frame
discussions about literature and education for subsequent
generations.
– Given the above statement develop a central thesis with
supporting arguments about how Emerson’s works have
helped to frame discussions about literature and
education.
Program level assessment
• A collection of individual student’s writing assignments
on constructing arguments taken from several English
courses
– All writing assignments are evaluated as a body of work
– Over the course of the whole degree program
Second learning outcome
Example learning outcome:
Students will be able to demonstrate knowledge of the
history and the structure of the English language.
Classroom level assessment
• Classroom exam
– Topics:
• History of English: Old to Early Modern English
• History of English: Modern and Future English
Topics should all relate to the learning objective
Program level assessment
• Exit exam
– Questions taken from all the courses that relate to the
learning outcome
Sample assessment map
• See handout
Discussion
• Have you tried this approach? What worked? What
didn’t?
• If you haven’t tried this, why not?
Agenda
1.
2.
3.
4.
Overview of SACS
Mapping Assessments to Program Outcomes
Writing Critical Thinking Exam Questions
Creating Rubrics
Writing Critical Thinking Exam Questions
•
•
•
•
•
•
Purpose of session
Learning Outcomes
Critical thinking exam questions
Objective and Subjective exam questions
Application activity
Final thoughts
Purpose
• Objective:
– To compose critical thinking questions for program or
classroom level exams
• WYMIWYG
• Aligning outcomes with methods
Learning Outcomes
• Program to classroom level (alignment)
• General statements to specific outcomes
– Person centered
– Clear, specific, and measurable
• Classroom learning outcomes should contain
– Procedure/method (assignments that will be provided)
– Behavior (what the students will do; usually the action verb)
– Criteria (how well a learner performs to be judged adequate and the
qualitative description of the criteria)
PhD Program Example
•
Program Educational Objective:
– Sixty percent of graduates from the College of Education will attain positions as
college faculty.
•
(Program) Learning Outcome 1:
– “Will prepare students as future researchers”
– Graduating students will demonstrate fundamental knowledge of quantitative,
qualitative, and mixed research designs.”
•
Procedure 1.1 (Classroom learning outcome):
– “Course EDP 382 will focus on the fundamental knowledge necessary to
conduct quantitative research.”
– “Given an exam administered in EDP 382, at least 80% of students will
demonstrate a fundamental knowledge of quantitative research designs by
achieving a score at or above an 80 out of 100. The exam will test the extent to
which students can compute common statistics, identify assumptions of
research designs, evaluate….
Writing Critical Thinking
Exam Questions
• For critical thinking:
– Focus on higher levels of Bloom’s taxonomy
Bloom’s taxonomy for the cognitive domain
• Remember
– recall or memorize information
• Understand
– translate from one form to another (e.g., restate in own words)
• Apply
– apply knowledge to a novel situation
• Analyze
– examine a concept and break it down into parts
• Evaluate
– make judgments using standards of appraisal
• Create
– put information together in an unique or novel way
Writing Critical Thinking
Exam Questions
• For critical thinking:
– Focus on higher levels of Bloom’s taxonomy
• Behavior component of learning outcome
– Ex: Given an exam administered in in EDP 382, at least 80%
of students will demonstrate a fundamental knowledge…
– Action verbs (handout)
Writing Critical Thinking
Exam Questions
• For critical thinking:
– Focus on higher levels of Bloom’s taxonomy
• Behavior component of learning outcome
– Ex: Given an exam administered in __ course, at least 80% of
students will demonstrate a fundamental knowledge…
– Action verbs (handout)
• Outcome to exam questions (and instruction)
– Can use the action verb
» Ex: Compute the standard deviation of….
– Test blueprint (handout)
Objective and Subjective
Exam Questions
•
•
•
•
•
Objective questions
– Select a correct response from several alternatives
Subjective questions
– Present an original answer
Strengths/weaknesses of each
Assumptions (True/ False/ or Undetermined)
1. Essay exams require more thorough student preparation and
study time than objective exams.
2. Essay and objective exams can be used to measure the same
content or ability.
Examples of objective, critical thinking exam questions (packet)
Application Activity
•
In groups or pairs and with using the Emerson content
from Dawn’s previous presentation,
1. Construct one objective and one subjective exam
question.
2. Explain why you believe your questions are assessing
critical thinking.
Application Activity
Objective examples:
• (Analysis) Emerson might be characterized as any of the
following EXCEPT:
(A) a transcendentalist
(B) an American Renaissance writer
(C) a public speaker
(D) a political prophet
(E) a literary critic
• (Evaluate) Determine whether the following statement is true or
false:
– Melville and Hawthorne were both disillusioned
transcendentalists.
Final thoughts
•
•
•
At the program/course level, what do you want your
students to learn/develop (outcomes)?
Are your exams (methods) matching these objectives?
Use multiple assessment methods
– Ex: Lower stakes testing
•
Can use rubrics for subjective exam questions
Agenda
1.
2.
3.
4.
Overview of SACS
Mapping Assessments to Program Outcomes
Writing Critical Thinking Exam Questions
Creating Rubrics
Creating Rubrics
1. What is a rubric?
2. When should you use a rubric?
3. What are the elements of a rubric?
4. How do you create a rubric?
What is a rubric?
-A scoring guideline that measures student
achievement systematically:
– specifies instructor expectations
– lists performance criteria
– describes levels of quality
When should you use a rubric?
• On assignments that require subjective grading
– Places structure on a subjective process
• Essays
• Presentations
• Multi-media
What are the elements of a rubric?
I.
Descriptors and criteria
that identify performance
expectation for each
point on the scale
What the
student needs
to do
Example:
Include a compelling, descriptive, and persuasive
conclusion that summarizes arguments set forth in
paper
II.
Standards that identify
the range of quality or
performance levels
How well the
student does it
Example:
--no or poor conclusion or summary of argument
--some summary of points made, but nothing beyond
summary; no broad conclusions/lessons
--a conclusion going beyond summary of what was written in
the body of the essay
III.
A scale of points
awarded to specific
elements of student
performance
Example:
5 = outstanding
3 = acceptable
1 = poor
What score the
student will
receive
Critical Thinking Rubric
1
A.
B.
C.
[2
Is unable to analyze
information, questions, and
problems or does so
superficially
Is unable to or infrequently
uses inference to reason
from clearly stated premises
or recognize implications and
consequences
Is unable to or infrequently
uses deductive and
inductive reasoning and
problem-solving skills
3
4]
5
A.
Clearly and precisely
analyzes key information,
questions, and problems
B.
Uses inference to reason
carefully from clearly stated
premises to important
implications and
consequences
C.
Uses deductive and
inductive reasoning and
problem-solving skills
consistently and with ease
•Analytic Rubrics
•Holistic Rubrics
How do you create a rubric?
• Develop assignment
• Identify learning objectives
• Decide which criteria to evaluate
• Decide what kind of rubric to use
• Define performance indicators
• Identify standards of performance
• Decide on grading or scoring scale
• Review rubric to ensure it measures what you intend
Determine key criteria
If the assignment
objectives are …
Creative products
Writing
Oral presentation
… then consider these criteria
•“Wow” factor, novelty
•Technical quality
•Adherence to conventions of the field
•Grammar/mechanics
•Development of ideas
•Clarity
•Voice projection
•Body language
•Organization
Identify Indicators
Concrete sign or evidence of a criterion being
met—not meant to be exhaustive
Example:
Learning objective: Students will be able to write in
an engaging and persuasive manner
Criterion: Student clearly and precisely analyzes
key information, questions, and problems
Indicator:
Identifies main tension in story and relates it
to sub-plots
Any Questions?