Transcript Keys to Quality Assessment Items - Clare

Jurying for
Quality Assessment Items
Clare-Gladwin RESD
August 8, 2007
Today’s Learning Targets
You will:
 Refresh your knowledge and understanding
of designing high quality assessments.
 Understand the step by step process for
“jurying” created assessment items.
 Practice the jurying process, preparing you
to facilitate the work with colleagues.
We’ve All Been There!
Best Assessment
 Targets are clear to ALL.
 Use of the assessment is
understood by ALL.
 Assessment method is
appropriate.
 Achievement is
adequately sampled.
 Assessment accurately
measures learning.
Worst Assessment
Alignment Relationships
Intersection of what is taught with what is tested
Assessment
Taught,
tested,
and in
the
standards
Instruction
Standards
Intersection
of what is
tested with
what is in
the
standards
Intersection of what is taught with what is in standards
Smithson, J.L., Using Alignment to Analyze Standards, Assessments, & Classroom Practice, Mega-SCASS, Orlando, FL, Jan, 2005.
Revisiting
Student Learning Targets
 Knowledge
 Reasoning
 Performance Skills
 Products
Knowledge Targets
Mastery of subject
content where mastery
includes both
knowing and
understanding
Examples of Knowledge
 Identify…
 Read and write exponential equations.
 Describe the …
 Know the life cycle of …
Reasoning Targets
The ability to use
knowledge and
understanding for
further learning and
problem solving
Examples of Reasoning
 Examine and interpret data or results to draw
conclusions or make further decisions.
 Distinguish between...
 Make a prediction based on information read in a
text.
 Justify…
Performance and Product Targets
Performance
Product
…proficiency where the
process is most
important.
…present concrete
evidence of academic
proficiency.
 Read aloud with fluency
 Construct a bar graph.
and expression.
 Dribble to keep the ball
away from an opponent.
 Participate in discussion.
 Write a personal narrative.
 Develop a personal health-
related fitness plan.
Alignment Relationships
Intersection of what is taught with what is tested.
Assessment
Taught,
tested,
and in
the
standards
Instruction
Standards
Intersection
of what is
tested with
what is in
the
standards
Intersection of what is taught with what is in standards.
Smithson, J.L., Using Alignment to Analyze Standards, Assessments, & Classroom Practice, Mega-SCASS, Orlando, FL, Jan, 2005.
Collaboratively Revisit
Types of Learning Targets
Revisiting
Unpacking the Standards
1. As a team, identify the verb(s) in the
standards.
 First define as educators.
 Consistency by content and grade level is
critical.
Next steps: Define the verb for students in
student-friendly language.
Revisiting
Unpacking the Standards
2. Repeat the process for content and skills
in the standard.
 First define as educators.
 Consistency by content and grade level is
critical.
Next steps: Define the content and skills for
students in student-friendly language.
Revisiting
Unpacking the Standards
3. Share the “unpacked” verbs, content, and
skills with grade span colleagues.
 Review and revise as part of the K – 12
alignment process.
 Grade level or content area consensus
 K – 12 alignment
 Instruction and assessment implications
Collaboratively Revisit
Unpacking the Standards
Assessment Audit
Step 1
Step 2
Step 3
Step 4
Step 5
Assessment Method
Design Considerations
Sampling Issues
Depth of Knowledge
Alignment/Validity Criteria
–
–
–
–
–
–
–
Content Matched
Performance Matched
No alternative approaches which do not align with the content
expectation.
Not solvable using only test wise ness
Focuses on only one content expectation
Foils appropriate and insightful
Context is appropriate and engaging.
Step 1
Target-Method Match Activity
Selected
Response
& Short
Answer
Essay
Performance
Assessment
Personal
Communication
Knowledge
X
X
No
X
limited
Reasoning
X
limited
X
X
strong
X
Skills
X
limited
X
writing
X
strong
X
Products
X
very
limited
X
writing
X
strong
X
limited
Disposition
Collaborative Work Time
Step 1
Assessment
Method
Step 2: Design Tips for Multiple
Choice Questions (Source: D. Wahlstrom)
Directions
Stem
Foils
Step 2: Design Tips for Writing
Stems for MC Items (Source: D. Wahlstrom)
Step 2: Design Tips for Writing
Responses for MC Items
(Source: D. Wahlstrom)
Step 2: Design Tips for Writing
Constructed Response Prompts
(Source: D. Wahlstrom)
Step 2: General Assessment
Design Guidelines







Align with learning targets.
Include directions.
Use complete sentences when designing items.
Present new situations. Do not use items
directly from your text.
Highlight key words.
Monitor the reading level of the test.
Do not use abbreviations.
Step 2: General Assessment
Design Guidelines







Use support materials such as rulers, formula
sheets, etc.
Use correct grammar.
Be consistent in presentation
Keep all parts of test question on one page.
Avoid crowding.
Group the questions on the test by type.
Use a style guide to ensure that style is uniform.
Collaborative Work Time
Step 2
Design
Considerations
Step 3: Sampling Issues
Are there enough questions here to get the
information needed about student mastery?
Will 82% on this assessment tell us what the
student knows or does not know? Will the
results guide what needs to happen next
with student learning?
Collaborative Work Time
Step 3
Sampling Issues
Step 4: Depth of Knowledge
1. Recall
2. Skills and Concepts
3. Strategic Thinking
4. Extended Thinking
Can be applied to Standards (Content Expectations),
AND to Assessment Items
(Dr. Norman Webb, University of Wisconsin, 1997, 2002)
Samples from Kentucky Dept. of Education
Collaborative Work Time
Step 4
Depth of Knowledge
Step 5:
Alignment/Validity of Items
 Content Match
 Performance Match
 Allows no alternative approaches
 Test Wise ness
 Connects to one content expectation
 Foils will provide insight and information
 Context is engaging and appropriate
Content and
Performance Matched
No Alternative Approaches
Minimize Test Wise ness
Engaging and Appropriate
Context
Collaborative Jury of Items
Step 5
Alignment Criteria
•
•
•
•
•
•
•
Content Matched.
Performance Matched.
No alternative
approaches which do not
align with the content
expectation.
Not solvable using only
test wise ness.
Focuses on only one
content expectation
Foils appropriate and
insightful.
Context is appropriate
and engaging.
We’ve All Been There!
Best Assessment
 Targets are clear to ALL.
 Use of the assessment is
understood by ALL.
 Assessment method is
appropriate.
 Achievement is
adequately sampled.
 Assessment accurately
measures learning.
Worst Assessment
Collaborative Work Time
Step 1 Assessment
Method
Step 2 Design
Considerations
Step 3 Sampling
Issues
Step 4 Depth of
Knowledge
Step 5
Alignment/Validity
Criteria
Successful Implementation
 Accurate assessments, both summative and
formative
 Descriptive feedback
 Student-involved record keeping and
communication
Next Steps
 Has communication, among all stake
holders, been planned as part of this
assessment process?
 Has discussion begun about how the results
will inform teaching and learning?
 How will students be actively involved in
this assessment process?
I taught Fido
how to whistle
I don’t hear
him whistling
I said I taught him- - I
didn’t say he learned it!