Transcript Alignment Studies: Powerful Tool for Focusing Instruction

TILSA Alignment Tool
Dissemination Workshop
July 25 and 26, 2005
WYNDHAM Hotel
Boston, Massachusetts
Funded by the U.S. Department of Education through a
contract to the state of Oklahoma and subcontracts to
CCSSO, WCER, HumRRO, and Tindal.
Alignment
Powerful Tool for Focusing
Instruction, Curricula, and
Assessment
Agenda
July 25, 2005
8:30 to Noon
Speakers:
Norman L. Webb
Lauress L. Wise
Gerald Tindal
Noon Lunch
1:00 to 5:00 PM
Concurrent sessions
Session A: Using the WAT
Session B: Interpreting reports and
coordinating an alignment study
Agenda
July 26, 2005
8:30 to 12:30 PM
Concurrent sessions
Session A: Using the WAT
Session B: Interpreting reports and coordinating
an alignment study
12:30 Lunch
1:30 to 3:00
Plenary
Technical issues with the CD alignment system
General questions and closing
Alignment Issues
Vertical Alignment
Grade to grade content linkages
Lauress Wise
Alternate Assessment Alignment
Operationalize the process
Gerald Tindal
Webb Alignment Process
Norman Webb, Rob Ely, Meredith Alt, &
Brian Vesperman
Workshop Expectations
Set up of an alignment study
Responsibilities of a group leader
Responsibilities of reviewers
Coding procedures
Special features
How to conduct an alignment analysis of
standards and assessments with the WAT
Alignment
The degree to which expectations
and assessments are in agreement
and serve in conjunction with one
another to guide the system toward
students learning what is expected.
Standards
Curriculum
Assessment
Degree of Alignment
Standards
Standards
Assessment
Assessment
Standards
Assessment
Standards
Assessment
Items
Alignment Process
• Identify Standards and Assessments
• Select 6-8 Reviewers (Content Experts)
• Train Reviewers on DOK Levels
• Part I: Code DOK Levels of the
•
Standards/Objectives
Part II: Code DOK Levels and Corresponding
Objectives of Assessment Items
Specific Criteria
Content Focus
A. Categorical Concurrence
B. Depth-of-Knowledge Consistency
C. Range-of-Knowledge Correspondence
D. Balance of Representation and
Source of Challenge
Depth of Knowledge
Level 1 Recall
Recall of a fact, information, or procedure.
Level 2 Skill/Concept
Use information or conceptual knowledge, two
or more steps, etc.
Level 3 Strategic Thinking
Requires reasoning, developing plan or a
sequence of steps, some complexity, more
than one possible answer.
Level 4 Extended Thinking
Requires an investigation, time to think and
process multiple conditions of the problem.
Items by Objectives Report
Low
Medium
High
0
5.20202
48
root
I
1
7
8
1
9
3
8
4
5
1.1.
2
2
2
2
7
7
7
7
7
7
7
1
9
1
9
3
8
3
8
3
8
3
8
3
8
3
8
3
8
4
5
4
5
1.2.
1
9
1
9
1
9
2
9
3
2
3
2
3
2
3
2
3
2
4
0
1.3.
2
9
3
6
3
8
4
5
1.4.
2
5
3
5
3
5
3
5
3
5
4
1
1.5.
2
2
2
2
7
8
8
1
0
1
0
2
5
8
8
8
1
0
1
0
4
3
4
3
4
5
4
5
4
5
4
5
2
5
2
5
2
5
2
9
3
6
4
1
1
0
4
5
1
9
1
9
1
9
What alignment is good enough?
Alignment Levels Using the Four Criteria
Alignment
Level
Acceptable
Weak
Unacceptable
Categorical
Concurrence
Depth of
Knowledge
Range of
Knowledge
Balance of
Representation
6 items per
standard
50%
50%
70%
---
40%-49%
40%-49%
60%-69%
Less than 6
items per
standard
Less than
40%
Less than
40%
Less than
60%
Categorical Concurrence
State B Grade 8 Mathematics
20
18
16
14
12
Mean Hits 10
8
6
4
2
0
N
G
PR
M
Standards
PS/R
PS
COMP
Depth-of-Knowledge Consistency
State B Grade 8 Mathematics
70
60
50
Mean Percent of 40
Hits
30
% Under
% At
% Above
20
10
0
N
G
PR
M
Standards
PS/R
PS
COMP
Range-of-Knowledge Correspondence
State B Grade 8 Mathematics
100
90
80
70
60
Mean % of
50
Objectives Hit
40
30
20
10
0
N
G
PR
M
Standards
PS/R
GPS COMP
Balance Index
State B Grade 8 Mathematics
0.9
0.8
0.7
0.6
0.5
Balance Index
0.4
0.3
0.2
0.1
0
N
G
PR
M
Standards
PS/R
P
COMP
Coding Process Tips
•
One Primary Objective and up to Two
Secondary Objectives (if necessary)
•
Source of Challenge (a correct/incorrect
response for the wrong reason)
•
Notes (any insights to share)
•
Consider Full Range of Standards
•
Use generic objectives sparingly
Structure of the Automated Alignment Process
Registration
Group Leader
Reviewers
Standards/Goals/Objectives Entry Process
Training on Depth-of-Knowledge Levels
Phase I Consensus Process on Assigning DOK
Levels to Objectives
Phase II Coding of Assessment Tasks
Phase III Analysis of Coding
Phase IV Reporting
Web Sites
http://facstaff.wcer.wisc.edu/normw/
Alignment Tool
http: //www.wcer.wisc.edu/WAT/index.aspx
Survey of the Enacted Curriculum
http://www.SECsurvey.org
EXAMPLE OF STANDARDS AND
DEPTH-OF-KNOWLEDGE LEVELS
CONTENT AREA: GEOMETRY
Mathematics Standard
State D
Grade 8
VI.2
State B
Grade 8
II.4
State A
Grade 6
IV.D.
IV.D.1
VI.
Depth-ofKnowledge
Level
Geometric and Spatial Sense
Explore transformations of geometric figures.
II. Geometry
3
Graph on a coordinate plane similar figures, reflections, and
translations.
IV. Geometry and Spatial Sense
2
Investigate and predict the results of transformations of
shapes, figures, and models including slides, flips, and turns.
Identify and describe the results of translations (slides),
reflections (flips), rotations (turns), or glide reflections.
2
EXAMPLE OF STANDARDS AND
DEPTH-OF-KNOWLEDGE LEVELS
CONTENT AREA: PROBABILITY AND
STATISTICS
Mathematics Standard
State D
Grade 8
VII. Data Analysis, Probability &
Statistics
VII.3
Formulate, predict, and defend positions
taken that are based on data collected.
State B
Grade 8
VI. Probability and Statistics
VI.1
Collect data involving 2 variables and
display on a scatter plot; interpret results.
Depth-ofKnowledge
Level
3
3
Which of the following numbers, when
rounded to the nearest thousand,
becomes 27,000?
(a) 26,099
(b) 26,490
(c) 27,381
(d) 27,550
(e) 27,640
Which of these means about the same
as the word gauge?
a. balance
b. measure
c.
select
d. warn
121
13
32
+ 34
1) 190
2) 200
3) 290
4) N
A car odometer registered 41,256.9 miles when a
highway sign warned of a detour 1,200 feet ahead.
What will the odometer read when the car reaches the
detour? (5,280 feet = 1 mile)
(a)
42,456.9
(b)
41,279.9
(c)
41,261.3
(d)
41,259.2
(e)
41,257.1
Did you use the calculator on this question?
Yes
No
This question refers to pieces N, P, and Q.
In Mr. Bell’s classes, the students voted for their favorite shape for a
symbol. Here are the results.
Class 1 Class 2 Class 3
Shape N
9
14
11
Shape P
1
9
17
Shape Q
22
7
2
Using the information in the chart, Mr. Bell must select one of the
shapes to be the symbol. Which one should he select and why?
The shape Mr. Bell should select: __________________
Explain:
Which of these conclusions is best supported by
information from the passage?
a. If a candidate meets the personal and educational
qualifications and is in fair physical shape, his or her
chances of becoming an agent are very good.
b. Compared with other law enforcement agencies in the
country, the F.B.I. has a low success rate for tracking
down and apprehending suspected offenders.
c. The job of an agent is not for everyone; it takes someone
with special training who is not afraid of danger and
doesn’t mind being socially isolated at times.
d. The life of a federal investigator is not as interesting as
most people think; agents spend most of their time
working at desks.
New Cubes
Your school is planning a casino night to raise funds
to construct a wall aquarium in your school. As a
mathematics student, you are given the job of
developing a dice game for this event.
A regular pair of “number dice” consists of two
cubes, each with its faces numbered 1 through 6.
Often, dice games are played by rolling the two dice
and then finding the sum of the two numbers turned
upward.
1. Show that, with a regular pair of number dice, the probability of
rolling a sum of 7 is greater than the probability of rolling any other
sum.
You decide to call your casino game “New Cubes.” To make it interesting, you decide to construct
new dice that have different numbers on their faces than regular dice. Here is how you will construct
them:




Only the single digits 0 through 9 can be used.
Any digit can be used more than once.
When the dice are rolled every sum from 4 to 14 must be possible and
no other sums can occur.
The two dice do not have to be identical.
2. What numbers would you put on the 6 faces of each of the two dice so that the above
conditions are met?
Die 1:
_____
_____
_____
_____
_____
_____
Die 2:
_____
_____
_____
_____
_____
_____
3. Which sum(s) do you think would turn up most frequently if your New Cubes were
rolled 1000 times?
Explain why.
Coordination of an Alignment Institute
Identify
• Content areas
• Grade levels
• Number of test forms
• Number of reviewers
• Computer facilities
• Standards and their structure
Coordination of an Alignment Institute
Ask if
• Tests include field test items
• Items have different point values
• Alternate assessments will be included
• English Language Learners will be included
WAT Adoption to State Needs
•
•
•
•
Assessment development (front end alignment)
District and local assessments
Test to test comparison analysis
Curriculum to standard analysis
Grades 9–12 Science Objectives and Depth-ofKnowledge (DOK) Levels for Michigan
Alignment Analysis
Standard
Total
Number of
Objs. Under
Standard
DOK
Levels
of Objs.
60
1
2
3
4
# of
Objs by
DOK
Levels
10
43
6
1
% of
Objs by
DOK
Levels
17
72
10
1
Comparison of Six Science Assessments on
Categorical Concurrence
Standard
Categorical Concurrence
MEAP ACT PLAN SAT
SAT
Biology Chemistry
SAT
Physics
I
Y
Y
Y
Y
N
N
II
N
N
N
N
N
N
III
IV
Y
Y
N
N
N
N
Y
N
N
Y
N
Y
V
Y
N
N
N
N
N
Comparison of Six Science Assessments on
Balance of Representation
Standard
Balance of Representation
MEAP ACT PLAN
SAT
SAT
Biology Chemistry
SAT
Physics
I
Y
Y
W
Y
n/a
n/a
II
n/a
n/a
n/a
n/a
n/a
n/a
III
Y
n/a
n/a
W
n/a
n/a
IV
Y
n/a
n/a
n/a
W
N
V
W
n/a
n/a
n/a
n/a
n/a
Cross-Assessment Summary
Three Analytic Methods
• Common Framework
• Expert Consensus
• Common Criteria
Survey of the Enacted
Curriculum
Time on Topic
Number
sense/Properties/
Relationships
Memorize
Facts/etc.
Perform
Procedures
Demonstrate
Understanding of
Mathematical
Ideas
0 1 2
3
Place value
0 1 2
3
0 1 2
3
0 1 2
3
0 1 2
3
Whole numbers
0 1 2
3
0 1 2
3
0 1 2
3
0 1 2 3
Operations
0 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
Fractions
0 1 2 3
0 1 2 3
0 1 2 3
0 1 2
3
Decimals
0 1 2
3
0 1 2
3
0 1 2
3
0 1 2
3
Percents
0 1 2
3
0 1 2
3
0 1 2
3
Enacted, Intended, and Assessed
Curriculum
Intended—What
standards require
Enacted—What
teachers teach
Assessed—What
state tests
Achieve Matrix
Grade 3 Mathematics
Data Analysis and Probability
Obj. #
Text of Objective
A
Organize, describe and make predictions from existing data
10.A.1a
Organize and display data using
pictures, tallies, tables, charts, or bar
graphs
81
10.A.1b
Answer questions and make
predictions based on given data.
5
20
41
53
66
69
B
Content
Centrality
Type of
Performance
Centrality
Source of
Challenge
Achieve Alignment Criteria
Item-Standard Match
Instrument-Standard
Match
•Content Centrality
•Level of Challenge
•Performance Centrality
•Balance
•Source of Challenge
•Range
Alignment Process
• Identify Standards and Assessments
• Select 6-8 Reviewers (Content Experts)
• Train Reviewers on DOK Levels
• Part I: Code DOK Levels of the
•
Standards/Objectives
Part II: Code DOK Levels and Corresponding
Objectives of Assessment Items