Transcript Alignment Studies: Powerful Tool for Focusing Instruction

Alignment
Powerful Tool for Focusing
Instruction, Curricula, and
Assessment
Major Questions
 What
 How
is alignment?
is alignment measured?
 How
are state standards and
assessments aligned?
 What
are the implications of
alignment results?
What Is Alignment?
Alignment
The degree to which expectations
and assessments are in agreement
and serve in conjunction with one
another to guide the system toward
students learning what is expected.
Degree of Alignment
Standards
Standards
Assessment
Assessment
Standards
Assessment
Standards
Assessment
Items
Alignment Is Important

Federal statutes require alignment

Education systems are easily fragmented
National
Standards
Purpose
Policy
Higher Education
Requirements
Best
Thinking
Standards
Public
Opinion
Frameworks
Teacher
Certification
Program
Practice
School
Organization
Work
Expectations
Assessments
Professional
Development
Textbooks
Student Outcomes
Classroom
Instruction
Alignment Is Important

Federal statutes require alignment

Education systems are easily fragmented
Teachers and students receive mixed
messages
 Learning expectations can be lowered
for some students

Curriculum Types
 Intended
 Implemented
 Assessed
 Achieved
Alignment has been an issue
for as many as forty years,
dating back to behavioral
objectives and
mastery-learning.
How is alignment
measured?
Three Methods:
 Common
 Expert
Framework
Consensus
 Common
Criteria
Content-by-Process Framework
T op ics
M easu rem en t
& C alcu lation
in S cien ce
C atego ries o f C og n itiv e D em an d
C o v erag e
M em o rize
F acts/etc.
U n d erstan d
C o n cep ts
P erfo rm
P ro ced u res
The
In tern a tio n a l
S y stem
0 1 2
3
0 1 2
3
0 1 2
3
0 1 2
3
M a ss & W eig h t
0 1 2
3
0 1 2
3
0 1 2
3
0 1 2
3
L en g th
0 1 2
3
0 1 2
3
0 1 2
3
0 1 2
3
V o lu m e
0 1 2
3
0 1 2
3
0 1 2
3
0 1 2
3
T im e
0 1 2
3
0 1 2
3
0 1 2
3
0 1 2
3
T em p era tu re
0 1 2
3
0 1 2
3
0 1 2
3
0 1 2
3
Categories of Cognitive Demand







Memorize
Understand Concepts
Perform Procedures
Generate Questions/Hypotheses
Collect Data
Analyze & Interpret Information
Use Information to Make Connections
Achieve Matrix
Grade 3 Mathematics
Data Analysis and Probability
O b j. #
T ex t o f O b jectiv e
1 0 .A .
1a
O rg an ize an d
d isp lay data u sin g
p ictu res, tallies,
tab les, ch arts, o r
b ar g rap h s.
81
10A .
1b
A n sw er q u estio n s
an d m ak e
p red ictio n s b ased
o n g iv en d ata.
5
20
41
53
66
69
A
B
C o n ten t
C en tra lity
T ype of
P erform a n ce
C en tra lity
O rg a n ize, d escrib e a n d m a k e p red ictio n s fro m ex istin g d a ta
S o u rce o f
C h a llen g e
Grade 8 Standards from Three States
State A:
The student will use proportions to solve scale-model
problems with fractions and decimals.
State B:
Students compute with rational numbers expressed in a
variety of forms; they solve problems involving ratios,
proportions, and percentages. Use ratio and proportion to
solve problems.
State C:
Apply proportional thinking in a variety of problem
situations that include, but are not limited to: 1) ratios and
proportions, and 2) percents, including those greater than
100 and less than one.
Achieve Alignment Criteria
Item-Standard Match
Instrument-Standard
Match

Content Centrality

Level of Challenge

Performance Centrality

Balance

Source of Challenge

Range
Five General Criteria
1. Content Focus
2. Articulation Across Grades and Ages
3. Equity and Fairness
4. Pedagogical Implications
5. System Applicability
Specific Criteria
Content Focus
A. Categorical Concurrence
B. Depth-of-Knowledge Consistency
C. Range-of-Knowledge Correspondence
D. Structure-of-Knowledge Comparability
E. Balance of Representation
F. Dispositional Consonance
Indiana Mathematics Standards
Grade 8
O b j. # T ex t o f O b jectiv e
5A
5B
6A
6B
E xp ress an y d ecim al in
scien tific no tatio n
O rd er a set o f ration al
n u m b ers
F raction s, P ercen ts, In teg ers,
& Irration als
R ein force an u nd erstand ing o f
fraction s, and dev elop an
u nd erstand ing of p ercen t,
in teg ers, an d irration als
U se m o d els to co m p are tw o or
m o re in teg ers and /or irratio n al
n u m b ers
R eco gn ize th e relation sh ip s
b etw een fraction , p ercen t,
ratio , and p ro p ortio n
DOK
Item # 1
Item # 2
D ep th o f K n o w led g e
L ev el 1
R ecall
R ecall o f a fact, in fo rm atio n , o r p ro ced u re.
L ev el 2
S k ill/C on cep t
U se in fo rm atio n o r co n cep tu al k n o w led g e, tw o
o r m o re step s, etc.
L ev el 3
S trateg ic T h in k in g
R eq u ires reaso n in g , d ev elo pin g p lan o r a
seq u en ce o f step s, so m e co m p lex ity , m o re
th an o n e po ssib le an sw er.
L ev el 4
E x ten d ed T h in k in g
R eq u ires an in v estig atio n , tim e to th in k an d
p ro cess m u ltip le co n d itio n s o f th e p ro b lem .
EXAMPLE OF STANDARDS AND
DEPTH-OF-KNOWLEDGE LEVELS
CONTENT AREA: GEOMETRY
M athem atics Standard
State D
G rade 8
V I.2
State B
G rade 8
II.4
State A
G rade 6
IV .D .
IV .D .1
V I.
D epth-ofK now ledge
Level
G eom etric and Spatial Sense
E xplore transform ations of geom etric figures.
II. G eom etry
4
G raph on a coordinate plane sim ilar figures, reflections, and
translations.
IV . G eom etry and Spatial Sense
2
Investigate and predict the results of transform ations of
shapes, figures, and m odels including slides, flips, and turns.
Identify and describe the results of translations (slides),
reflections (flips), rotations (turns), or glide reflections.
2
EXAMPLE OF STANDARDS AND
DEPTH-OF-KNOWLEDGE LEVELS
CONTENT AREA: PROBABILITY AND
STATISTICS
M ath em atics S tan d ard
S tate D
G rad e 8
V II. D ata A n aly sis, P ro b ab ility &
S tatistics
V II.3
F o rm u late, p red ict, an d d efen d p o sitio n s
tak en th at are b ased o n d ata co llected .
S tate B
G rad e 8
V I. P ro b ab ility and S tatistics
V I.1
C o llect d ata in v o lv ing 2 v ariab les an d
d isp lay o n a scatter p lo t; in terp ret resu lts.
D ep th -o fK n o w led g e
L ev el
4
3
W h ich o f th e fo llo w in g n u m b ers, w h en
ro u n d ed to th e n earest tho u san d ,
b eco m es 2 7 ,0 0 0?
(a) 2 6 ,0 9 9
(b ) 2 6 ,4 9 0
(c) 2 7 ,3 8 1
(d ) 2 7 ,5 5 0
(e) 2 7 ,6 4 0
121
13
32
+ 34
1) 190
2) 200
3) 290
4)
N
A car o d o m eter reg istered 4 1 ,2 5 6 .9 m iles w h en a
h ig h w ay sig n w arn ed o f a d eto u r 1 ,2 0 0 feet ah ead .
W h at w ill th e o d o m eter read w h en th e car reach es th e
d eto u r? (5 ,2 8 0 feet = 1 m ile)
(a)
4 2 ,4 5 6 .9
(b )
4 1 ,2 7 9 .9
(c)
4 1 ,2 6 1 .3
(d )
4 1 ,2 5 9 .2
(e)
4 1 ,2 5 7 .1
D id y o u u se th e calcu lato r o n th is q u estio n ?
Y es
No
T h is q u estio n refers to p ieces N , P , and Q .
In M r. B ell’s classes, th e stu d en ts v o ted for th eir favo rite sh ape for a
sy m b o l. H ere are th e resu lts.
C lass 1 C lass 2
C lass 3
S h ap e N
9
14
11
S h ap e P
1
9
17
S h ap e Q
22
7
2
U sing th e inform atio n in th e ch art, M r. B ell m u st select o n e of th e
sh ap es to b e th e sym b o l. W h ich on e sh ou ld h e select and w h y?
T h e sh ap e M r. B ell shou ld select: _ ___ ___ __ ____ __ ___
E xp lain :
New Cubes
Your school is planning a casino night to raise funds
to construct a wall aquarium in your school. As a
mathematics student, you are given the job of
developing a dice game for this event.
A regular pair of “number dice” consists of two
cubes, each with its faces numbered 1 through 6.
Often, dice games are played by rolling the two dice
and then finding the sum of the two numbers turned
upward.
1. Show that, with a regular pair of number dice, the probability of
rolling a sum of 7 is greater than the probability of rolling any other
sum.
Y ou decide to call you r casino gam e “N ew C ubes.” T o m ake it interesting, you decide to construct
new dice that have different nu m b ers on their faces than regular dice. H ere is how you w ill construct
them :




O nly the single digits 0 through 9 can be used.
A ny digit can be used m ore than once.
W hen the dice are rolled every su m fro m 4 to 14 m ust be possible and
no other su m s can occur.
T he tw o dice do not have to be identical.
2. W hat nu m bers w ould you put on the 6 faces of each of the tw o dice so that the above
conditions are m et?
D ie 1:
_____
_____
_____
_____
_____
_____
D ie 2:
_____
_____
_____
_____
_____
_____
3. W hich su m (s) do you think w ould turn up m ost frequently if your N ew C ubes w ere
rolled 1000 tim es?
E xplain w h y.
Categorical Concurrence
State B Grade 8 Mathematics
20
18
16
14
12
Mean Hits 10
8
6
4
2
0
N
G
PR
M
Standards
PS/R
PS
COMP
Depth-of-Knowledge Consistency
State B Grade 8 Mathematics
70
60
50
Mean Percent of 40
Hits
30
% Under
% At
% Above
20
10
0
N
G
PR
M
Standards
PS/R
PS
COMP
Range-of-Knowledge Correspondence
State B Grade 8 Mathematics
100
90
80
70
60
Mean % of
50
Objectives Hit
40
30
20
10
0
N
G
PR
M
Standards
PS/R
GPS COMP
Balance Index
State B Grade 8 Mathematics
0.9
0.8
0.7
0.6
0.5
Balance Index
0.4
0.3
0.2
0.1
0
N
G
PR
M
Standards
PS/R
P
COMP
Four State Study
Percent of Standards with Acceptable Alignment
Category
100
90
80
70
60
Science
50
Math
40
30
20
10
0
Grade
3
Grade
8
Grade
3
Grade
6
Grade
4
Grade
8
Grade
10
Grade
4
Grade
8
Grade
3
Grade
7
Grade
10
Grade
4
Grade
8
Four State Study
Percent of Standards with Acceptable Alignment
Depth of Knowledge
100
90
80
70
60
50
Science
40
Math
30
20
10
0
Grade 3 Grade 8 Grade 3 Grade 6 Grade 4 Grade 8 Grade Grade 4 Grade 8 Grade 3 Grade 7 Grade Grade 4 Grade 8
10
10
Four State Study
Percent of Standards with Acceptable Alignment
Range
100
90
80
70
60
50
Science
40
Math
30
20
10
0
Grade 3 Grade 8 Grade 3 Grade 6 Grade 4 Grade 8 Grade Grade 4 Grade 8 Grade 3 Grade 7 Grade Grade 4 Grade 8
10
10
Four State Study
Percent of Standards with Acceptable Alignment
Balance
100
90
80
70
60
50
Science
40
Math
30
20
10
0
Grade 3 Grade 8 Grade 3 Grade 6 Grade 4 Grade 8 Grade Grade 4 Grade 8 Grade 3 Grade 7 Grade Grade 4 Grade 8
10
10
Average Measure Intraclass Correlation of Depth-of-Knowledge
Levels Ratings for Mathematics
State/Grade
Number of
Reviewers
Number of
Items
Alpha
95% CI
Lower-Upper
A4
4
61
.47
.11-.68
A7
4
61
.93
.89-.95
A7 (Repl)
4
63
.77
.66-.85
A9
4
63
.81
.72-.88
B5
4
49
.79
.67-.87
B8
4
46
.58
.34-.75
C4
4
48
.88
.81-.93
C4 (Repl)
4
52
.89
.84-.93
C8
4
146
.86
.82-.89
C11
4
56
.91
.86-.94
Average
.79
Average Measure Intraclass Correlation of Depth-of-Knowledge
Levels Ratings for Language Arts
State/Grade
Number of
Reviewers
Number of
Items
Alpha
95% CI
Lower-Upper
A4
5
84
.52
.34-.67
A7
3
74
.70
.56-.80
A10
3
82
.62
.45-.74
B5
6
38
.79
.66-.88
B8
6
35
.60
.36-.78
B11
6
88
.85
.79-.89
C4
6
45
.92
.88-.95
C8
6
40
.87
.80-.92
C11
6
192
.86
.83-.89
Average
.75
Table 1
Categorical Concurrence Between Standards and Assessment as Rated by Four Reviewers
Grade 4 Mathematics
(Number of Assessment Items—65 Multiple Choice Items)
Standards
Title
Level by Objective
Goals
#
I. Number Sense
4
II. Algebra &
Functions
2
III. Measurement &
Geometry
IV. Statistics, Data
Analysis &
Probability
Total
1Includes
3
2
11
Hits
% w/in
std by
Level
Mean
S.D.
Categorical
Concurr.
Acceptable
Objs#
Level
# of
objs by
Level
17.751
1
2
11
7
61
39
26.75
.43
YES
7.25
1
2
5
2
71
29
20.50
1.12
YES
172
1
2
3
15
1
1
88
6
6
15.25
.43
YES
5
1
2
3
3
1
1
60
20
20
2.75
.43
NO
46.75
1
2
3
34
11
2
72
24
4
65.25
.83
one generic objective because coded items did not correspond to existing objectives.
2Includes two generic objectives because coded items did not correspond to existing objectives.
Table 2
Depth-of-Knowledge Consistency Between Standards and Assessment
as Rated by Four Reviewers
Grade 4 Mathematics
(Number of Assessment Items—65 Multiple Choice Items)
Level of Item w.r.t. Standard
Standards
%Under
Title
%At
%Above
Depth-of-Knowledge
Consistency Acceptable
Mean
S.D.
Mean
S.D.
Mean
S.D.
I. Number Sense
26
44
73
44
2
8
YES
II. Algebra & Functions
10
29
88
29
3
8
YES
III. Measurement &
Geometry
4
20
95
21
1
7
YES
IV. Statistics, Data
Analysis & Probability
50
50
50
50
0
0
WEAK
Total
18
38
81
38
1
7
Table 3A
Range-of-Knowledge Correspondence and Balance of Representation
Between Standards and Assessment as Rated by Four Reviewers
Grade 4 Mathematics
(Number of Assessment Items—65 Multiple Choice Items)
Range of Objectives
Standards
#Objs Hit
Title
% of Total
Range of Knowledge
Acceptable
Mean
S.D.
Mean
S.D.
I. Number Sense
13.5
1.80
76
9
YES
II. Algebra & Functions
5.25
.43
72
2
YES
III. Measurement &
Geometry
11.50
.50
68
3
YES
IV. Statistics, Data Analysis
& Probability
2.50
.50
50
10
WEAK
Total
8.19
4.59
67
13
Table 3B
Balance of Representation Between Standards and Assessment
as Rated by Four Reviewers
Grade 4 Mathematics
(Number of Assessment Items—65 Multiple Choice Items)
Balance Index
1 (perfect-0 no Balance)
Standards
Title
% Hits in Std/Ttl
Hits
Balance of Representation
Acceptable
Index
Mean
S.D.
Mean
S.D.
I. Number Sense
41
1
.81
.01
YES
II. Algebra & Functions
31
1
.67
.05
WEAK
III. Measurement &
Geometry
23
1
.81
.01
YES
IV. Statistics, Data Analysis
& Probability
4
1
.96
.07
YES
Total
25
14
.81
.12
Table 4
Summary of Attainment of Acceptable Alignment Level on Four Content Focus Criteria
Grade 4 Mathematics
(Number of Assessment Items—65 Multiple Choice Items)
Alignment Criteria
Standards
Categorical
Concurrence
Depth-ofKnowledge
Consistency
Range of
Knowledge
Balance of
Representation
I. Number Sense
YES
YES
YES
YES
II. Algebra & Functions
YES
YES
YES
WEAK
III. Measurement &
Geometry
YES
YES
YES
YES
IV. Statistics, Data Analysis
& Probability
NO
WEAK
WEAK
YES
What are the
implications of
the results?
IMPLICATIONS

Confirm quality of assessment items and
standards

Attend to depth-of-knowledge levels

Write standards at a moderate level of
specificity

Use multiple measures

Identify acceptable levels for alignment
criteria