Transcript An Introduction to Progress Monitoring in Mathematics

An Introduction to Progress
Monitoring in Mathematics
Center on Instruction
Mathematics Strand
Funded by U.S. Department of Education
The Center on Instruction is operated by RMC Research Corporation
in partnership with
the Florida Center for Reading Research at Florida State University;
Instructional Research Group;
the Texas Institute for Measurement, Evaluation, and Statistics at the University of Houston; and
The Meadows Center for Preventing Educational Risk at The University of Texas at Austin
The contents of this PowerPoint were developed under cooperative agreement S283B050034 with
the U.S. Department of Education. However, these contents do not necessarily
represent the policy of the Department of Education, and you should not
assume endorsement by the Federal Government.
2009
The Center on Instruction requests that no changes be made to the content or appearance of this product.
To download a copy of this document, visit www.centeroninstruction.org
Funded by U.S. Department of Education
Overview of the Presentation
 Describe progress monitoring
 Explain common techniques that are often mistaken
for progress monitoring
 Discuss features of progress monitoring
• Application to elementary grades
• Application to secondary grades
 Provide resources for additional information
Funded by U.S. Department of Education
A Disclaimer…
 This presentation makes use of visual images of a variety
of mathematics progress monitoring measures, some of
which are commercial products. The Center on Instruction
is NOT endorsing any of these products. Audience
members for this presentation must keep in mind that there
are many alternatives available to practitioners and those
we have included in this presentation serve as illustrations
of the larger range of options.
Funded by U.S. Department of Education
Progress Monitoring
 Supports formative evaluation of student learning
 Informs teacher instructional decision making
Funded by U.S. Department of Education
General Definition of
Student Progress Monitoring
 Collecting and evaluating data to make decisions
about the adequacy of student progress toward a
goal
 Evaluating student rate of change (slope) as
compared to the slope of anticipated progress
 Informing teacher planning for instruction
Funded by U.S. Department of Education
General Definition of
Student Progress Monitoring
 Requires:
• Technically sound measures
• Multiple forms of the same measure
• Assessment systems that are sensitive to student
growth
• Standardized administration procedures
• Frequent measurement (occurs at least monthly)
Funded by U.S. Department of Education
Sample Progress Monitoring
Measure
Name _______________________________
Date ________________
 First page of three-page
concepts and applications
measure (24 problems total)
 Measure taken from
Monitoring Basic Skills
Progress: Basic Math
Concepts and Applications
(1999)
Applications 4
Column A
(1) Write the letter in each blank.
•
•K
•M
(5)
(A) line segment
Z
L•
N•
Column B
Write a number in the blank.
1 week = _____ days
(B) line
(C) point
Vacation Plans for Summit
School Students
(6)
(D) ray
Summer
school
(2) Look at this numbers.:
356.17
Camp
Which number is in the hundredths
place?
Travel
Stay
home
(3) Solve the problem by estimating the
sum or difference to the nearest ten.
0 10 20 30 40 50 60 70 80 90 100
Number of Students
Jeff wheels his wheelchair for 33
hours a week at school and for 28
hours a week in his neighborhood.
About how many hours does Jeff
spend each week wheeling his
wheelchair?
(4) Write the number in each blank.
3 ten thousands, 6 hundreds, 8
ones
2 thousands, 8 hundreds,
4 tens, 6 ones
Funded by U.S. Department of Education
Test 4 Page 1
Use the bar graph to answer the questions.
The P.T.A. will buy a Summit School TShirt for each student who goes to summer
school. Each shirt costs $4.00. How much
money will the P.T.A. spend on these T
shirts?
$ .00
How many students are planning to travel
during the summer?
How many fewer students are planning to go
to summer school than planning to stay
home?
(7)
To measure the distance of the bus ride
from school to your house you would use
(A) meters
(B) centimeters
(C) kilometers
Display of Progress
Monitoring Data
40
38
Baseline
Intervention 1
Intervention 2
36
34
32
30
Correct responses
on math probe
Number of correct responses
28
26
24
Median of
Baseline
22
Annual Goal
20
18
16
14
12
10
8
6
4
2
0
Funded
by U.S. Department of Education
Week of School
Common Assessment
Approaches
That ARE NOT
Progress Monitoring
Funded by U.S. Department of Education
Common Assessment
Approaches That Are Not
Progress Monitoring
 Screening tools
 Diagnostic assessments
 Curriculum-embedded assessments
• Teacher created
• Publisher developed
Funded by U.S. Department of Education
Curriculum-Embedded Assessments
 Teachers use these assessments to determine whether
students learned a particular concept/skill or learned what was
taught in the chapter or unit.
 Teachers may use these assessments to track mastery of shortterm instructional objectives.
 Sampling of items is representative of a limited set of problems,
concepts, or skills.
 Assessment materials mirror instructional materials.
Funded by U.S. Department of Education
Curriculum-Embedded Assessments
 Teacher-created
• Teacher develops assessments that focus on a
particular concept or skill.
• Teacher creates multiple forms.
• Teacher gives assessment until student has learned
that skill or concept.
• Teacher often uses assessments with students who
are struggling with particular concepts or skills.
Funded by U.S. Department of Education
Curriculum-Embedded Assessments
 Publisher-developed
• Teacher gives chapter and unit exams included with the
textbook series to evaluate student learning.
• Teachers typically use assessments with the entire
class.
Funded by U.S. Department of Education
Curriculum Embedded Assessments
An Example from an
Elementary Tutoring Context
 Mr. Jones is tutoring a fourth-grade student who
struggles with math computation skills.
 He examines the sequence of skills for fourthgrade computation and develops a criterionreferenced test for each skill within the
sequence.
Funded by U.S. Department of Education
Curriculum Embedded Assessments
An Example from an
Elementary Tutoring Context
 Mr. Jones provides instruction and gives
alternate forms of the criterion-referenced test
until the skill is learned.
 Then, he changes instruction to focus on the
next skill in the sequence.
Funded by U.S. Department of Education
Curriculum Embedded Assessments
Hypothetical Fourth-Grade
Computation Curriculum
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
Multidigit addition with regrouping
Multidigit subtraction with regrouping
Multiplication facts, factors to 9
Multiply 2-digit numbers by a 1-digit number
Multiply 2-digit numbers by a 2-digit number
Division facts, divisors to 9
Divide 2-digit numbers by a 1-digit number
Divide 3-digit numbers by a 1-digit number
Add/subtract simple fractions, like denominators
Add/subtract whole number and mixed number
Funded by U.S. Department of Education
Curriculum Embedded Assessments
Multidigit Addition Test
Funded by U.S. Department of Education
Curriculum Embedded Assessments
Mastery of Multidigit Addition
Funded by U.S. Department of Education
Curriculum Embedded Assessments
Hypothetical Fourth-Grade
Computation Curriculum
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
Multidigit addition with regrouping
Multidigit subtraction with regrouping
Multiplication facts, factors to 9
Multiply 2-digit numbers by a 1-digit number
Multiply 2-digit numbers by a 2-digit number
Division facts, divisors to 9
Divide 2-digit numbers by a 1-digit number
Divide 3-digit numbers by a 1-digit number
Add/subtract simple fractions, like denominators
Add/subtract whole number and mixed number
Funded by U.S. Department of Education
Curriculum Embedded Assessments
Multidigit Subtraction Test
Date
Name:
Subtracting
6 52 1
3 75
5 42 9
6 34
8 45 5
7 56
6 78 2
9 37
7 32 1
3 91
5 68 2
9 42
6 42 2
5 29
3 48 4
4 26
2 41 5
8 54
4 32 1
8 74
Funded by U.S. Department of Education
Curriculum Embedded Assessments
Number of problems correct
in 5 minutes
Mastery of Multidigit Addition
and Subtraction
Multidigit Subtraction
Multidigit
10 Addition
Multiplication
Facts
8
6
4
2
0
2
4
Funded by U.S. Department of Education
6
8
WEEKS
10
12
14
Curriculum Embedded Assessments
An Example from a
Secondary Classroom Teacher
 Ms. Harwood teaches ninth-grade algebra.
 Her district has adopted a textbook series that
uses a traditional instructional approach for
algebra.
 She provides instruction on each chapter, then
gives the chapter test to evaluate student
learning.
Funded by U.S. Department of Education
Curriculum Embedded Assessments
Hypothetical Algebra Curriculum
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
Connections to algebra
Properties of real numbers
Solving linear equations
Graphing linear equations and functions
Writing linear equations
Solving and graphing linear inequalities
Systems of linear equations and inequalities
Exponents and exponential functions
Quadratic equations and functions
Polynomials and factoring
Rational expressions and equations
Radicals and more connections to geometry
Funded by U.S. Department of Education
Curriculum Embedded Assessments
Hypothetical Algebra Curriculum
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
Connections to algebra
Properties of real numbers
Solving linear equations
Graphing linear equations and functions
Writing linear equations
Solving and graphing linear inequalities
Systems of linear equations and inequalities
Exponents and exponential functions
Quadratic equations and functions
Polynomials and factoring
Rational expressions and equations
Radicals and more connections to geometry
Funded by U.S. Department of Education
Curriculum Embedded Assessments
Potential Difficulties with
Curriculum-Embedded Assessment
 Assessments do not reflect maintenance or
generalization of the concepts/skills.
 Assessments typically are designed by teachers or
sold with textbooks with unknown reliability and
validity.
 Number of concepts/skills or chapters passed does
not relate well to performance on high-stakes tests.
Funded by U.S. Department of Education
Curriculum Embedded Assessments
Potential Difficulties with
Curriculum-Embedded
Assessment
 Sequence of concepts/skills or chapters is logical,
not empirical.
 Difficulty of tasks may vary from test to test.
 Performance on limited-skill assessments can be
misleading.
Funded by U.S. Department of Education
Features
Specific Features of
Mathematics Progress
Monitoring
Funded by U.S. Department of Education
Progress Monitoring
 The process of collecting and evaluating data to
determine whether students are making progress
toward instructional goals and/or are responding to
instructional interventions
Funded by U.S. Department of Education
Progress Monitoring
 Uses:
Estimate rate of student improvement
Describe student response to instructional methods
Inform teachers about instructional decision making
Aid teachers in targeting areas/skills that need
remediation
• Help teachers build potentially more effective instruction
for particular students
•
•
•
•
Funded by U.S. Department of Education
Research Supports the Use of
Progress Monitoring
 Progress monitoring data produce accurate,
meaningful information about student academic
levels and corresponding rates of improvement.
 Progress monitoring data are sensitive to student
improvement.
Funded by U.S. Department of Education
Research Supports the Use of
Progress Monitoring
 Performance on progress monitoring measures
corresponds well to performance on high-stakes
tests.
 When teachers use progress monitoring data to
inform their instructional decisions, students make
greater learning gains.
Funded by U.S. Department of Education
Process of Progress Monitoring
 Progress monitoring is a data-based, instructional
decision-making tool.
 Steps for using data:
•
•
•
•
•
Gather baseline performance data,
Set instructional goals,
Provide targeted instruction,
Monitor progress toward goal, and
Adjust goal upward or modify instruction as needed.
Funded by U.S. Department of Education
Using
Using progress
progress monitoring
monitoring data
data to
to
test
test effectiveness
effectiveness of
of adaptations
adaptations to
to
class
class instruction
instruction
This
Thisgraph
graphshows
showsthat
thatthree
threeinstructional
instructionalmodifications
modificationshave
havebeen
been
implemented
implementedtotothe
theoriginal
originalinstructional
instructionalprogram
programfor
forDonald.
Donald.
Donald Ross
Computation 4
70
D
I
G
I
T
S
60
50
G
38
40
30
20
10
0
Sep
Oct
Funded by
Department of
of Education
Education
Funded
by U.S.
U.S. Department
Nov
Dec
Jan
Feb
Mar
Apr
May
Features of Progress Monitoring
Systems
 Data are collected and evaluated frequently.
• Schedule is determined by current level of
student performance and goal.
• Frequency of assessment typically ranges from
two times per week to monthly.
• Adjust measurement frequency based on
severity of student difficulties.
Funded by U.S. Department of Education
Features of Progress Monitoring
Systems
 Teachers may choose to monitor the progress of all
students in class.
 Typically, students who are at risk of failure are
assessed until they reach proficiency.
 Data-based decision rules are applied to graphed
data to determine when goals should be raised or
instruction should be modified.
Funded by U.S. Department of Education
Features of Progress Monitoring
Measures
 Difficulty of tasks remains consistent across the
year.
 Allotted time typically does not allow for
completion of test, so student growth still can be
assessed.
Funded by U.S. Department of Education
Features of Progress Monitoring
Measures
 Uses standardized administration and scoring:
• Test administration is timed (relatively short
tests in duration).
• Specific scoring rules are applied.
• Scoring typically uses counts, rather than
percent correct.
Funded by U.S. Department of Education
Two Approaches to Developing
Progress Monitoring Measures
(Fuchs, 2004)
 Curriculum Sampling
• Systematically sample items from the annual
curriculum on each measure
 Robust Indicator
• Identify a global behavior that either
encompasses many skills taught in the annual
curriculum or is predictive of proficiency in the
annual curriculum
Funded by U.S. Department of Education
Curriculum Sampling
 Each probe is a proportional sampling of the annual
curriculum.
 Advantages
• May conduct skills analysis
• May evaluate maintenance and generalization of skills
 Disadvantages
• Tend to be longer in duration
• May not generalize to other curricular programs
• Are grade-level specific
Funded by U.S. Department of Education
Robust Indicators
 Also referenced as general outcome measures
• Comprised of tasks that represent proficiency in the
content domain
• INDICATORS--not the “whole” of instruction
– Examples: oral reading fluency; estimation
• Empirically determined through correlations with
other indicators of proficiency in mathematics
Funded by U.S. Department of Education
Robust Indicators
 Advantages
• Do not have to be grade specific
• Tend to be shorter in duration
• May be used across curricular programs
 Disadvantages
• May not be tied closely to instructional content
• May not be able to provide skills analysis on instructional
content
• May not be able to evaluate maintenance and generalization
of instructional skills
Funded by U.S. Department of Education
Elementary
Mathematics
Progress Monitoring in
Elementary Grades
Funded by U.S. Department of Education
Measuring Mathematics Progress of
Elementary Students
 Elementary measures include examples of both
curriculum sampling and robust indicators.
 Several measures are available commercially in printed
format or as Web-based systems.
Funded by U.S. Department of Education
Elementary-Level Measures:
Curriculum Sampling Approach
 Test items represent the critical skills in the gradelevel curriculum (or represent grade-level state
standards).
 Although administration time is held constant across
the year, it may vary by grade level.
Funded by U.S. Department of Education
Elementary-Level Measures:
Curriculum Sampling Approach
 Measures may contain only computation problems,
only problems representing concepts and
applications, or a combination of both.
 Because the same skill types are tested repeatedly,
analysis of student performance with respect to
specific skills is possible.
Funded by U.S. Department of Education
Examples of Progress Monitoring
Measures
Developed Through
Curriculum Sampling
Funded by U.S. Department of Education
Monitoring Basic Skills
Progress: Basic Math
 Computation
• For Grades 1-6, test administration varies from
2-6 minutes, depending on grade level
• Scored as number of digits correct in answers
(using specified scoring algorithms)
Funded by U.S. Department of Education
Example
Computation 4
Sheet #2
 Random placement of
problem types on page
 Random numerals
within problems
Measure taken from Monitoring
Basic Skills Progress: Basic Math
Computation (2nd ed.). (1998).
Password: AIR
Name:
Date
A
B
9 )2 4
F
G
9
x 0
H
6 )3 0
L
8
x 6
Q
10 7
x 3
U
7
x 9
5 )6 5
2) 9
6 )3 0
41 6
44
15 04
14 41
3
1
=
3
4
- 1 =
7
T
5
3
+
=
11
11
X
9 )8 1
2
3
O
S
W
8285
4304
90
+
J
N
R
V
1
+ 6 =
2
4 )7 2
4
x 5
M
32
x 23
E
I
35
x 74
K
4
D
52 85 2
+ 64 70 8
P
Funded by U.S. Department of Education
C
6
x 2
Y
13 0
x 7
5 )1 0
Monitoring Basic Skills Progress:
Basic Math
 Concepts and Applications
• For Grades 2-6, test administration varies from
6-8 minutes, depending on grade level
• Scored as number of blanks correct
Funded by U.S. Department of Education
Sample Progress Monitoring
Measure
Name _______________________________
Date ________________
 First page of three-page
concepts and applications
measure (24 problems total)
 Measure taken from
Monitoring Basic Skills
Progress: Basic Math
Concepts and Applications
(1999)
Applications 4
Column A
(1) Write the letter in each blank.
•
•K
•M
(5)
(A) line segment
Z
L•
N•
Column B
Write a number in the blank.
1 week = _____ days
(B) line
(C) point
Vacation Plans for Summit
School Students
(6)
(D) ray
Summer
school
(2) Look at this numbers.:
356.17
Camp
Which number is in the hundredths
place?
Travel
Stay
home
(3) Solve the problem by estimating the
sum or difference to the nearest ten.
0 10 20 30 40 50 60 70 80 90 100
Number of Students
Jeff wheels his wheelchair for 33
hours a week at school and for 28
hours a week in his neighborhood.
About how many hours does Jeff
spend each week wheeling his
wheelchair?
(4) Write the number in each blank.
3 ten thousands, 6 hundreds, 8
ones
2 thousands, 8 hundreds,
4 tens, 6 ones
Funded by U.S. Department of Education
Test 4 Page 1
Use the bar graph to answer the questions.
The P.T.A. will buy a Summit School TShirt for each student who goes to summer
school. Each shirt costs $4.00. How much
money will the P.T.A. spend on these T
shirts?
$ .00
How many students are planning to travel
during the summer?
How many fewer students are planning to go
to summer school than planning to stay
home?
(7)
To measure the distance of the bus ride
from school to your house you would use
(A) meters
(B) centimeters
(C) kilometers
Yearly Progress Pro
TM
 Web-based progress monitoring system
 Both computation and problem-solving items
included on each form
 Test administered for 15 minutes at each grade level
1-8
 Multiple-choice format for most answers (scratch
paper allowed)
Funded by U.S. Department of Education
Yearly Progress Pro
TM
 Scores entire answer as correct or incorrect (out of
a total of 30 problems)
 Provides skills analyses for class and individual
students
 Contains instructional exercises for each skill type
Funded by U.S. Department of Education
Yearly Progress Pro: Sample screen taken from
an instructional exercise but also illustrates how
items are presented (one by one) on progress
monitoring measure
Funded by U.S. Department of Education
See http://www.mhdigitallearning.com
YPP: Skills Feedback Across Class
Shows specific skills
tested for algebra
cluster at Grade 6
See http://www.mhdigitallearning.com
Funded by U.S. Department of Education
Green circle indicates
mastery; yellow circle
indicates partial
mastery; red circle
indicates skill is not
mastered
Examples of Robust Indicators
Progress Monitoring
in Mathematics
Funded by U.S. Department of Education
EdCheckup: Cloze Math
 Web-based progress monitoring system
 Robust indicator consisting of basic facts in addition,
subtraction, multiplication, and division--80
problems administered for 2 minutes
 Paper-and-pencil administration option or electronic
option for administration and scoring
Funded by U.S. Department of Education
EdCheckup: Cloze Math
Funded by U.S. Department of Education
AIMSweb
 Web-based progress monitoring system
 Paper-and-pencil test administration
 Variety of measures for Grades 1-8:
• Basic facts by single operation or mixed operations (robust
indicator)--score by correct digits in answers
• Mixed skills by grade level (curriculum sampling)--no skills
analysis available; score by correct digits in answers or by
correct digits in answers and critical processes (as indicated
on answer key)
 Graphs of student progress
Funded by U.S. Department of Education
Sample AIMSweb
Mathematics Measures
Funded by U.S. Department of Education
Secondary Grades
Mathematics Progress Monitoring
in Secondary Grades
Funded by U.S. Department of Education
Measuring Mathematics
Progress of Secondary Students
 Development of progress monitoring measures for
secondary grades (especially beyond general math) is in
its infancy.
 Project AAIMS is developing and validating measures for
pre-algebra and first year algebra.
 Algebra measures have been created using both the
robust-indicators and the curriculum-sampling methods.
Funded by U.S. Department of Education
Examples of Algebra
Progress Monitoring Measures




Basic Skills
Algebra Foundations
Translations
Content-Analysis-Multiple Choice
Funded by U.S. Department of Education
Basic Skills
Funded by U.S. Department of Education
Basic Skills in Algebra
 Robust indicator of pre-algebra/algebra proficiency
 60 items; 5 minutes
 Problems include:
• Solving basic “fact” equations
• Applying the distributive property
• Working with integers
• Simplifying expressions
• Applying proportional reasoning
 Scoring: # of problems correct
Funded by U.S. Department of Education
Algebra Foundations
Funded by U.S. Department of Education
Algebra Foundations
 Robust indicator of pre-algebra/algebra proficiency
 42 items (50 points); 5 minutes
 Problems represent five core concepts/skills:
• Writing and evaluating variables and expressions
• Computing expressions (integers, exponents, and order of
operations)
• Graphing expressions and linear equations
• Solving one-step equations and simplifying expressions
• Identifying and extending patterns in data tables
 Scoring: # of problems correct
Funded by U.S. Department of Education
Translations
Funded by U.S. Department of Education
Translations
 Robust indicator of pre-algebra/algebra proficiency
 43 items; 7 minutes
 Problems are modeled on tasks drawn from the
Connected Mathematics Project middle school algebra
curriculum
 Task requires students to translate varying
representations for relationships between two variables:
data tables, graphs, equations, story scenarios
 Scoring: # problems correct - # problems incorrect
Funded by U.S. Department of Education
Content Analysis
Funded by U.S. Department of Education
Content Analysis-Multiple Choice
 16 items; 7 minutes
 Problems are sampled from core concepts in the
chapters in the first 2/3 of a traditional Algebra 1
textbook
 Must show work to obtain partial credit
 Scoring: Up to 3 points per problem awarded using a
scoring rubric, -1 point penalty for circling an incorrect
answer without showing any work (guessing)
Funded by U.S. Department of Education
Algebra
Progress Monitoring Research
Results
 Project AAIMS is conducting research on the technical
adequacy and teachers’ use of the algebra measures.
 Reliability, criterion validity, and sensitivity to growth
over time has been documented for the Basic Skills,
Algebra Foundations, and Content Analysis-Multiple
Choice measures.
Funded by U.S. Department of Education
Algebra
Progress Monitoring Research
Results
 Fewer data on the Translations measure due to
mismatch with existing curriculum materials in
participating school districts
 Ongoing research to continue refinement of the
measures
Funded by U.S. Department of Education
Funded by U.S. Department of Education
Reference
 Fuchs, L. S. (2004). The past, present, and future of
curriculum-based measurement research. School
Psychology Review, 33, 188-192.
Funded by U.S. Department of Education
Additional Resources
Progress Monitoring Measures
 AIMSweb Web site: http://www.aimsweb.com
 Edcheckup Web site: http://www.edcheckup.com
 Monitoring Basic Skills Progress (blackline masters available
for computation and for concepts/applications):
http://www.proedinc.com
 Project AAIMS Web site (algebra progress monitoring
measures and research results) www.ci.hs.iastate.edu/aaims
 Yearly Progress ProTM Web site:
http://www.mhdigitallearning.com
Funded by U.S. Department of Education
Additional Resources
National Centers
 National Center on Student Progress Monitoring (NCSPM):
http://www.studentprogress.org
 Research Institute on Progress Monitoring (RIPM):
http://www.progressmonitoring.org
 National Center on Response to Intervention (NCRI)
http://www.rti4success.org
Funded by U.S. Department of Education