Transcript PPT - Michigan State University

Assessing K-12 Students’ Learning
Progression of Carbon Cycling Using
Different Types of Items
Jing Chen
Charles W. Anderson
Michigan State University
Jinnie Choi
Yong Sang Lee
Karen Draney
University of California, Berkeley
•analyze how well ordered multiple choice (OMC), multiple
True or False (MTF) items differentiate students among
achievement levels by comparing students’ responses to
these items to their responses to the same problems set in
open-ended (OE) formats
• inform the development of OMC and MTF items to measure
students’ achievement levels more reliably
OMC and MTF items
• OMC item-- each of the possible answer choices is linked to a particular
developmental level of student understanding for the construct being
measured. (Briggs, Alonzo, Schwab, & Wilson, 2006).
• MTF item-- a set of true or false questions asks students to judge True or
False.
PURPOSE OF THIS STUDY
PURPOSE
• How well can our mixed types of items assess
students’ performances?
RESEARCH QUESTIONS
• How well can OMC and MTF items diagnose students’
achievement levels?
LEVELS
Explaining Progress
variable
LEVELS
Naming Progress Variable
4
Linking processes with matter
and energy as constraints
4
Scientific statements
3
Changes of Molecules and
Energy Forms with
Unsuccessful Constraints
3
Scientific words of organic molecules, energy forms,
and chemical change
2
Force-dynamic accounts with
hidden mechanisms
2.5
2
1
Macroscopic force-dynamic
accounts
1.5
1
• Related studies
Easier scientific words with mixed meanings
Hidden mechanism words
Easier hidden mechanism words
Words about actors, enablers, and results
• Test scores based on OMC items have greater validity than test scores based on TMC
items, without sacrificing reliability. There a weak to moderate positive correlation
between students’ scores on OMC items and their scores on traditional MC items (Briggs,
Alonzo, Schwab, & Wilson, 2006).
• Compare to OE items, OMC items appear to provide more precise diagnoses of students’
learning progression levels and to be more valid, eliciting students’ conceptions more
similarly to cognitive interviews (Alonzo & Steedle, 2008)
• Inconsistency in students’ responses to items in different formats, students performed
differently on multiple-choice and short-answer items (Steedle, 2006)
THEORETICAL FRAMEWORK
• Carbon cycle learning progression framework
• students in grade 4 to 12
• 1044 assessment (454 pre, 550 post, 181 elementary, 377 middle, 446
high)
• collected during 2008-2009
Assessments
• 48 items in total
• include 8 OMC, 8 MTF items and their paired OE items in this study (32 in total)
• matter energy transformation in 5 macroscopic events and large scale events.
RESEARCH METHOD
Participants
• OMC items
 cross-tabulation
 Spearman rank-order correlation between OMC and OE levels
• MTF items
 cross-tabulation
 point-biserial correlation between MTF and OE responses
• For all items
 IRT partial credit model; IRT analysis
 qualitative evaluation by a group of researchers
RESEARCH METHOD
Data Analysis
A loaf of bread was left uncovered for two weeks. Three different kinds of mold grew
on it. Assuming that the bread did not dry out, which of the following is a reasonable
prediction of the weight of the bread and mold together?
A) The mass has increased, because the mold has grown. (Level 1)
B) The mass remains the same as the mold converts bread into biomass. (level 2)
C) The mass decreases as the growing mold converts bread into energy. (level 2)
D) The mass decreases as the mold converts bread into biomass and gases. (level 3)
OE response levels
OMC response levels
1 (A)
2 (B,C)
3 (D)
0
1
3
4
1
42
15
2
2
1
24
32
3
0
0
12
RESULTS OF OMC ITEMS
BREAD ITEM
OMC
ITEM
PAIRED
OE
SAMPLE
SIZE
CORRELATI
ON
WTLOS
WTLOS_N
306
.272**
WTLOS_E
306
.342**
BODYTE_N
138
.313**
BODYTE_E
138
.476**
BREAD_N
137
.581**
BREAD_E
137
.651**
MATCH_N
283
.217**
MATCH_E
283
.258**
LEAVES_N
306
.300**
LEAVES_E
306
.320**
STOREN
STOREN_E
287
.360**
TROP
TROP_E
269
.411**
LIGHT
LIGHT_E
289
.219**
TOTAL
TOTAL_N
.654**
TOTAL_E
.755**
BODYTE
BREAD
MATCH
LEAVES
** correlation is significant at the .01 level (2-tailed)
• There are weak to moderate
positive correlations between
students’ OMC levels and their
OE levels.
• The correlation between
overall OMC scores and
overall OE naming scores is .
654, and the correlation
between overall OMC scores
and overall OE explaining
scores .755.
RESULTS OF OMC ITEMS
Correlation between OMC scores and paired OE item scores
OMC (5 items
administrated at high
school form C)
OE (5 paired items
administrated at high
school form C)
Cronbach’s alpha
(N of cases =138)
.441
.704
Expected reliability for a
longer test (using
spearman-brown
prophecy formula)
0.703
(a test with 15 OMC
items)
RESULTS OF OMC ITEMS
Reliability of OMC and OE items
• Compare the item
difficulties for 8 OMC
and 8 paired OE items
(both naming and
explaining), OMC
items are easier than
OE naming, which is
easier than OE
explaining.
Test information comparison (one test includes 8 OMC items ******
only,
the other included 8 paired OE items only)
• The test that contains
8 OE items will get
more information at the
higher ability range
compare to a test that
includes 8 paired OMC
items.
RESULTS OF OMC ITEMS
Item difficulties for OMC items, paired OE items (naming and explaining)
• Students’ choices to “sunlight”, “air”, and “plant create their own energy” do not have
significant correlation to their levels in the paired OE item.
• Students who circled N for water (water0), N for nutrients (nutrients0) are more likely to
be at higher levels compare to students who circled Y for these 2 indicators.
RESULTS OF MTF ITEMS
PLANT ENERGY: Which of the following are sources of energy for plants? Circle yes or
no for each of the following:
a). Water
Yes / No
b). Light
Yes / No
c). Air
Yes / No
d). Nutrients in soil
Yes / No
e). They make their own energy
Yes / No
Explain what you think is energy for plants
MTF ITEMS
A
B
C
D
E
F
G
TREEG ROWTH
SUN
WATER
AIR
SOIL
HUMAN ENERGY
SUN
WATER
NUTRIENTS
FOOD
EXER
CO2
O2
N
N
Y
N
N
N
SUN
WATER
NUTRIENTS
OWN
ENERGY
PLANT ENERGY
AIR
N
HUMAN GROW
RUN
.
N
SUN
WATER
AIR
FOOD
.
FOOD
WATER
AIR
ENERGY
SLEEP
SOIL
PLANT
CO2
INSECT
ENERGY
HEAT
WATER+
GAS
WASTE
OTHER
MATERIALS
Y
CARBON
ANIMAL WINTER
Y
STONE
WARM
RUN
CLOTH
FOOD
Y
RESULTS OF MTF ITEMS
MTF ITEMS (INDICATORS)
RESULTS OF ALL ITEMS
• The classical item discrimination index shows the correlation
between students’ scores on this item and their total scores.
There is a moderate to strong correlation between students’
score on most of the items to their total score.
• The red dot items are the OMC items. They generally show
lower correlation between students’ scores on the OMC item and
their total scores compare to the OE items.
• All the items are fitting well to the model (for each item, the weighted MNSQ
is within the range from .77 to 1.33).
• This graph indicate that our assessment items can fairly differentiate
students with different abilities among levels.
RESULTS OF ALL ITEMS
Average person location for each score
• OMC items are generally easier than OE items, OMC items do not
measure students at high ability levels as precise as OE items do
• OMC items have weak to moderate correlation with OE items that
measure the same content
• lower reliability compare to OE items.
• The choices in many MTF items did not associate with students’ levels in
OE format
• The entire assessment differentiate students among levels well
Implications
1. Though it’s hard to write OMC options at higher achievement levels without
using “science-y” terminologies, we still need to develop options for students at
high level.
2. The design of MTF items need to be informed by more research. So the options
should be better indicators that actually differentiate students among levels.
3. The number of OMC items in a test should be relatively large in order to get
equivalent amount of test information (test reliability) compare to a test with OE
items or with mixed types of items.
CONCLUSIONS and IMPLICATIONS
Conclusions
Thank you for your attention!
a
b
c
d
1
8
5
11
17
2
3
1
65
10
3
0
0
10
1
4
0
0
1
0
•OMC, MTF items are informed by educational research, options in OMC
items represent typical students’ understandings, from the least to most
scientifically sophisticated accounts, found in science education research.
• OMC and MTF items are more diagnostic compare to traditional MC and T
or F items, can be used to provide teachers quick feedbacks.
• however, OMC and MTF items do not give students opportunity to create
their own responses
• given the advantages and limitations of the OMC and MTF items, the
analysis of how well the OMC and MTF items can diagnose students’
achievement levels can direct the future use of OMC and MTF items.
WHY STUDY THIS?
• OMC and MTF items can be more easily used in large-scale assessments
compare to OE items