Transcript STANDARDS-BASED ACCOUNTABILITY SYSTEMS: DESIGN ISSUES …

Making Consistent Decisions
About Accommodations for
English Language Learners
– Research Summit –
Texas Comprehensive Center @ SEDL
Austin, Texas
March 16–17, 2009
Language Factors in the
Assessment of English Language
Learners
Dr. Jamal Abedi
CRESST
University of California, Davis
School of Education
How Important is Assessment for
ELL Students?
Assessment results affect ELL students in the
following areas:
• Classification
• Instruction
• Accountability (the NCLB issues)
• Promotion
• Graduation
Are the Standardized Achievement
Tests Appropriate for ELLs?
The Standards for Educational and Psychological Testing
(American Educational Research Association, American
Psychological Association, & National Council on Measurement
in Education, 1999) elaborated on this issue:
For all test takers, any test that employs
language is, in part, a measure of their
language skills. This is of particular concern
for test takers whose first language is not the
language of the test.
Performance/Reliability-Gap Between ELL and
Non-ELL Students
Performance-Gap
Reliability-Gap
Reading
20% - 60%
15% - 40%
Science/Social
Sciences
Math Problem
Solving
Math
Computation
10% - 40%
12% - 35%
8% - 25%
10% - 30%
0% - 10%
10% – 15%
Are the Standardized Achievement Tests
Reliable and Valid for These Students?
• The reliability coefficients of the test
scores for ELL students are substantially
lower than those for non-ELL students.
• ELL students’ test outcomes show lower
criterion-related validity.
• Structural relationships between test
components and across measurement
domains are lower for ELL students.
Site 2 Stanford 9 Sub-scale Reliabilities
(Alpha), Grade 9
Non-LEP Students
Sub-scale (Items)
Hi SES
Low SES
English Only
FEP
RFEP
LEP
Reading, N=
205,092
35,855
181,202
37,876
21,869
52,720
-Vocabulary (30)
.828
.781
.835
.814
.759
.666
-Reading Comp (54)
.912
.893
.916
.903
.877
.833
Average Reliability
.870
.837
.876
.859
.818
.750
Math, N=
207,155
36,588
183,262
38,329
22,152
54,815
-Total (48)
.899
.853
.898
.898
.876
.802
204,571
35,866
180,743
37,862
21,852
52,863
-Mechanics (24)
.801
.759
.803
.802
.755
.686
-Expression (24)
.818
.779
.812
.804
.757
.680
Average Reliability
.810
.769
.813
.803
.756
.683
163,960
28,377
144,821
29,946
17,570
40,255
.800
.723
.805
.778
.716
.597
204,965
36,132
181,078
38,052
21,967
53,925
.803
.702
.805
.784
.722
.530
Language, N=
Science, N=
-Total (40)
Social Science, N=
-Total (40)
Principle of Equity and Fairness
• Therefore, the Principle of Equity
and Fairness demands assistance to
these students.
•Provide assistance in the form of
accommodations.
Samples of Accommodations Used for
ELL Students That May Not Be Relevant
•
•
•
Test-taker marks answers in a test booklet
•
Paper is secured to work area with
tape/magnet
•
Physical assistance is provided
Copying assistance provided between drafts
Test-taker indicates answers by pointing or
other similar method
Samples of Accommodations Used
for ELL Students
•
Enlarged answer sheets provided
•
Breaks provided
•
Tests individually administered
•
Tests administered in small groups
•
Tests administered in a location with
minimal distractions
Presenting Language-Related
Accommodations for ELLs
•
•
•
•
•
•
English Dictionary
English Glossary
Bilingual Dictionary/Glossary
Customized Dictionary
Native Language Testing
Linguistically Modified Test
Clear Language of Instruction
and Assessment Works for
ELLs, SWDs, and Everyone
• What is
language
modification
of test items?
Language of Assessment
• Clear and concise language is a requirement for
reliable and valid assessments for ELL students.
• It may also be an important consideration for
students with learning disabilities.
• Students with learning disabilities may have
difficulty processing complex language in the
assessment.
• Reducing the unnecessary linguistic complexity of
test items can help students with the most
challenging academic career.
Linguistic Modification
Concerns
• Familiarity/frequency of non-math vocabulary:
Unfamiliar or infrequent words changed
A certain reference file > Mack’s company
Original:
A certain reference file contains approximately six billion facts. About how many millions is
that?
A. 6,000,000
B. 600,000
C.
60,000
D.
6,000
E.
600
Modified:
Mack’s company sold six billion pencils. About how many millions is that?
A. 6,000,000
B. 600,000
C.
60,000
D.
6,000
E.
600
Linguistic Modification Concerns
Cont.
•
Length of nominals: long nominals shortened
Last year’s class vice president > Vice president
The pattern of puppy’s weight gain > The pattern above
•
Question phrases: complex question phrases changed to
simple question words
At which of the following times > When
Which is best approximation of the number >
Approximately how many
Linguistic Modification cont.
• Voice of verb phrase: passive verb forms changed to active
The weights of 3 objects were compared > Sandra compared the weights of 3 rabbits
If a marble is taken from the bag > If you take a marble from the bag
The boy was bitten by the dog > The dog bit the boy
• Conditional clauses: conditionals either replaced with
separate sentences or order of conditional and main clause
changed
If two batteries in the sample were found to be dead > He found two broken pencils in the sample
If Lee delivers x newspapers > Lee delivers x newspapers
CRESST Studies on the
Assessment and Accommodation of ELL
Students:
Impact of Language Factors On Assessment
of ELLs
A Chain of Events
Fourteen studies on the assessment and three on the
instruction (OTL) of ELL students
Examining Complex Linguistic Features
in Content-Based Test Items
Feature
Feature Description
Categories Combined
1
Item length
1, 2, 4, 45
2
Vocabulary
3, 26, 27
3
Nominal heaviness
5, 6, 29, 30, 31, 32
4
Verb voice
7, 33
5
Modal
8, 34
6
Relative clause
9, 10, 11, 35, 36, 37
7
Adverbial modification
12, 13, 14, 15, 16, 17, 38, 39, 40,
41
8
Conditional clause
18, 19
9
Complement clause
20, 44
10
Sentence structure
28, 42, 43, 46
11
Preferred argument structure
22, 23, 47, 48
12
Question form
21
13
Global difficulty
24
14
Content interest
25
Additional Complex Linguistic
Features
More recent research has identified these
additional features:
•
Complex verbs
•
Subordinate clauses (including relative clauses)
•
Complex noun phrases
•
Various entities as subjects
Studies on Linguistic Modifications
•
Results of national studies are not conclusive.
•
Most of the CRESST studies found significant gain for ELL
students on the linguistically modified version.
•
However, the outcome of national research on the impact of
linguistic modification is mixed (Francis, et al. 2006).
•
Factors that lead to such discrepancies include variation in
methodology in implementing linguistic modification
approach, sampling and power issues, variation in test items
and the nature of linguistic complexity, etc.
Improvements in the
Language of Instruction
•
Unnecessary linguistic complexity may impact
students’ understanding of instructional materials
•
Opportunity to learn for ELLs and SLDs may be
affected
•
Research show ELLs and SLDs benefiting less from
instructions with complex linguistic structure
Conclusions and
Recommendations
Assessments and instructions for ELLs :
•
•
•
•
•
Must be free of construct irrelevant factors
Must be free of unnecessary linguistic complexities
Must include a sufficient number of ELLs and SWDs in its
development process (field testing, standard setting, etc.)
Must be free of cultural biases
Must be sensitive to students’ linguistics and cultural needs
Impact of Linguistic Modification
Research on Practice
•
Many test publishers have incorporated the concept of
linguistic modifications in their test development process.
•
Among these test publishers: Educational Testing Service
(ETS), Harcourt Assessment, CTB/McGraw Hill, Measurement
Incorporated, etc.
•
Many states have adopted this approach into their
assessments
•
The U.S. Department of Education incorporated this process
into the National Assessments (NAEP) and has highly
recommended this approach to be incorporated into the state
NCLB Title I assessments.
For more information, please
contact Jamal Abedi at
CRESST/UC Davis:
(530) 754-9150
or
[email protected]