Transcript Boletim da Escola - NAEP State Analysis

School-level Correlates of
Achievement: Linking NAEP,
State Assessments, and SASS
NAEP State Analysis Project
Sami Kitmitto
CCSSO National Conference on LargeScale Assessment
June 2006
Overview of the Study
Create a valuable data set for policy analysis by adding
achievement scores to a comprehensive school
survey
 School and Staffing Survey (SASS)
 Extensive information from a national survey of
schools, but no achievement scores
 National Assessment of Educational Progress (NAEP)
 Nationally representative scores comparable between
states
 State Assessment Database (NLSLSASD)
 Collection of all available school-level state
assessment data
 Scores comparable within states
Research Questions
 What are the important school
characteristics that correlate with
achievement?
 Do the results of Don McLaughlin and Gili
Drori (2000) compare to the results from a
larger and more recent set of data?
 2000 SASS vs. 1994 SASS
 36-38 states vs. 20 states
Data Assembly
NAEP Data
NAEP 1998, 2000 and 2002
 Used 2000 Math Grades 4 & 8 and 1998 &
2002 Reading scores for Grades 4 & 8
 Used full population estimates
 Mean and standard deviation at the school
level
 Mean and standard deviation at the state
level
 Replicate weights used
Data Assembly
NLSLSASD 2000 Data
NLSLSASD 2000
 Selected two scores for each grade/subject:
 Grade 4 Math, Grade 4 Reading
 Grade 8 Math, Grade 8 Reading
 Remove between state variation
 Create standard score within each state:
X is
SchoolMeanis  StateMeans

StateStd s
Data Assembly
NAEP and NLSLSASD School-Level
NAEP and NLSLSASD Correlation
 Using only schools in both NAEP and
NLSLSASD:
 Calculated correlation between NAEP and
NLSLSASD scores at the state level for
matched schools
Data Assembly
NAEP State-Level and NLSLSASD
 Used NAEP to introduce between state
differences and variation to standardized
scores
 Yis  ( X is  NaepStd s  NaepSaCorrs )  NaepMeans
 Rescaled to mean of 50 and standard
deviation of 10

(Yis  Mean(Y ))  10
ACHIEVEis  50 
Std (Y )
Data Preparation Step 2
SASS 2000
School Level Information
 From school, principal, teacher and district
surveys
 Social Background
 Organizational Characteristics
 School Behavioral Climate
 Teacher Characteristics
Data Set Used for Analysis
Analysis Sample
 Dropped schools with less than 50 students
 Did not include schools that were combinations of
elementary, middles and or high schools
 Missing values: list-wise deletion of observations
Teacher Qualifications Dropped
 Teacher sample is not random or representative at
the school level
 High percent of variation was within schools not
between schools
 Results indicated that these measures were mostly
noise
Data Numbers
Number of Schools With Two Valid Scores
NAEP/
NLSLSASD
Elementary School
Middle School
SASS Schools
# Schools
# Schools
# States
Math
Reading
34,106
34,099
2,287
2,273
38
37
Math
17,524
1,414
38
Reading
15,707
1,333
36
Number of Schools in Analysis Sample
# Schools
Elementary School
Middle School
Math
1,885
Reading
1,883
Math
723
Reading
698
Analysis Methodology
Structural Equation Modeling
 Similar to multiple regression analysis
 Allows for multiple measures of concepts
 Models measurement error
 Observed variables = Measures
 Conceptual factors = Latent Variables
Model
Path Model Relating Latent Variables
Limited
English
Proficiency
Poverty
Race
School Size
Class Size
Normative
Cohesion
Teacher
Influence
Student
Behavioral
Climate
Student
Academic
Achievement
Model
Measurement Model
% Free Lunch
Eligible
Poverty a
Problem
Poverty
% LEP
% Non-White
Limited
English
Proficiency
Race
Average Class
Size
Enrollment
School Size
Clear Norms
Parcel
Class Size
Student/
Teacher Ratio
Teacher
Attitudes and
Opinions
Class Size a
Problem
Normative
Cohesion
Student
Behavioral
Climate
Cooperation
Parcel
Climate
Problems
Parcel #1
Teacher
Influence
Climate
Problems
Parcel #2
Influence on School
Policies Parcel
Student
Academic
Achievement
Score #1
Control of
Classroom
Parcel
Score #2
Replication Results
Fit Statistics
Elementary School
Math
Reading
GFI
AGFI
RMR
Chi-Square
Chi-Square DF
RMSEA Estimate
90% Lower Limit
90% Upper Limit
Bentler's CFI
0.976
0.948
0.029
381
63
0.052
0.047
0.057
0.979
0.973
0.941
0.032
446
63
0.057
0.052
0.062
0.978
Middle School
Math
Reading
0.973
0.942
0.030
156
63
0.045
0.037
0.054
0.986
0.973
0.942
0.030
149
63
0.044
0.035
0.053
0.988
Replication Results (cont)
Estimated Coefficients for Achievement Equation
Elementary School
Math
Reading
Class Size
School Climate Problems
Normative Cohesion
Teacher Influence
School Size
Poverty
Race
Limited English
R-squared
-0.230
-0.257
0.149
-0.017
0.011
-0.473
-0.184
0.068
0.625
*
*
*
*
*
*
-0.242
-0.085
0.039
0.022
-0.052
-0.346
-0.347
-0.032
0.533
Middle School
Math
Reading
*
*
*
*
-0.181 *
-0.743
0.153
0.038
0.042
-0.324
-0.106
0.106 *
0.738
-0.487
0.082
-0.125
0.120
0.033
-0.415
-0.455
0.082
0.637
*
*
*
*
Interpretation of Coefficients
 Latent variables are scaled to one of their measures
 ‘Class Size’ is scaled to student/teacher ratio
 Coefficients are standardized
 A one standard deviation increase in ‘Class Size’
is correlated with a -.23 standard deviation
difference in math achievement in elementary
schools
 Standard deviation of student/teacher ratio in the
sample is ~ 4 students/teacher
 Mean is 15.5 students/teacher
Literature on ‘Class Size’
Reported Estimated Effects of
Student/Teacher Ratio and Class Size
Variable
Student/Teacher
Ratio
No Effect or Not
Significant
Small Effect
Sizeable Effect
Hanusheck 1986
Prais 1996
Hedges and Greenwald 1996
Nye, Hedges &
Konstantopoulos 2000
Krueger and Whitmore 2001
Ferguson 1991
Coates 2003
Krueger 1999
Eide & Showalter 1998
Todd and Wolpin 2004
Class Size
Hoxby 2000
Boozer & Rouse 1995
Angrist & Lavy 1999
Fertig & Wright 2005
Avenues for Future Research
 Add principal responses to school climate questions
 Add additional controls: urbanicity, % IEP, magnet
school indicator
 ‘Principal Leadership’
 ‘Resources’
 Per pupil expenditures (district level)
 Number of computers
 ‘Parent Involvement’
 Teacher and principal reports of parent
involvement being a problem
 School programs to involve parents
Conclusions
 Linking NAEP, NLSLSASD and SASS provides a
powerful national sample of schools matched to
achievement scores
 SASS provide multiple measures of key
conceptual factors
 SEM provides a methodology to take advantage
of the depth of SASS information
 Class size found to be correlated with achievement
 In middle schools, more important for reading
than math
 Results on achievement are similar to McLaughlin
and Drori 2000 with improved fit