Transcript Duvall Reviewing the Indep Fed Eval of RRX

REVIEWING THE
INDEPENDENT FEDERAL
EVALUATION OF READING
RECOVERY
Cathleen Duvall
Dr. Mary Jackson
November 17, 2016
WHY THIS TWOSOME
FROM TEXAS
 We’re retired!!
 25 year implementation of Reading Recovery in a
large school district
 Compulsive about quality implementation over
time
 One of the largest implementations in North
America
 We’re huge fans!!!!
WHY THIS REPORT
ROCKS!!!!
 We didn’t do it!!! Independent external evaluation over
multiple years
 Nothing tepid about the methodology! Rigorous
experimental, quasi-experimental design along with
qualitative study
 No small study!
 Reading Recovery rocks! Strong evidence of the effectiveness
of Reading Recovery!
 Not just about us!!! Contribution that this study of Reading
Recovery makes to the field of research on the effective
implementions of innovations
WHY THIS REPORT ROCKS
FOR IMPLEMENTERS
o Great public relations tool
o Informs our practice
o Has potential to make us
even more successful
CONCLUSION
“One of the most ambitious
and well-documented
expansions of an
instructional program in
U.S. history and it was
highly successful” p. 4
Despite Challenges…..
Capacity
Fiscal climate
Scaling up with fidelity
Scaling up with sustainability
PROGRESS TOWARD SCALE-UP GOALS
GOAL
SCALE-UP
TARGET
TOTAL
% OF GOAL
MET
RR Teachers
Trained
3,675
3,747
102
Teacher Leaders
Trained
15
46
306
Students Served 67,264
with RR Lessons
62,000
92
Other Students
Served by RR
Teachers
325,500
108
302,688
Research: Immediate Impacts
1. Estimate the immediate impact of Reading Recovery
at the end of the 12 to 20 week intervention on
students’ reading achievement, as measured by the
Iowa Test of Basic Skills (ITBS)
2. Estimate the impacts of Reading Recovery on
Comprehension and Reading Words subscales of the
ITBS as well as the Observation Survey.
3. Estimate the impacts of Reading Recovery on two
subgroups of particular interest under the i(3) scaleup: English language learners and students in rural
schools
IMMEDIATE IMPACTS OF
READING RECOVERY
ACTIVITY
• Read Page 44
• Highlight important findings
• Share with a partner
one PR message you
could use from these findings
READING RECOVERY
WORKS!!!!
Research: Sustained Impacts
CONCLUSION
“At this time there simply are not
enough students with 3rd grade state test
scores recorded to produce a precise
impact estimate or perform subgroup
analysis, and the true sustained impact
may very well be quite substantial.” p. 67
Implementation Fidelity
•
integral component of program evaluation;
• important in understanding how effects were
achieved;
•
adherence to or departure from program’s model
important in providing context when program
effects are absent.
Implementation Fidelity Research
Model
Implementation Fidelity Findings
CHALLENGES
DEVIATIONS
SUCCESSES
Fiscal Climate
Commitment to
Implementation
Training Met
Costs of
Implementing
Communication Met
Trainer Support for
Teacher Leaders
Ongoing PD for
Trained Teachers
Student Selection
Timing of Lessons
Started
Lesson Delivery Met
Conclusions: Implementation
Fidelity
“A consistent finding is that Reading
Recovery was implemented with high fidelity
to the program model over the course of the
i3 scale-up”
“All in all, this finding suggests that the
impacts we observed were, indeed, the result
of faithful implementation to the intervention
and that the i3 scale-up successfully replicated
Reading Recovery in the schools involved in
the expansion”
CPRE/CRESP Report, p.82
Chapter 5
Lesson-level Implementation
Key finding of randomized experiment:
“While all Reading Recovery
students outperformed control
students, some students made
especially large gains.” p.83
Data Sources for the Research
on Reading Recovery Instruction
Interviews: 349 over 4 years
1. University trainers
2. Reading Recovery teachers
3. i3 project staff & OSU leadership
4. Teacher leaders 5
5. First grade classroom teachers
6.
Principals
Data Sources for the Research
on Reading Recovery Instruction
Annual surveys – 8350 over 4 years
1. Reading Recovery teachers
2. Teacher leaders
3. Site coordinators
4. 1st grade teachers
Data Sources for the Research
on Reading Recovery Instruction
Focus Groups in year 1 & 4
83 teacher leaders total
Instructional strength survey
123 total
Field-based lesson observations
108 over 4 years
Student record reviews
50 total
Can you say depth & breadth?
Instructional Strength an
Emerging Theme
“…respondents consistently
identified the quality of
individual Reading Recovery
teachers’ instruction as a critical
and variable determinant of
student learning.” p. 88
Instructional Strength in
Reading Recovery
DELIBERATENESS
an encompassing commitment
to thoughtful practice
Deliberate teachers engage in:
1. purposeful analysis of students’ progress that is
guided by close, carefully documented observation;
2. ongoing reflection on their own instruction; and
3. active engagement with their own continual
learning, both individually and through participation
in a community of practice. p. 92
INSTRUCTIONAL DEXTERITY
flexible application of deep skill
These teachers exhibit:
• a supportive rapport that
continually pushes the
student toward maximal
growth;
• in-the-moment decision-making that draws on both
prior understandings and real-time observations;
• judicious use of language;
• a sense of urgency that is evident in the pace of the
lesson and the efficiency of the instructional moves.
ATTITUDES & DISPOSITIONS
the building blocks of strength
These teachers demonstrate:
1. openness to change;
2. excellent interpersonal
skills;
3. a strong work ethic;
4. belief in the ability of
all students to learn.
School-Level Support for
Strong Instruction
___ I prioritize Reading Recovery.
___ I select strong teachers for
Reading Recovery training.
___ I collaborate with the teacher
leader to ensure teachers receive the
support they need to develop their skills.
___ I protect Reading Recovery teachers’ lesson schedules.
District-Level Support for
Strong Instruction
District leaders who are knowledgeable about literacy
instruction generally, and Reading Recovery in particular
1. work to protect Reading Recovery lesson time;
2. draw on the resources of teacher leaders; and
3. ensure communication in support of Reading Recovery
students.
In summary…
In the case of Reading Recovery, we find,
principal prioritization
and district-level protection of the
intervention are vital
and necessary supports.”
p. 106
School Level Implementation:
Case Study Research
GOAL:
To explore processes and successes of
implementation at the school level, particularly as a
way to complement our work to understand program
processes and successes at the lesson level;
To better understand how the program is
practiced in schools and districts and what factors
shape and mediate its enactment within and across
specific school contexts
CPRE/CRESP Report, p. 107
CONTEXT IN WHICH READING RECOVERY
EXISTS: WHY DOES IT MATTER?
• Cannot thoroughly understand Reading Recovery
without attention to school level enactment;
• Implementation is a recursive process of action,
interaction and interpretation in context”(p.108);
• Dr. Marie Clay recommended a system-wide view
of the program-paying particular attention to how
it fits in each setting;
• RRCNA describes Reading Recovery as a systemic
intervention
Important Contextual Factors Identified
• Type of relationships formed around Reading
Recovery
• Level of program understanding by those outside
of it
• Depth of commitment by teachers and
administrators
• Extent Reading Recovery was part of a school’s
broader approach to literacy
Issues of Understanding/
Issues of Commitment
ISOLATED
INTERVENTION
CENTERPIECE OF
COMPREHENSIVE
RTI PROGRAM
SCHEMAS OF IMPLEMENTATION
ISOLATION
OBSTRUCTION
ENDORSEMENT
INTEGRATION
PARTICIPANT ACTIVITY
LINE UP DESCRIPTORS UNDER
APPROPRIATE SCHEMA
EXAMPLE:
INTEGRATION
“SCHOOL-WIDE SHARED UNDERSTANDING OF RR”
Main Factors Influencing the Impact and
Sustainability of Reading Recovery: A Common
Perspective
Important Phenomena
COMMUNICATION
ABOUT READING
RECOVERY
PRINCIPAL ENGAGEMENT
COMMUNICATION
LACK OF COMMUNICATION
INTEGRATED
COMMUNICATION
NO SENSE THAT THE PROGRAM HAS
VALUE IN THE BUILDING
COMMUNICATION HIGH/COMMITMENT
HIGH
NARROW SCOPE OF COMMUNICATION
WIDE SCOPE OF COMMUNICATION
NO SENSE OF POSITIVE RELATIONSHIPS
ACTIVE COMMUNICATION BETWEEN
CLASSROOM AND READING RECOVERY
TEACHERS WHICH LEADS TO
COLLABORATIVE EFFORTS TO IMPROVE
INSTRUCTION
ROLE CONFUSION
LACK OF INFORMATION LEADS TO
IMPLEMENTATION STRATEGIES THAT
THREATEN FIDELITY
TEACHERS UNDERSTAND HOW TO
PARLAY THEIR UNDERSTANDING
ABOUT READING RECOVERY INTO
SCHOOL IMPROVEMENT
PRINCIPAL ENGAGEMENT
LOW ENGAGEMENT
INTEGRATED ENGAGEMENT
ROLE IS LIMITED TO PROVISION OF
SPACE/MATERIALS
VISIBLE AND VOCAL IN THEIR
ENTHUSIASM ABOUT READING
RECOVERY
LACK OF PARTICIPATION IN DECISIONS
RELATED TO STUDENT SELECTION, RR
TEACHER SELECTION, ETC.
NO EXPECTATION FOR TEACHERS TO
INTERACT
POSITIONS READING RECOVERY
TEACHERS AS LITERACY EXPERTS AND
LEADERS
PUTS STRUCTURES IN PLACE SUCH AS
SET-ASIDE TIME, CREATING TIME AND
SPACE
ACTIVELY PARTICIPATES IN DECISIONS
RELATED TO IMPLEMENTATION
IMPORTANT LESSONS
CLOSING ACTIVITY
SHARE A
LEARNING/COMMITMENT WITH
A PARTNER
CONTACT INFORMATION
Mary Jackson, Ed.D.
[email protected]
Cathleen Duvall
[email protected]