Transcript Slide 1

“She Was My Backbone”:
Measuring the Impact of
Literacy Coaching
Kelly Feighan, Research for Better Schools
Dr. Elizabeth Heeren, Memphis City Schools
http://www.ed.gov/programs/strivingreaders/awards.html
Grantee: Memphis City Schools
Memphis, Tennessee
Total Grant Award: $16,074,687
Memphis' Striving Readers project is designed to test the efficacy of the Memphis
Content Literacy Academy (MCLA) professional development model for
improving reading achievement and content literacy in high-need urban middle
schools serving grades 6-8. All content teachers in Striving Readers treatment
schools are eligible for participation in MCLA, which includes:
1. University Coursework (2 years:12 hours of upper division college credit)
2. Support from a site-based literacy coach
3. Access to differentiated instructional materials
Challenges in Hiring/Training
Literacy Coaches
1. No certification available in state of
Tennessee (and many other states)
2. Standards for coaching not set (we used
IRA and NCTE as a guide)
3. Roles are undefined and differ
depending on context
Qualifications We Used for Hiring
• 5+ years successful teaching in middle
school
• Advanced degree (Masters +)
• Experience with literacy
• Experience with professional development
• Various content areas
• Principal recommendations
Coaching Cycle
Coaches Report on
Their Roles in the
Schools
•Coaches consistently report
their role as: providing support,
advocating for teachers, and
modeling lessons.
•All coaches feel part of the
“school family.”
•The coaching experience has
improved with time.
•Main challenge: limited time to
observe CAP implementation
between assignments.
The data collection
tool used was
designed by the team
after one year of
coaching experience
in the grant. Slight
modifications were
made after beginning
use, and the tool has
now been used for 3
years.
Coaches establish rapport with teachers
95.2% of 62 teacher
survey respondents
reported “I can
confide in my
coach.”
• Trust between the
coach and teacher(s)
is critical:
• To the provision of CAP
implementation support
• Pre-conference meeting
• CAP Observation
– Videotapes for use to train
teachers, coaches, evaluators
– Co-teaching; modeling
• Post observation conference
• To the effective and strategic
selection of CRC & supplemental
resources
“She has been so available for me as a literacy coach… I don’t
know what I would have done without her assisting me…we’re
just basically working as a great team together.” (MSRP Focus
Group Report, 2007)
The Coaching Cycle:
1. Teacher attends CAP
modeling or discussion
2. Teacher discusses lesson
plan with coach
3. Coach observes teaching
rehearsal
4. Debrief/revise lesson plan
5. Coach observes
performance teaching
6. Final debrief
“Equipping middle and
high schools with trained
literacy coaches is at least
one line of attack to
combat “the quiet
resignation that seems to
pervade education
circles…that little if
anything can be done”
(Joftus, 2002, pg. 1).
Content Teachers Support Literacy Strategies
“I think literacy is so
important because no
matter what they do
or where they go they
are going to run into
something they have
to read… if they have
to apply for a job, it
requires them to be
able to read…” (2007)
Purpose of our Study
Impact:
Implementation:
Effect on
Student
Achievement
Effect on pedagogy
What coaches
do
Effect on student
achievement
Implementation Questions
• What daily tasks do literacy coaches typically perform at
the middle school level?
• How much of their time is spent involved in substantive
tasks that support teacher practice?
• To what extent do teachers perceive coaching services
as beneficial?
• What are some of the challenges that coaches face?
Impact Questions
• To what extent has working with a coach improved
teachers’ pedagogy?
– Do those who received literacy coaching report higher
frequency of strategy use?
– How prepared do teachers feel to use literacy
strategies in their content classes?
• $64,000 question: To what extent has literacy coaching
for teachers increased students’ academic
achievement?
Data Sources
• Measuring what coaches do:
– Coaches’ daily activity logs (N = 847)
– Teacher surveys (three waves: N = 48, 62, and 54)
– Teacher focus groups: four waves of 30 sessions
– Coach interviews (four waves with six coaches)
Data Sources
• Measuring teacher impact:
– Baseline and follow-up survey: MCLA completers and
control group
– Focus group interviews
– Program feedback survey (three waves)
– Follow-up checklist six months after program ended
• Measuring student impact:
– TCAP reading (Spring 2007 and Spring 2008)
– ITBS reading (Spring 2007 and Spring 2008)
Coach Log Analysis
• Coaches’ logs represented from 52% to 86% of their
190-day work year
• We entered a total of 5,791 individual records from 847
daily activity logs
• Tasks fell into 12 overarching categories including
observing, modeling, helping a teacher prepare for class
and administrative tasks or school-related activities
Time Spent with Teachers
• Two months into the 2007-08 school year, almost
60 percent of respondents reported that they had met
with their coaches more than four times.
• By spring 2008, three-quarters (75.9%) reported that
they had met with their coaches more than four times.
These figures are corroborated by data in the coaching
logs.
Coaching “Dosage”
• Analyses of logs showed that all (100%) MCLA
completers in two schools received high levels of
coaching assistance
• Approximately three-quarters (76.9%) received
high levels of coaching at the third school
• One-third (35.7%) of MCLA completers received
high levels of coaching at the fourth school
Teacher Surveys: Fall 2006, Fall 2007, and Spring 2008
Data source: RBS MCLA Feedback Surveys
Data source: RBS MCLA Feedback Surveys
Nature of Coach-Teacher Collaboration:
Focus Group Findings: Wave One
Teachers shared universally positive perceptions about
coaching support. Stated one teacher:
“She’s there. She’s not intrusive. She never comes off
as being a judge, a threat. If she comes in, and if she
sees something that wasn’t going like it should, she
would offer advice, tips, as opposed to ‘Well that wasn’t
right’ and leave. She would say ‘Maybe you should try
something like this.’ ”
Despite initial growing pains related to
scheduling issues, coaches were highly
valued:
“My coach tends to be hard to find sometimes… But
she’s very helpful I’ve always found… I’ve had some
struggles… being a first-year teacher, and she took time
out to help me plan a different lesson altogether, trying to
figure out how to teach them, how to write better
sentences… Actually it’s been a great resource even
though it does seem she’s stretched a little too thin.”
Second Wave of Focus Groups
• Although teachers issued strong praise of coaches, few
accepted her offer to model lessons because they did
not feel they needed it. One science teacher stated:
“She did ask… but I told her ‘No, no. Just go over what I
need to do and I’ll take care of it.’ ”
• A few teachers in a mathematics focus group said that
although their coach made them feel comfortable, they
did not “need” her to model a lesson because “she would
explain it so well in class.”
Third Wave of Focus Groups
• Strong praise for coaches: very helpful, approachable,
and committed to helping teachers succeed
•
Teachers said coaches “went out of their way” to supply
them with needed materials and resources, and cited
benefits from observation feedback
“She has been so available for me as a literacy coach…
I don’t know what I would have done without her
assisting me… We’re just basically working as a great
team together.”
Third Wave of Focus Groups
• Satisfaction with the coach’s accessibility increased
“I’m very impressed with [the coach] this year. Last year
we were trying to figure each other out (or her role or my
role or something), but this year– Well, every time I’d
look for her last year, she wasn’t around. I’d ask for
something and I couldn’t get it, but this year she is
dynamite.”
Fourth Wave of Focus Groups
• All nine focus groups held positive views about their
coach, and characterized them as very helpful
• Advice to others expecting MCLA at their school: avail
yourself of the literacy coach’s services
“Basically, the literacy coaches are there to help you
and sometimes we as teachers, as secondary teachers,
we don’t like to open our classrooms up to other people
to come in and show us things.”
Focus Group Summary
• Across the 30 focus group sessions, most respondents
described their coach as someone who "goes that extra
mile" to provide assistance, and one who showed
understanding and patience
• Many teachers shared examples of coaches’ dedication
• Initial concerns about accessibility and scheduling
conflicts dissipated over time
Coaches’ Challenges
• Helping teachers to see that literacy strategies
were not “add-ons”
• Limited opportunities to see student data in
schools with less principal support
• Learning to mentor on-the-job
• Fitting some literacy activities in with
mathematics
Impact on Teachers
• This study examines matched baseline and
follow-up survey data for 30 MCLA teachers
and 34 control group teachers
• Teachers were asked how prepared they felt
to use, and how frequently they used, 24
literacy strategies
Levels of “Preparedness” and Strategy Use
• No baseline differences in mean responses about
preparedness on 23 of 24 items
• Only one difference emerged: MCLA teachers had a
higher mean response (3.80) than control teachers
(3.12) on how prepared they felt to have students read
aloud for at least five minutes per period (F = 4.82, df =
62, p <.05)
• No baseline differences between control and MCLA
group with respect to reported frequency of strategy
use
Changes Over Time
• Paired t tests showed a significant increase in mean
responses for both MCLA and control group on most
preparedness items
• ANOVA results showed significant differences for
frequency of strategy use, favoring MCLA group:
• Showing relationships with graphic organizers
• Establishing purpose for reading text
• Modeling use of thinking maps
• Using cooperative learning groups
Six-month Follow-up: Fall 2008
• Surveys distributed to schools no longer
participating in MCLA asked teachers if they had
engaged in five specific literacy activities in the
past week
• Forty-two respondents had completed at least
one semester of MCLA
• 83 percent of these respondents identified an
MCLA activity they had used in the past week
Data source: Fall 2008 RBS WIS Checklist
Student Impact
• This study includes 3,612 with baseline and
follow-up TCAP test scores
• 1,830 students were linked to the 30 MCLA teachers
• 1,782 students were linked to the 34 control teachers
• Baseline mean number correct were higher in control
schools than MCLA schools (F = 5.44, df = 3411,
p<.05) but scale scores were not significantly different
TCAP Baseline Reading Scores
SCALE
SCORE 2006-07
*
[Standard Deviation]
Mean number of items correct ‘06-07*
[Standard Deviation]
CONTROL
(N=1,667)
MCLA
(N=1,745)
507.52
[32.5]
504.24
[31.8]
36.68
[11.0]
35.25
[10.4]
2.02
1.98
*(F = 5.44, df = 3411, p<.05)
Mean Performance Level
[Standard Deviation]
TCAP Follow-up Scores (N=3,520)
CONTROL
(N=1,735)
MCLA
(N=1,785)
SCALE SCORE 2007-08
[Standard Deviation]
518.19
[33.4]
514.21
[33.4]
Mean number of items correct ‘07-08*
[Standard Deviation]
36.11
[10.48]
34.66
[10.05]
2.08
[.56]
2.02
[.55]
* (F = 4.97, df = 3518, p <.05)
Mean Performance Level
[Standard Deviation]
ITBS Scores
CONTROL
MCLA
N = 899
N = 963
29.2
26.6
[Standard Deviation]
[15.76]
[15.32]
Number of students with follow-up score
N = 759
N=802
31.3
28.3
[16.84]
[15.83]
Number of students with baseline score
Total Reading NCE Mean Score
Spr. ’07
Total Reading NCE Mean Score
Spr. ’08
[Standard Deviation]
Overall Student Achievement Results
• Although TCAP scores were higher among control
students, the magnitude of the difference is very small:
two points
• ITBS scores followed the same pattern
• Initial ANOVAs and linear regression results did not
show a positive MCLA impact on tests scores; however,
more variables must be added to the model
• Did MCLA teachers have more challenging students?
Were there more behavioral problems in MCLA schools?
Qualitative Findings about Student Impact
• Teachers were generally optimistic that using literacy
strategies would improve student achievement
“My students have definitely improved a lot. I have a
couple of students who haven’t, but I’ve had students
who’ve already like, over two years of growth by their
mid-year assessment...”
“I think some of the strategies have given them– they
want to do things. They’re not as apprehensive as they
once were, especially when it comes to fluency.”
Qualitative Findings about Student Impact
• On average, teachers felt that learning the literacy
strategies helped most students to read better; however,
several expressed concern that “nonreaders” needed
additional help.
“Some have improved, but if they are nonreaders,
they’re still nonreaders. It did not help. But those that
were struggling, it gave them a different avenue to use, a
different method, a different strategy.”
Conclusions
• Although student-level results showed no effect
of MCLA on academic performance in one year,
teacher findings suggest an increased use in
literacy strategies
• Findings mirror those of other educational
researchers who have examined the impacts of
coaching models on teaching and learning
(Murray, Ma, and Mazur, 2008)
Contact Us:
• Kelly Feighan: [email protected]
• Elizabeth Heeren: [email protected]
• Or visit our website at http://www.rbs.org/msrp