Transcript A Case Study of Math for America Los

Strategies for Incorporating Evaluation
into Teacher Scholarship Programs:
A Case Study of Math for America Los
Angeles Teacher Fellows Program
Pam Mason
Executive Director, MƒA LA
Rebecca Eddy
President, Cobblestone Applied Research & Evaluation, Inc.
www.mathforamerica.org/la
A Partnership of
Mathematicians,
Business People
and Educators
The mission of Math for America is to
improve mathematics and science
education in US public secondary schools by
building a corps of outstanding STEM
teachers and leaders. Our goal is to support
those outstanding STEM teachers already in
the classroom and to increase the number
of mathematically talented individuals
entering teaching.
In 2008, The University of Southern California (USC), Claremont
Graduate University (CGU) and Harvey Mudd College (HMC) teamed
together with MƒA to create Math for America Los Angeles (MƒA LA).
2008 Cohort
2008 - 2012 Cohorts
To improve math
education in our public
schools by attracting,
training, and retaining
highly qualified
secondary school
mathematics teachers.
To provide the support
teachers require to
answer a critical need
and bring about a real,
sustained change.
2012 Master Teacher Cohort at PCMI
• Ongoing classroom observations
and instructional support by
experienced advisors
• Social and network building
opportunities
• Monthly Professional
Development
• Curriculum and Teaching
Support
• National Board Certification
Clusters of Fellows at
individual schools create
effective learning
environments for students
Hawkins High School
Montclair High School
Miguel Contreras
Learning Complex
Alain Leroy Locke Senior High School
Green Dot Public School
James Monroe High School
Helen Bernstein High School
Fairfax High School
Los Altos High School
East Los Angeles
Performing Arts Academy
Ramon C. Cortines
High School
Professional Development Outreach
Program Designed by
 Pam Mason,
MƒA LA Executive Director
 Darryl Yong,
Associate Chair of
Mathematics at
Harvey Mudd College
Relevant, “Just in Time”
Professional Development
MƒA LA Fellows have attended
regional and national
conferences for mathematics
teachers and have given
formal presentations.
Conferences attended during the 2012-2013 school year:
• CMC South Conference in Palm Springs
• CMC North Conference in Asilomar
• NCSM Conference in Denver
• NCTM Conference in Denver
Photo by Solares Photography
Reprinted with permission from the National Council of Teachers of Mathematics
A tightly networked community
•Regular meetings
•Experienced coaches
•Social events
The MƒA LA Effect …
ACCORDING TO MƒA LA COACHES
“… a feeling of
camaraderie …”

Purpose of Evaluation
Setting goals
 Identifying indicators to measure program
implementation and outcomes
 Keeping the program on track
 Drawing conclusions about the merit of the
program




Evaluation designed to answer key
evaluation questions
Uses a combination of qualitative and
quantitative methods
Quasi-experimental design to compare
students of MfA LA Fellows vs. students
on non-Fellows in the same schools
Evaluation Question
To what extent is the implementation of MfA LA program
activities occurring as planned?
How well have Fellows’ students performed on mathematics
achievement outcomes such as CST scores compared to
other students?
Do students of Fellows show any additional benefits from
being in their classrooms such as course-taking patterns or
attitudes about mathematics in comparison to students of
non-Fellows?
Are there differences between the Fellows’ experiences for
those at cluster sites versus individual sites?
Is there evidence that the MfA LA program improves teacher
recruitment, quality, and retention in the teaching
profession?
What are the strengths of the MfA LA program?
What are areas of improvement for MfA LA?
Math for America Los Angeles is committed to a
comprehensive evaluation of its programs and operations
Purpose
Track program
implementation and
measure outcomes
Design
Mixed method: Quantitative and
qualitative data sources to
answer evaluation questions
Matched control group quasiexperimental design for student
achievement data
Evaluation Activities
Focus groups, interviews,
classroom observations,
document analysis, assessment
of student test scores & surveys
•
•
What is a logic model?
Common elements in a logic model:
•
•
•
•
Inputs
Activities
Outputs
Outcomes
“Logic models help us plan, implement, evaluate, and
communicate more effectively.” Taylor-Powell, E., & Henert, E.
(2008)
•
University of Wisconsin – Extension:
templates, examples
http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html
•
W. K. Kellogg Foundation: Development
Guide
http://www.wkkf.org/knowledge-center/resources/2006/02/wk-kelloggfoundation-logic-model-development-guide.aspx
H
E
A
D
A
C
H
E
Situation
Get Pills
Take Pills
INPUTS
OUTPUTS
Feel Better
OUTCOMES
MfA LA Teacher Fellows Program Logic Model
Outputs
Inputs
What is invested
in
MfA LA
Situation
Students
underachieve in
math; U.S.
needs STEM
workforce
to be
globally
competitive
Staff
Exec.
Director
Coaches
Financial
Support
(Grants,
Donors)
Time
Cred.
Program
Activities
What MfA LA
does
Fellows receive
stipends +
tuition support
Fellows receive
coaching from
mentors
Fellows receive
prof. support
through PD
Fellows receive
moral/
emotional/
instructional
support from
other Fellows
Fellows teach in
classrooms/
work in math
depts
Participation
Who MfA LA
reaches
Fellows
receive training
Teachers at
Fellows’
school
collaborate
with Fellows
Students
receive math
instruction
from trained
Fellow
Schools
receive trained
teachers/
leaders
Districts
receive trained
teachers
Assumptions


Highly qualified math teachers produce high achieving
students
Recruiting, training & supporting highly-qualified
individuals will produce highly qualified teachers who
stay in the profession
Outcomes
Short-term
Fellows have
improved
pedagogical
skills
Med-term
Fellows
continue to
teach in
schools
Fellows maintain
content
knowledge in
math
Students
acquire strong
math skills
Fellows feel
supported &
respected as a
teacher
Students
maintain
achievement &
interest in
math/ other
STEM fields
Fellows are
more confident
to lead in
classroom & in
department
National Board
Certification &
continued
professional
growth


Long-term
Fellows
retained in
math
profession
LA schools/
districts have
highly qualified
math teachers
LA schools/
districts have
high achieving
math students
Students from
LA competitive
in global
economy
External Factors
Students in high poverty schools in LA less
likely to receive instruction from highly
qualified teachers
Students in high poverty schools in LA less
likely to perform well in mathematics
Outputs
Need
Inputs
What needs are we
addressing?
Teachers lack support
and professional
development experiences
that prevents recruitment
& retention of highquality mathematics
secondary school
teachers
Activities
Participation
Product
Outcomes
Short-term
Long-term
Outputs
Need
Inputs
What
resources do
we need for
the program?
Director/ staff
expertise, funding
Activities
Participation
Product
Outcomes
Short-term
Long-term
Outputs
Need
Inputs
Monitoring
Implementation
Activities
Participation
Product
What activities are
planned to address the
needs?
Who will participate?
Fellows attend monthly PD
sessions 10 months per year
& attend professional
conferences; receive
coaching support; stipends
Outcomes
Short-term
Long-term
Outputs
Need
Inputs
Activities
Participation
Product
Determining
Program Merit
Outcomes
Short-term
Long-term
What are the shortterm and long-term
changes we expect to
see in participants?
Fellows acquire content
knowledge, classroom
management & planning
skills (short-term)
Fellows are retained in the
profession & are effective
secondary math teachers
(long-term)

See the following list and determine in
which category each applies

For example: Bake cookie dough
 Is this a need, activity, outcome, etc.?
1. Start with a need
2. Identify one activity (Outputs)
• Specify how much of that activity needs to
happen to address the need (Activities)
• Establish who will be reached by the activity
and what is required of their participation
(Participation)
3. What resources are required? (Inputs)
4. What is your goal for meeting the needs?
(Outcomes)


What are some common needs that you
identified?
In thinking about designing an
evaluation, what are some indicators that
we will track?



Outputs
Outcomes
How can we design an evaluation to
effectively address the key evaluation
questions and include the indicators for
implementation and outcomes?
Evaluation Question
To what extent is the
implementation of MfA LA program
activities occurring as planned?
How well have Fellows’ students
performed on mathematics
achievement outcomes such as CST
scores compared to other
students?
Do students of Fellows show any
additional benefits from being in
their classrooms such as coursetaking patterns or attitudes about
mathematics in comparison to
students of non-Fellows?
Are there differences between the
Fellows’ experiences for those at
cluster sites versus individual sites?
Is there evidence that the MfA LA
program improves teacher
recruitment, quality, and retention
in the teaching profession?
What are the strengths of the MfA
LA program?
What are areas of improvement for
MfA LA?
Answer
Implementation: Main activities were implemented as planned (e.g., PD,
coaching); Slight adjustments to coaching and professional development to
meet the needs of more experienced Fellows
CST: Fellows’ students performed similarly to a matched group of students
of non-Fellows across all subject areas
CAHSEE: Fellows’ students performed similarly to students of non-Fellows;
11th grade students of Fellows were more likely to pass than students of
non-Fellows
Course-Taking Patterns: Students of Fellows (57%) were more likely to
advance to another math class in the next school year than students of
non-Fellows (51%) for both years (2010/11 & 2011/12)
Student Attitudes: At pretest, students of Fellows and non-Fellows had
similar attitudes towards mathematics; fall survey to be administered and
analyzed to determine if there are differences at posttest
Fellows at “cluster sites” reported more support, lesson collaboration, and
similar philosophies about teaching than Fellows at “individual sites”
Recruitment: Some Fellows reported that MfA LA provided additional
motivation to enter the teaching profession
Retention: 97% of Fellows have been retained; 91% of Fellows plan to
stay in teaching field after MfA LA
Teacher Quality: Fellows have strong academic backgrounds (i.e., Praxis,
college major) and build from this basis as indicated by coaches comments
& classroom observations
Program Strengths: Professional and Community Support; Quality of
Program Activities (e.g., PD, workshops); Financial Support
Professional Development: Lesson planning; Small-group discussions
Coach-Fellow Mentoring Model: Critical and constructive feedback from
coaches; Clarification of coaches’ roles; Training for coaches
2009 Cohort
“I am very thankful for all that
MƒA LA has been done to
make me a better teacher.”
2009 Cohort
“I am grateful for such a
wonderful chance to inspire
young minds ...”
2009 Cohort
“I always consider it an honor to
say that I am a fellow in such a
wonderful organization
Renewing and expanding our
commitment to improve
student achievement and
mathematics education in the
greater Los Angeles
community with each new
cohort of fellows.