Transcript Brief Critique of IES Student mentoring evaluation

+
Jean Rhodes
National Mentoring Partnership Professor
Minnesota Mentoring Partnership
October 24, 2010
For Which Youth is
Mentoring Most Effective?
+




How effective is youth mentoring for all youth?
For which youth is it most effective?
Under what conditions is it most effective?
What are the implications for policy, practice, and research?
+
Recent Meta-Analysis of Youth
Mentoring

Encompassed 73 independent evaluations (1999-2010).

The overall effect size was .21*, collapsing across studies and
outcomes

The average follow-up effect size across the studies was .17.
DuBois, Portillo, Rhodes, Silverthorn, & Valentine (in press). Psychological
Science in the Public Interest.
Comparison
of
mean
post-tx
+
effects with school and community
interventions with children/teens
Moderator
Mentor Meta-analysis
Other Meta-analyses
Attitudinal/Motivational
0.19
0.23- 0.25
Social/Relational
0.17
0.15-0.26
Psychological/Emotiona
l
0.15
0.10-.0.24
Conduct Problem
0.21
.02-0.41
Academic/School
Attitudes
Grades
Achievment tests
0.21
0.19
0.24
0.18
0.11-0.27
0.14
0.22
0.11-0.24
Physical Health
0.06
0.08-0.41
Size of Effect on Youth Outcomes
+
Effect sizes
0.5
Medium
Effect
0.4
0.3
0.2
EmpiricallyBased
Practices
Small
Effect
0.1
Theory-Based
Practices
0
-0.1
0
1
2
3
4
5
6
7
Number of Practices
8
9
10
11
+ Study level variables (moderators)
associated with different effects
Youth, Mentor, Program
Characteristics
Effect Size
Problem Behavior Involvement
Yes: .29
No: .20
Youth Gender
>50% Male: .25
<50% Male: .18
Individual/Environmental Risk
Low/High: .33
High/Low: .31
Evidence-based Mentor Training
Below median:
.19
Above median:
.24
Mentor role function: Advocacy
Yes: .26
No: .20
Matching based on shared interests Yes: .44!!!
No: .21
+




How effective is youth mentoring for all youth?
For which youth is it most effective?
Under what conditions is it most effective?
What are the implications for policy, practice, and research?
+

Stronger effects when…
Youth with

With moderate personal/environmental risk

Who are male

satisfactory, but not strong baseline relationships.
+
Effects of Mentoring on Youth with
Different Relational Profiles
BASELINE
Relationship
s
Poor Relationships
Satisfactory but
not Strong
Strong
Relationships
Overall
Academics
.00
.21***
.05
Prosocial
.04
.19*
.04
Effort
.05
.18*
.00
Self-Esteem
-.04
.07
-.01
(Schwartz, Rhodes, & Chan (2010). Developmental Psychology
+ Stronger effects when…

Mentors who

Fit of background/ training with program goals

Play an active, advocacy role

Are sensitive to socioeconomic & cultural influences

Have higher self-efficacy

Hold positive attitudes toward youth
+
Measuring
mentors’ attitudes
Grossman et al., 2007

The scale asked mentors to rate
how many “kids in your community”
could be characterized by
indicators of youth development:
 work hard at school
 respect adults
 are trouble-makers
 are fun to be around
 expect things to be handed to
them
 try to do their best
 are interested in learning
+

Vulnerable mentees who were paired
with high school mentors with
positive attitudes about youth
were more emotionally engaged with
mentor than those paired with more
negative mentors

Mentor attitudes
and youth outcomes
Karcher, Rhodes, Herrera, &
Davidson (2010). Applied
Developmental Science
Those who were paired with high
school mentors with negative
attitudes about youth were less
emotionally engaged with and showed
some negative outcomes.
Stronger effects when…

Relationships characterized by

consistency

closeness

structure

appropriate meeting times

duration
+ Does meeting time matter?…
Mentoring by Meeting Time
Afterschool/Lunch
After School During
School
Interaction
Effect
Academic Achievement
.19*
-.09
-.33*
Math
.18
-.42**
-.60**
Science
.14
-.19
-.36*
Social Studies
.11
-.16
-.29
Reading
.27*
-.02
-.32
Language
.21*
-.05
-.33*
Unexcused Absences
-.35
-.08
.21
Grades
-.20
-.15
-.51
Scholastic Efficacy
.12
.13
.03
*Note ES= Effect Size
The role of duration
Grossman & Rhodes (2002). American Journal of Community Psychology
+
Re-matching?
+ Test of Time 2: Results

Only youth in matches lasting 24 wks
or more benefited academically

All mentored youth were less likely
to skip school, regardless of match
length

After controlling for selection bias:

Positive academic impacts
observed only for youth with
intact matches

No academic impact for youth
with early terminations

Negative academic impacts
for rematched youth

Grossman, Chan, Schwartz, & Rhodes (in
press). American Journal of Community
Psychology.
+ Stronger effects when…

Programs characterized by

careful recruitment

training

monitoring

Multi-modal

matching on interest
+




How effective is youth mentoring?
When are programs most beneficial?
How does mentoring promote positive youth
development?
What are the implications for policy, practice, and research?
+




How effective is youth mentoring for all youth?
For which youth is it most effective?
Under what conditions is it most effective?
What are the implications for policy, practice, and research?
Pathways of mentor
influence
mediator
Parental/peer
relationships
Social-emotional
development
Mutuality
Trust
Empathy
Identity
development
Mentor
Relationship
Cognitive
development
Interpersonal history, social competencies, relationship duration,
developmental stage, family and community context
moderators
moderators
Positive
Outcomes
e.g., reduced
health risk,
better psych.
outcomes
Pathways of mentor influence
Scholastic
Competence
.26
.25
.08
Grades
.29
Mentoring
.22
Quality of
Parental
relationship
.25
.26
.11
School value
.18
Skipping School
.09
-.28
Child Development, (2002), 1662-1671
.19
Self-worth
Pathways of mentor influence
Mentoring
.23
Quality of
Parental
relationship
-.46
.18
Self-worth
-.08
-.04
.10
.14
Quality of
Peer
relationships
Rhodes, Reddy, & Grossman (2005) Applied Development Science
Substance Use
Pathways of mentor influence
Academic
Attitudes
.25
.78
Grades
.27
Quality
of Mentoring
.27
Quality of
Teacher
relationship
.13
.53
.32
Quality of
Parent
Relationship
-.28
Chan,Rhodes, Schwartz, & Lowe (2011)
Self-worth
.09
.18
School
Behavior
+



How effective is youth mentoring for all youth?
For which youth is it most effective?
Under what conditions is it most effective?
+
Recommendations

Promote
 evidence-based practice
 include rigorous evaluation
 measured replication and dissemination

Reward sustainability and quality over growth
+
Research to practice
Research
Implementation
Product
Development.
• Youth Serving
Organizations
Distribution
• MENTOR/NMP
+
Action steps
 Develop
and improve training and support
around relationships
 Improve

mentor retention
Employee retention, qualitative studies
 Export
to other youth-serving settings
+
Give Psychology Away
+
Credits
Big Brothers Big Sisters of America
WT Grant Foundation
Edna McConnell Clark Foundatin
NICHD