Transcript Debate on Effect Size - Durham University Community

Effect Size
Robert Coe
@ProfCoe
ResearchED 2013, Dulwich College, 7 Sept 2013
The case for using effect size
I.
II.
What is Effect Size?
The case for using effect size
(5 reasons)
III. Problems in using effect size
(7 problems)
IV.
Recommendations
(13 recommendations)
V.
Further reading
2
What is Effect Size?
3
© 2003 Robert Coe, University of Durham
Effect Size is the difference
between the two groups, relative
to the standard deviation
Mean of experimental group – Mean of control group
Effect Size =
Standard deviation
The case for using
effect size measures
6
© 2003 Robert Coe, University of Durham
1. Effect size allows differences in
uncalibrated measures to be
interpreted

How would you compare/interpret
o 5 marks difference on a home-made test
o A gain of 8 points on a standardised test
o Improvement of 1 sub-level at KS2
o An effect of half a grade at GCSE

Comparing each to the spread of values in the
population gives it some meaning
7
© 2003 Robert Coe, University of Durham
2. Effect size allows the accumulation
of knowledge

Meta-analysis can combine results from
different studies:
o What is the average effect?
o What factors mediate the effect?
o Which interventions have the biggest effects?

Small studies are worth doing because they can
be added to the pool
8
© 2003 Robert Coe, University of Durham
www.educationendowmentfoundation.org.uk/toolkit
Impact vs cost
Effect Size (months gain)
Promising
8
May be
worth it
Feedback
Meta-cognitive
Peer tutoring
Homework
(Secondary)
Early Years
1-1 tuition
Collaborative
Behaviour
Small gp
Phonics
Parental
tuition
involvement
ICT
Social
Individualised Summer
schools
learning
Mentoring
Homework
(Primary)
Performance Aspirations
0
pay
£0 Ability grouping
Cost per pupil
Smaller
classes
After
school
Teaching
assistants
£1000
Not
worth it
3. Effect size emphasises amounts,
not just statistical significance



The dichotomous “significant/not” decision is
almost never appropriate
The size of a difference is almost always
important
“Significance” has many meanings, but is
inevitably related to the size of the difference
10
© 2003 Robert Coe, University of Durham
Don’t ignore amounts
3
2.5
2
1.5
1
0.5
0
-0.5
-1
(a)
"not significant"
(b)
"significant"
11
© 2003 Robert Coe, University of Durham
(c)
"significant"
4. Effect size draws attention to the
margin of error


Statistical power is important, but often
overlooked
Much apparent disagreement is actually just
sampling error
12
© 2003 Robert Coe, University of Durham
5. Effect size may help reduce
reporting bias


The “file-drawer” problem is alive and well
Within-study reporting bias can also be a
problem
13
© 2003 Robert Coe, University of Durham
Problems in using
effect size measures
14
© 2003 Robert Coe, University of Durham
Problems with effect size
1.
2.
3.
4.
5.
Alternative effect size measures
Which standard deviation?
Measurement reliability
Non-normal distributions
Interpreting effects: small, medium,
large
6. Incommensurability: outcomes,
treatments, populations
7. Is it really an ‘effect’?
15
© 2013 Robert Coe, University of Durham
Further reading …
16
© 2003 Robert Coe, University of Durham
1
An introduction, guide and tool for calculating effect sizes:
www.cem.org/evidence-based-education/effect-size-resources
2
Coe, R. (2002) It's the effect size, stupid:
what effect size is and why it is important.
Paper presented at the Annual Conference of
the British Educational Research Association,
University of Exeter, England, 12-14 September
2002.
www.cem.org/attachments/ebe/ESguide.pdf
3
Coe, R.J. (2012) ‘Effect Size’
in J. Arthur, M. Waring, R.
Coe, and L.V. Hedges (Ed.s)
(2012) Research Methods and
Methodologies in Education.
London: Sage.
17
© 2013 Robert Coe, University of Durham
Coe, R. (2004) ‘Issues
arising from the use of
effect sizes in analysing
and reporting research’
in I. Schagen and K.
Elliot (Eds) But what
4
does it mean? The use
of effect sizes in
educational research.
Slough, UK: National
Foundation for
Educational Research.
http://www.nfer.ac.uk/nfer/publications/SEF01/SEF01.pdf
18
© 2013 Robert Coe, University of Durham