Transcript Presenting Data - Durham University

Making Comparisons
Doctor of Education (EdD)
Analysing, Interpreting and Using Educational
Research (Research Methodology)
D
University
of Durham
Dr Robert Coe
University of Durham School of Education
Tel: (+44 / 0) 191 334 4184
Fax: (+44 / 0) 191 334 4180
E-mail: [email protected]
http://www.dur.ac.uk/r.j.coe
Effect size
Average score of
person taught
‘normally’
-4
-3
-2
-1
Average score of
person taught by
experimental method
0
Student Achievement
© 2007 Robert Coe, University of Durham
(standardised)
1
2
3
4
2
-4
-3
-2
-1
0
Student Achievement
© 2007 Robert Coe, University of Durham
1
2
3
4
3
Effect Size is the difference
between the two groups,
relative to the
standard deviation
Mean of experimental group – Mean of control group
Effect Size =
Standard deviation
© 2007 Robert Coe, University of Durham
4
Examples of Effect Sizes:
ES = 0.2
58%
of
control
group
below
mean of
experimental
group
“Equivalent to the
difference in heights
between 15 and 16
year old girls”
Probability you could guess which group a person was in = 0.54
Change in the proportion above a given threshold:
from 50% to 58%
or from 75% to 81%
© 2007 Robert Coe, University of Durham
5
ES = 0.5
69%
of
control
group
below
mean of
experimental
group
“Equivalent to the
difference in heights
between 14 and 18
year old girls”
Probability you could guess which group a person was in = 0.60
Change in the proportion above a given threshold:
from 50% to 69%
or from 75% to 88%
© 2007 Robert Coe, University of Durham
6
ES = 0.8
79%
of
control
group
below
mean of
experimental
group
“Equivalent to the
difference in heights
between 13 and 18
year old girls”
Probability you could guess which group a person was in = 0.66
Change in the proportion above a given threshold:
from 50% to 79%
or from 75% to 93%
© 2007 Robert Coe, University of Durham
7
Effect size



The difference between the two means,
expressed as a proportion of the standard
deviation
ES =(Me – Mc) / SD
Issues







Which standard deviation?
Statistical significance?
Margin of error?
Normal distribution?
Restricted range
Reliability
See www.cem.dur.ac.uk/ebeuk/research
© 2007 Robert Coe, University of Durham
8
Exercise 3.

On each plot, estimate the effect size
© 2007 Robert Coe, University of Durham
9
Bloom’s ‘two-sigma problem’
The search for methods as effective as one-to-one tutoring
-4
-3
-2
-1
0
1
2
Student Achievement
© 2007 Robert Coe, University of Durham
3
4
5
6
10
Effect sizes for other interventions
Effect of
Reducing class size
from 24 to 15
on
Student
achievement
is
0.15
Setting vs. mixed
ability classes
Student
achievement
0.00
Computer based
instruction
Student
achievement
0.24
School based drug
education
Substance
use
0.12
Practice test taking
Test scores
0.32
© 2007 Robert Coe, University of Durham
0.08 for high achievers
-0.06 for low achievers
0.02 in well-controlled
studies
11
Further examples of
effect sizes
© 2007 Robert Coe, University of Durham
12
The importance of effect size
Ofsted? DfES? LEA?
Standards Unit?
Headteacher?
Consultant?
The next time somebody
tries to tell you what to do…
What is the evidence?
 How big is the effect size?
Advice, Policy

© 2007 Robert Coe, University of Durham
13
Effect Size vs Statistical Significance







Emphasises amounts, not just directions
Avoids inappropriate dichotomies
Avoids confusion over ‘significance’
Draws attention to power
Avoids ‘file drawer’ problem
Promotes synthesis rather than disagreement
Allows accumulation of knowledge
© 2007 Robert Coe, University of Durham
14
Comparing percentages
Disorganised learners
become exam failures
… Medical students at the
university were asked to provide
a passport photo at the start of a
module in paediatrics. A total of
366 (93 per cent) of students
handed in the photo. Of the 29
who failed to do so, 13 went on
to fail their end-of-year exams.
(From THES, 28.6.02 & BMJ)
© 2007 Robert Coe, University of Durham
15
Contingency table
Handed in
photo
Didn’t
Total
Failed
Passed
Total
164
0
336
202
366
16
29
13
(100%)
(55%)
(55%)
395
What percentage of the ‘photo’ group have to pass
before it is clear there is a difference?
© 2007 Robert Coe, University of Durham
16







2
(Chi-squared) test
If both groups were from the same population, you
would expect approximately the same proportion
passing in each
But not exactly the same proportion
Small differences are quite likely, large differences
more unlikely

2
test tells you how unlikely
p = probability of getting such a big difference purely
by chance
Depends on


difference in percentages and
the sample size
© 2007 Robert Coe, University of Durham
17
Chi-squared test gives critical (p<0.05) numbers as…
265
300
250
200
101
150
100
50
0
Photo
Failed
Passed
No photo
No photo
Photo
© 2007 Robert Coe, University of Durham
18
Comparing percentage changes
38
Percentage gaining 5+ GCSE grades at A*-C
40
35
30
30
26
25
20
19
Asian
African Caribbean
15
10
5
0
1991
1993
Has the gap got bigger or smaller?
© 2007 Robert Coe, University of Durham
19
Has the gap got bigger or smaller?

38
Percentage gaining 5+ GCSE grades at A*-C
40
35
30
30
26
Asian up 8pts, African
Caribbean up 7pts, so
Asian has increased
more
25
20
19
Asian
African Caribbean
15
10

5
0
1991
© 2007 Robert Coe, University of Durham
1993
Asian increased by 8
from 30 (27%), African
Caribbean by 7 from 19
(37%), so African
Caribbean has increased
more.
20
Logit transformation
1
0.9
0.8
Probability
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
-5
-4
-3
-2
-1
0
1
2
3
4
5
Logit (equal interval scale)
© 2007 Robert Coe, University of Durham
21
Odds ratios

38
Percentage gaining 5+ GCSE grades at A*-C
40
35


30
30
Asian
26

Odds before: 30/70 = 0.43
Odds after: 38/62 = 0.63
Odds ratio: 0.63/0.43 = 1.4
25
20
19
Asian
African Caribbean
15

10
5
African Caribbean

0
1991
1993


© 2007 Robert Coe, University of Durham
Odds before: 19/81 = 0.23
Odds after: 26/74 = 0.35
Odds ratio: 0.35/0.23 = 1.5
22
Intergenerational mobility
© 2007 Robert Coe, University of Durham
23