Transcript The effect of regression towards the mean in assessing crime

The effect of regression towards the mean in
assessing crime reduction interventions
using non-randomised trials
Campbell Collaboration Colloquium
London 14-16 May 2007
Paul Marchant
Innovation North,
Leeds Metropolitan University
[email protected]
Paul Baxter
Department of Statistics, University of Leeds
1
The basic point



If non-RCTs are used, we need a sound
understanding of the system being studied
and a quantitative model to work out what
is lost and what the effect is.
The effects being sought may be small so
impact of small systematic errors can be
important.
Need rigorous scientific evaluation of the
implementation of policy.
2
Scientific Methods Scale



5 point crime research ‘Scientific Methods
Scale’ which orders trial designs. (RCT is
the top )
While the ordering may be fine there
seems little formal indication of what is
lost by using ‘a 4’ rather than ‘a 5’.
The potential exists to draw false
inference.
3
The Randomised Controlled Trial
(A truly marvellous scientific invention)
Population
Take Sample
Randomise to 2 groups
Note to avoid bias:
 Register trial /
protocol.
 Allocation is best made
tamper-proof.
(e.g. use ‘concealment’)
Old Treatment
New Treatment

Compare outcomes (averages)
recognising that these are
sample results and subject to
sampling variation when
applying back to the population
Use multiple blinding
of:




patients,
physicians,
assessors,
analysts …
4
Counts of those cured and not
cured under the two treatments
Cured
Not Cured
New Treatment
a
b
Control
(Standard treatment)
c
d
By comparing the ratios of numbers ‘cured’ to
‘not cured’ in the 2 arms of the trial, the Cross
Product Ratio (CPR)= (ad)/(cb), it is possible to
tell if the new treatment is better.
5
Confidence Interval for estimate of treatment effect



However there is sampling variability, because
we don’t study everybody of interest; just our
random sample. There is uncertainty in the
estimate.
Need to know how to calculate the CI
appropriately. This can be done under
assumptions, which seem reasonable for the
case of a clinical RCT and leads to a simple
expression for the approximate CI .
So should be able to obtain a valid estimate of
treatment effect,….. providing there is no trouble
from biases (e.g. differential drop out,
publication bias …).
6
Crime counts before and after in two areas
one gets a Crime Reduction Intervention
CRI (e.g. 3 on the Methods Scale)
Before
After
Treatment Area
a
b
(Intervention is introduced
between the 2 periods )
Comparison Area
c
d
(Nothing is changed)
A similar table results. But this is not the same as the RCT
set up as:
1 Not randomised, so no statistical equivalence exists at
the start.
2 The unit is area, rather than crime event.
7
This creates 2 problems


1 The areas are not statistically
equivalent at the outset.
2 The natural variation will be different
(overdispersed) from the simple
expression appropriate in an individually
randomised trial with statistically
independent outcomes.
8
The lack of equivalence between areas



It is the most crime-prone area that gets
the intervention, whereas the relatively
crime-free comparison area does not
receive anything new.
Lack of equivalence at the start allows
‘regression towards the mean’ (RTM) to
operate.
The name ‘Control Area’ is a misnomer.
‘Comparison Area’ is a better name.
9
Regression towards the mean




First discovered by Francis Galton (1880s) who
found that:
Tall parents tend to have tall offspring but not as
tall as themselves (on average); i.e. offspring
tend to be less tall than their parents.
Short parents tend to have short offspring but
not as short as themselves (on average); i.e.
offspring tend to be less short than their
parents.
This tendency for both to become more average
is ‘regression towards the mean’ RTM.
10
Regression towards the mean
Y The after measurement
100
Line of Equality
Line of mean of Y
for a given X
(The conditional
Mean)
Cloud of
Data
Points
50
0
0
50
X The before measurement
100
11
RTM needs to be accounted for in making
comparisons


Imagine a thought experiment for a
treatment to reduce the height of the next
generation!
Problematic if the treatment is given only
to tall parents and the results (height of
offspring) are compared with those of
short parents, who receive placebo,
(because of RTM, as in Galton’s
discovery).
12
Crime reduction assessment



RTM will be a problem in assessing crime
levels changing from one period to the
next. Is any reduction seen due to an
intervention?
However what is the bivariate distribution
for ‘Before to After’?
Log Normal seems a reasonable
candidate.
13
Burglary counts in successive periods: data from Tilley et al.
Scatterplot of Burg2 vs Burg1
1000
800
Burg2
600
400
200
0
0
200
400
600
800
1000
Burg1
14
Log Normal Probability Plot for Burglary count data (from Tilley)
Probability plot of burglary in year 1 and year 2
Lognormal - 95% CI
10
Percent
99.9
Burg1
99.9
99
99
95
90
95
90
80
70
60
50
40
30
20
80
70
60
50
40
30
20
10
10
5
5
1
1
0.1
0.1
10
100
1000
100
Burg2
1000
Burg1
Loc
Scale
N
AD
P-Value
5.488
0.6695
116
0.407
0.345
Burg2
Loc
Scale
N
AD
P-Value
5.432
0.6525
123
0.546
0.157
15
Estimating the effect of RTM
On the basis of a log normal crime distribution it
can be shown that if the intervention has no
effect, the Expected ( ln CPR )
= (1-ρσy/σx) ln x1/x2
x1/x2 is the crime ratio; σx , σy the standard
deviations, before and after, on the log scale and
ρ the correlation also on the log scale.
Var( ln CPR ) = 2 σy2(1-ρ2)
16
LnCPR with limits
LnCPR 95% Limits vs Ratio of Intital Counts in 2 areas (Based onTilley data)
1.0
LnCPR_Limits
0.5
0.0
-0.5
-1.0
0.0
0.5
1.0
1.5
2.0
RatioOfCounts
2.5
3.0
17
Can work with rates instead (still lognormal)
Probability plot of the natural log of burglary rates (burglaries per
household) in successive periods, using the same Tilley data
-4
99.9
-3
-2
-1
99
95
90
Percent
80
70
60
50
40
30
20
10
5
1
0.1
-4
-3
-2
-1
Rates will give the same CPR as counts in any one study if
the denominator does not change.
18
Log burglary rates (no. per household) in successive periods
Natural Log Burglary Rate in Period 2
-1.0
-1.5
-2.0
-2.5
-3.0
-3.5
-4.0
-3.5
-3.0
-2.5
-2.0
Natural Log Burglary Rate in Period 1
-1.5
-1.0
19
On the larger geographical scale


Have also examined year to year data
from Crime and Disorder Partnership CDRP
Areas (376 covering all England and
Wales). In particular burglary counts and
rates.
Lognormal seems OK for this too
(correlation higher than for smaller area
data).
20
Not just of ‘mathematical’ interest



It is claimed that a study of the effect of lighting
is not threatened by RTM. Examined using
successive years in Police Basic Command Unit
Areas (of similar size to CDRPs). CPRs, formed
from comparing quintile bands of crime rate,
show the effect is a only a few percent.
This value is what our simple lognormal model
predicts from the parameters of the data.
However the geographical scale of the lighting
study is much smaller than the BCU scale.
21
Another possible distribution


The bivariate lognormal gives a rather
neat solution and seems to be in accord
with data.
However a less neat result comes from a
bivariate Gamma distribution. (Assuming
constant coefficient of variation).
22
Gamma probability plot to the marginal distributions
Probability Plot of Burg1, Burg2
Gamma - 95% CI
10
Burg1
Percent
99.9
99.9
99
99
95
90
80
70
60
50
40
30
20
95
90
80
70
60
50
40
30
20
10
10
5
5
1
1
0.1
10
100
1000
100
Burg2
1000
Burg1
Shape
2.488
Scale
120.4
N
116
AD
1.050
P-Value 0.011
Burg2
Shape
2.790
Scale
99.00
N
123
AD
0.526
P-Value 0.204
0.1
23
Disadvantage of the Gamma
The expressions for the Expectation and Variance of ln CPR
contain the initial values, x1 and x2 , as a ratio with the
mean of the distribution.
For example:
Expected ln CPR =
ln(x1/x2 ) - ln((1+ρ(x1/μx-1)/(1+ρ(x2/μx-1))
Here ρ is the correlation and μx the mean (on the original
scale)
---------------------------------------------------------------------Note: Forthcoming presentation by Paul Baxter to RSS2007
‘Statistics and public policy-making: hope versus reality’
24
Some conclusions




A ‘Methods Scale’ seems to suggest that designs
weaker than RCTs might suffice, but there is a loss.
RTM is one problem in non-RCT studies.
RCTs can be problematic enough. (We need
registered trials, published protocols, pre-defined
analysis plans, concealment of allocation, blinding
etc…..)
Difficult to calculate the effect of RTM because of
model choice, outcome choice, appropriate
parameter values. (Problem of unobserved
confounders in non-RCTs)
Evaluations of policies once implemented need to be
done to a high scientific standard.
25
References
Farrington D.P. and Welsh B.C. (2006) How
Important is Regression to the Mean in AreaBased Crime Prevention Research?, Crime
Prevention and Community Safety 8 50
Marchant P.R. (2005) What Works? A Critical Note
on the Evaluation of Crime Reduction Initiatives,
Crime Prevention and Community Safety 7 7-13
Tilley N., Pease K., Hough M. and Brown R. (1999)
Burglary Prevention: Early Lessons from the
Crime Reduction Programme, Crime Reduction
Research series Paper1 London Home Office
26