Transcript How are we Doing? Developing Performance Indicators

A Bayesian, Meta Cost-Benefit
Model
John Roman, Ph.D.
P. Mitchell Downey
District of Columbia
Crime Policy Institute
Partnership for Greater
Washington Research
The Brookings Institution
June 30, 2010
The Urban Institute
The Brookings Institution
The views expressed are those of the authors and should not be
attributed to The Urban Institute, its trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Presentation Overview
• Background
–
–
–
–
Existing Models
DCPI
Drug Courts
Meta-Analysis
• Proposed Model
• Analytic Strategy
• Results
• Next Steps
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
General Approach
Estimate:
• Costs of drug court in DC;
• Benefits of reduced crime (general);
• Expected benefits for a DC drug court
population;
• Effect of drug court on criminal behavior;
• Translate effects into benefits, difference the
costs.
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Presentation Overview
• Background
– Existing Models
– DCPI
– Drug Courts
– Meta-Analysis
• Proposed Model
• Analytic Strategy
• Results
• Next Steps
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
The Origins of DCPI
Funded by the Justice Grants Administration (JGA) in the
Washington, D.C. Mayor’s office using Recovery Act
funds.
Key goal was to create an entity similar to the Washington
State Institute of Public Policy (WSIPP).
• WSIPP is a non-partisan, non-profit that provides
evidence to support the Legislature;
• Created in the late 1980s, WSIPP (under the leadership
of Steve Aos and Roxanne Lieb) conducted research
funded by the legislature to inform decision-making.
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
The Origins of DCPI
In 1998, WSIPP researchers create a meta, cost-benefit
model (often called the Aos model).
• Performs meta analysis across a range of policies and
programs;
• Links to Washington state specific costs of operations;
• Monetizes outcomes;
• Prioritizes policy choices and makes recommendations to
the legislature.
In 2008, recommended funding several cost-effective
initiatives. Savings from those programs allowed plans to
build two new prisons to be shelved. Passed by the
Legislature.
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
How Can Existing Models be Improved?
The WSIPP model embodies a quantum leap is evidencebased policymaking. However, there are two (intractable)
problems with the model:
• First, general problem in meta analysis that if the
underlying studies are not identified, no aggregate causal
relationship can be established;
• Second, the model does not account for uncertainty in
several steps of the estimation process, nor does it
account for uncertainty in combining multiple estimates;
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Identification Issues in the Existing Model
Consider the question of whether DC should
implement an Adult Drug Court in Washington.
In order to believe that there is a causal
relationship between drug court participation
and successful outcomes (Y) must believe:
YT > YC and, that assignment is exogenous, such
that in the absence of drug court YT = YC
If assignment is not exogenous and YT ≠ YC then
no causal attribution is possible. Problem gets
worse with aggregation.
We have not solved this identification problem.
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Integrating Uncertainty into the Model
If model is not identified, and causal attribution is not
possible, must include all sources of uncertainty into the
model. Two types of uncertainty:
• Estimation. Each step in the process should yield both a
point estimate and a distribution.
• Aggregation. Final results should integrate uncertainty
from each estimate.
WSIPP model produces a single point estimate with no
confidence interval. Cannot determine probabilistically
whether expected outcomes will occur.
Problem akin to comparison of fingerprints and DNA
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Integrating Uncertainty into the Model, con’t
Why isn’t uncertainty included in the WSIPP
model?
The problem was intractable using classical
techniques at the time the model was conceived
due to limitations in computing power.
Also, difficult to revisit estimates in a policy
environment.
Solution: Bayesian inference allows distributions
rather than ‘moments’ to be aggregated,
maximizing how much uncertainty is included in
the model.
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Donald Rumsfeld on the Importance of
Incorporating Uncertainty
"Reports that say that something hasn't
happened are always interesting to me,
because as we know, there are known knowns;
there are things we know we know. We also
know there are known unknowns; that is to say
we know there are some things we do not
know. But there are also unknown unknowns -the ones we don't know we don't know.“
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Other Benefits of a Bayesian, Meta, Cost-Benefit
Analysis
• The model is designed to be replicable;
• Does not requires abundant reputational capital
with stakeholders, because;
– It allows for stakeholder input;
– Can be adjusted in real-time to test different
assumptions;
– All assumptions are transparent.
• Reduces the number of stages of estimation.
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Presentation Overview
• Background
– Existing Models
– DCPI
– Drug Courts
– Meta-Analysis
• Proposed Model
• Analytic Strategy
• Results
• Next Steps
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
What is the District of Columbia Crime Policy
Institute (DCPI)?
A nonpartisan, public policy research organization
focused on crime and justice policy in Washington, DC.
DCPI’s mission is to support improvements in the
administration of justice and public safety policies,
through evidence-based research.
Collaboration of UI and Brookings researchers to inform
long-term strategic and short-term operations research
on juvenile and criminal justice mattres.
Goal is timely, practitioner-oriented research to develop
and implement evidence-based crime policy in DC.
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
What will DCPI Do?
Year 1 DCPI activities include:
• Develop a cost-benefit model to identify cost-effective DC
based interventions;
• Develop a research library (DCCrimePolicy.org);
• Conduct independent research:
– An Evaluation of the Mayor’s Focused Improvement
Area Initiative;
– A Study of Promising Practices at the Metropolitan
Police Department;
– A Study to Understand the Impact of Pre-Trial
Detention on Defendants and its Implications for
Evidence-Based Practice
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Presentation Overview
• Background
– Existing Models
– DCPI
– Drug Courts
– Meta-Analysis
• Proposed Model
• Analytic Strategy
• Results
• Next Steps
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
What are Drug Courts?
Drug courts use court-based treatment as an alternative to
incarceration.
• Begun in Miami in 1989, drug courts are specialized
dockets that process drug-involved offenders.
• Client behavior closely monitored by a judge.
Key premises:
• Drug involved offenders would commit fewer crimes if
they desisted from drug use.
• Relapse is part of recovery.
• Treatment participation can be encouraged (coerced?).
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Do Drug Courts Work?
Yes? Can’t randomly assign courts to have a drug
court or not. So next best studies find client
outcomes are moderately better using;
• RCTs in a single site;
• Meta-analyses of many studies with varying
designs;
• Simulation studies using population data.
Generally, recidivism is reduced 10-15 percent.
Potential for large net social benefits.
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Presentation Overview
• Background
– Existing Models
– DCPI
– Drug Courts
– Meta-Analysis
• Proposed Model
• Analytic Strategy
• Results
• Next Steps
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
What is Meta Analysis?
Policymakers seem most convinced by metaanalysis as it uses evidence from many settings.
Steps in meta analysis are to:
•
Calculate an effect size (standardized mean
effect size comparing T and C);
•
Sum across studies (weighting by the inverse of
the variance of each effect size);
•
Account for heterogeneity;
•
Apply ad hoc weights.
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
What is Meta, Cost Analysis?
Translate effect size into net benefits to account for:
•
Savings to law enforcement - reducing new crime;
•
Savings to courts – not prosecuting new offenders
•
Savings to corrections - not locking those people up;
•
Savings to those who are not victimized;
Add in costs of new policies and programs.
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Presentation Overview
• Background
–
–
–
–
Existing Models
DCPI
Drug Courts
Meta-Analysis
• Proposed Model
• Analytic Strategy
• Results
• Next Steps
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
What is Meta, Cost Analysis (Again)?
What are the criticisms?
•
Garbage in, garbage out (identification);
•
No uncertainty in these estimates.
Can’t solve identification issues at this time;
Can introduce additional uncertainty.
How? Use Bayesian inference rather than classical
(inferential, repeated sample) statistics.
Analogous to measuring using the metric system.
Represents the same fact, but standardized Base
10 can be more efficient than Base Whatever.
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Why go Bayesian (Statistically)
Accounts for uncertainty in the presence of
common problems:
• Multi-stage model.
– Output of one stage is next stage’s input.
– Bayesian inference takes a weighted average of
all output (weighted by the probability) versus
just using the mode (frequentist).
• Non-symmetric distributions. Default is ‘mode’
in frequentist inference.
• Presence of Missing Data. Again, uses
distribution rather than MI or listwise deletion.
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Why Should DCPI Go Bayesian?
• Policy Buy-In.
– Presents data more intuitively (e.g. weather forecaster
does not predict an expected mean precipitation of 0.5
inches with a standard deviation of .25).
– Allow Policymaker input. (e.g. can provide information
about prior beliefs that can improve model
performance (effect of policy changes on various
problems, changes in crime patterns in DC,
demographic changes, etc.).
• Flexibility. Can adapt to all manner of common problems
in inference (missing data, etc.);
• Can’t combine multiple estimates with uncertainty
without going Bayes. Replaces sensitivity analyses.
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Presentation Overview
• Background
–
–
–
–
Existing Models
DCPI
Drug Courts
Meta-Analysis
• Proposed Model
• Analytic Strategy
• Results
• Next Steps
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Analytic Strategy: General
Estimate:
• Costs of drug court in DC;
• Benefits of reduced crime (general);
• Expected benefits for a DC drug court
population;
• Effect of drug court on criminal behavior;
• Translate effects into benefits, difference the
costs.
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Analytic Strategy: Costs
• Taken from a prior study of a DC drug court
(Harrell, et al. 1998)
• Inflation adjusted to get present day per
participant costs of drug court operations
• Estimates $11,500 per participant (for entire
process)
• Limitations:
– Based on FY 1995 costs – could have changed
– Assumes same size drug court (returns to scale)
• Advantages:
– DC specific
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Analytic Strategy: Valuing Benefits of Reduced
Crime
The cost of crime is the product of the price of crime (P)
and the quantity of crime (Q), so:
Costi= Pi*Qi
Where i’s are the categories of crime costs.
• Law enforcement costs of investigating crime;
• Court costs of processing crime;
• Corrections costs of supervision;
• Victims costs of being victimized.
Main source of uncertainty: price of crime to victims.
Extant estimates produce point estimates, Roman
(2009) includes distribution of prices.
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Analytic Strategy: Estimates of the Price of
Crime to Victims
Mean
Median
Standard Deviation
$1,445,463
$1,380,246
$771,395
Rape
149,542
23,143
355,583
Robbery
279,085
88,915
452,485
Assault
134,770
66,644
212,510
Burglary
4,444
814
384,591
Motor Vehicle Theft
15,175
6,174
76,477
Theft
1,996
192
459,817
Crime Type
Murder/Non-negligent
Manslaughter
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Analytic Strategy: Calculating Benefits
The goal of this analysis is to create an estimate
of the distribution of net harms averted.
Calculate reduced crime (impact)
• From large scale, multi-site evaluation of drug
courts find:
– the rate of re-arrest among comparison group
(not participating in drug court);
– the distribution of crimes committed by those
comparisons who were re-arrested;
– the proportion of arrests that led to
incarceration.
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Analytic Strategy: Benefits (cont.)
In the next step, we will estimate how many
crimes are prevented by drug court.
• For each arrest prevented, estimate what the
crime would have been by sampling from the
distribution of crimes committed by controls;
• Estimate the price of that crime by sampling
from Roman (2009);
• Find the probability that would have led to
incarceration (calculate costs of incarceration);
• These last two prices are the benefits of
preventing one crime.
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Analytic Strategy: Impact Estimates
• Meta-analysis
• Data:
– From Shaffer (2006)
– Coded typical meta-data (effect size, research
design, length of follow-up, sample size, etc.)
– Called drug courts which had been evaluated
and interviewed about policies and practices
• Our approach:
– Bayesian linear regression of estimated effect
size (correlation between re-arrest and drug
court participation)
– Condition effect size on court and study
characteristics
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Analytic Strategy: Impact Estimates
• Variable selection (court traits):
– Preliminary regressions of effect size on
theoretically meaningful characteristics
(treatment type, program design, response to
graduation/failure, eligibility, etc.)
– Selected those which were significant: violent
offenders eligible and motivation required for
enrollment
• Variable selection (study traits):
– Length of follow-up;
– Methodology (random trial, matching, etc.);
– Similarity between treatment and control groups
(race, age, gender, prior criminal history);
– Composition of control group (dropouts,
ineligible, etc.).
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Presentation Overview
• Background
–
–
–
–
Existing Models
DCPI
Drug Courts
Meta-Analysis
• Proposed Model
• Analytic Strategy
• Results
• Next Steps
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Results: Effect Size Estimates
• The final results are many simulated possible effect
sizes
• From these simulations, we can estimate which effect
sizes are more probable than others
• The following slide presents the probability (on the yaxis) that the effect size will be less than (larger
reduction) various values (on the x-axis)
• It is calculated separately for courts that require
motivation, don’t require motivation, and allow violent
offenders
• The vertical blue line indicates the probability that the
effect size will be below zero (drug court reduces crime)
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
0.6
Does not
Require
Motivation
0.4
Requires
Motivation
0.2
Probability
0.8
1.0
Results: Effect Sizes, Tests Eligibility Rules
0.0
Allows
Violent
Offenders
-0.25
-0.20
-0.15
-0.10
-0.05
0.00
Effect Size
Good
Effects
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
0.05
Bad Effects
URBAN INSTITUTE
Justice Policy Center
Results: Effect Size Estimates
• The results before clearly indicate that courts that do
not accept violent offenders are nearly guaranteed to
reduce offending
• However, often, it is not enough to know that there will
likely be some reduction, but the size of that reduction
is critical
• A key advantage of Bayesian methods is that they allow
one to estimate the probability that the reduction will
be greater than any particular value
• If we arbitrarily define a large effect to be an effect size
less than -0.08, then we can calculate the probabilities
of a large effect under each of the three scenarios
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
0.6
Does not
Require
Motivation
0.4
Requires
Motivation
0.2
Probability
0.8
1.0
Results: Effect Sizes, Probability of a Large
Effect
0.0
Allows
Violent
Offenders
-0.25
-0.20
-0.15
-0.10
-0.05
0.00
Effect Size
Good
Effects
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
0.05
Bad Effects
URBAN INSTITUTE
Justice Policy Center
Results: Effect Size Estimates
• The previous slide indicates that there is a
large difference in the probability that a court
will achieve a large reduction, depending on
whether or not motivation is required
• This result is important because both courts
had near equal likelihood of leading to a
reduction in reoffending
• Motivation eligibility requirements only start to
matter when there is a large target effect
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
The Effect Size Over Time
• The following plot traces the effect size over
time, based on the relationship between
measured effect and length of follow-up
• The findings indicate that studies with longer
follow-up periods tended to estimate larger
effects, suggesting that the effect of drug court
grows over time
• For simplicity, only the mean effect size at
each time point is displayed
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
0.00
0.05
Allows
Violent
Offenders
-0.05
Does not
Require
Motivation
-0.10
Mean Effect Size
0.10
0.15
Results: Effect Size Over Time
-0.15
Requires
Motivation
0
5
10
15
20
25
30
35
Months of Follow-Up
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
The Problem with Effect Sizes
• The past three plots indicate that:
– Drug courts which do not accept violent offenders
are almost certain to have a true re-arrest reducing
effect
– The effects are larger in courts that require
motivation for enrollment
– The effects appear to grow over time
• However, all of the above results use the effect size
• Effect sizes are basically measures of the mean effects
• There is no certainty that any jurisdiction’s experience
will match the mean effect size
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
The Problem with Effect Sizes
• The problem with presenting effect sizes can best be
thought of by considering the sample size problem
ubiquitous in social science statistics
• Some effect is measured, but with a small number of
units, we cannot be sure whether the observed effect
reflects the true effect (hence tests for statistical
significance)
• This problem is reversed here
• We estimate the true effects of a drug court, but if only
a small number of offenders are enrolled, we cannot be
sure that the realized, observed effect will match the
underlying true effect
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
The Problem with Effect Sizes
• In other words, when presenting effect sizes,
we are presenting estimates of what the true
effect is, not what effect could be expected
from establishing a new drug court
• This problem is greatest when considering
programs with a relatively small number of
participants, which typifies most criminal
justice programs
• The following plot displays the densities of
expected number of arrests prevented, given
230 participants (the mean number from our
sample)
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
0.0 0.5 1.0 1.5 2.0 2.5 3.0
Density
Densities of Change in Rearrest
-0.6
-0.4
-0.2
0.0
0.2
0.4
Percent Change in Rearrests
Reduction in Rearrest
|
Increase in Rearrest
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
The Problem with Effect Sizes
• These densities indicate that there is substantial
probability that drug courts will not reduce re-arrests,
even though there is nearly a 100% chance that the
mean effect is a reduction
• Telling policymakers that there is a 99% chance of a
mean reduction would have left out the critical detail
that there is only a 90% chance that they will
experience a reduction in arrests
• The blue lines indicate the general findings from past
meta-analyses, which suggest that the mean drug court
effect is a 10-15 percent reduction in re-arrests
• While the distributions match this mean estimate very
well, this small interval masks considerable variation in
estimated effects
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
The Problem with Effect Sizes
• The following slides illustrate this problem
• The blue lines indicate the distribution of the
effect size based on the simulations (solid line
is mean estimate, dashed lines are the 95%
confidence interval of the estimate)
• These lines are horizontal because the effect
size estimate is not based on sample size (the
x-axis)
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
The Problem with Effect Sizes
• For each population size displayed on the xaxis, based on the distribution of the effect
size, we simulated the number of arrests
prevented 10,000 times
• Rather than the effect size, these simulations
represent the reduction that courts of different
sizes would likely experience
• This information is displayed by the black lines
• Again, the mean is the solid line and the
dashed lines are the 95% confidence interval
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
0.20
0.15
0.10
0.05
-0.05
0.00
Averted Arrests per Participant
0.25
0.30
Results: Averted Arrests
0
100
200
300
400
500
Number of Participants
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
The Problem with Effect Sizes
• The results indicate the problem with presenting effect
sizes to policymakers
• While the means are identical (as is to be expected) the
95% interval is considerably different
• While the simulations approach the effect size as court
size increases, particularly for small courts, the range of
expected effects is drastically larger than the range of
effect sizes
• This indicates that the specificity of the effect size
estimate inaccurately underestimates the uncertainty in
expected outcomes, which can mislead policymakers
and make researchers seem less credible
• While the effect size is virtually guaranteed to produce
arrest reducing effects, the black line indicates that
there is substantial probability an increase in arrests
will actually be experienced in a small court
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Why Effect Sizes are Misleading
• To further illustrate, the following plots holds
the “true” reduction constant at 15% and
randomly simulates reductions, given different
sized populations
• For each population, there is only 1 simulation
• These simulations are then plotted (the black
lines connect each simulation and the blue line
is the true 15%)
• The results display tremendous variability in
outcome for smaller courts, and demonstrate
that even with 2,000 participants, the “true”
reduction will not always be achieved
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
0.2
0.1
0.0
-0.1
Observed Reduction (true = 15%)
0.3
Why Effect Size is Misleading
0
500
1000
1500
2000
N
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Final Results
• The next slide displays potential reductions in
the number of arrests on the x-axis and the
probability that more than that number of
arrests will actually be prevented on the y-axis
• The blue lines indicate that if, hypothetically,
the mayor was only interested if more than 20
arrests were prevented, there is a 60% chance
that this will occur
• That information is far more useful to the
Executive Office, because it allows them to
determine their own risk aversion
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
0.6
0.4
0.0
0.2
Probability
0.8
1.0
Results: Averted Arrests
-20
0
20
40
60
80
Averted Arrests
n:2500 m:0
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Final Results
• Using the distribution of crimes committed
from a large multi-site study of drug courts
(Rossman, et al, 2011), the distribution of the
costs of victimization from Roman (2009), and
the costs of arrest from Miller and Cohen
(1996), we estimated the benefits of drug
court, and subtracted the costs
• The following plot displays possible target net
benefits per participant (x-axis) and the
probability (y-axis) that the net benefits will
exceed that target
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
0.6
0.4
0.0
0.2
Proportion Greater Than
0.8
1.0
Results: Distribution of Net Benefits
-10000
0
10000
20000
30000
40000
50000
Net Benefits per Participant
n:2500 m:0
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Final Results
• The results highlight the importance of full
distributions of possible outcomes
• The mean expected net benefits (not
displayed) is highly positive, however there is
only a 35% chance that net benefits will
exceed zero
• This is because there’s a small chance of huge
benefits; there’s a 15% chance net benefits
per participant will exceed $30,000
• In other words, drug court will probably not be
cost effective, but there’s a chance it will be
hugely cost effective
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Final Results
• The results indicate the importance of simulation based
modeling
• The true drug court effect is virtually guaranteed to
reduce the likelihood of reoffending
• However, given finite samples (participants) this does
not translate into a guaranteed reduction in arrests
• Still, in all likelihood, drug court will reduce arrests
• Whether or not it is cost effective, however, depends on
the types of arrests that were prevented, which is
determined by chance (it will not always be the same)
• If big, costly crimes are prevented (robbery, assault,
etc.), drug court costs will pay off, but if only small
drug arrests are prevented, it will not
• Simply taking mean net benefits masks this whole story
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Further Examination of Final Results
• What drives this conclusion (benefits unlikely, but there
is some chance of huge payoff)?
• The crimes committed by drug involved offenders are
rarely serious
• Thus, most crimes prevented, most of the time, will be
minor crimes
• Even when serious crimes (robbery, assault) occur,
there’s tremendous variation in the damage done in
particular instances
• Occasionally, a very serious crime is prevented, making
drug court completely worth the investment
• The following plot presents possible costs per prevented
crime (x-axis) and the probability (y-axis) that a
random prevented crime will exceed that
• Clearly, most crimes are nearly insignificant, but there’s
a moderate probability of huge crimes being prevented
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
0.6
0.4
0.0
0.2
Proportion Less Than
0.8
1.0
Costs Per Crime
0
5000
10000
15000
Cost per Crime
n:77104 m:0
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Presentation Overview
• Background
–
–
–
–
Existing Models
DCPI
Drug Courts
Meta-Analysis
• Proposed Model
• Analytic Strategy
• Results
• Next Steps
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
Next Steps
• Finalize the estimates of costs of drug court in DC
(currently based on 15 year old data adjusted for
inflation)
• Code data on other alternatives to incarceration for
meta-analysis and determine what works best in
what circumstances
• Code other interventions to see other effective crime
prevention strategies
• Rank policy priorities based on what is expected to
yield the greatest benefits
• Link to fiscal policy to explore and predict ripple
effects throughout city government agencies
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center
The Last Word
"I would not say that the future is necessarily
less predictable than the past. I think the past
was not predictable when it started."
- Donald Rumsfeld
“All models are wrong, some are useful."
- George E.P. Box and Norman Draper
The views expressed are those of the authors and should not be attributed to the Urban Institute, its
trustees, or its funders.
URBAN INSTITUTE
Justice Policy Center