Transcript here - Queen Mary University of London

Impediments and opportunities to using
statistics and probability in legal arguments
The City Law School, London 9 February 2016
Norman Fenton
Director of Risk & Information Management Research (Queen
Mary University of London)
and
CEO of Agena Ltd
[email protected]
Overview
1.
2.
3.
4.
Motivation and relevant rulings on Bayes
Probability fallacies and the law
The likelihood ratio
How to do probability properly in legal
arguments and the challenges involved
5. Conclusions and way forward
Fenton N.E, Neil M, Berger D, “Bayes and the Law”, Annual Review of
Statistics and Its Application, Volume 3, 2016, doi: 10.1146/annurevstatistics-041715-033428
Questions
• What is 723539016321014567 divided by
9084523963087620508237120424982?
• What is the area of a field whose length
is approximately 100 metres and whose
width is approximately 50 metres?
Court of Appeal Rulings
“The task of the jury is to evaluate evidence and
reach a conclusion not by means of a formula,
mathematical or otherwise, but by the joint
application of their individual common sense and
knowledge of the world to the evidence before
them” (R v Adams, 1995)
“..no attempt can realistically be made in the
generality of cases to use a formula to calculate the
probabilities. .. it is quite clear that outside the field
of DNA (and possibly other areas where there is a
firm statistical base) this court has made it clear
that Bayes theorem and likelihood ratios should
not be used” (R v T, 2010)
Ramifications of R v T
Fairly rigorous and ‘correct’ forensic analyses
withdrawn or rewritten
Resulting analyses misleading, meaningless and
even ‘wrong’
The Prosecutor’s Fallacy
Some of those who were at
the scene of the crime
Fred
Police discover
shoeprint of person
who committed the
crime – it’s size 13
Nationally only about 1 in a 100
men are size 13
Fred is size 13
What is the probability Fred is innocent?
Are these statements correct/
equivalent?
1. the probability of finding this evidence p(E|H)
(matching size 13) given Fred is innocent
is 1 in 100
2. the probability Fred is innocent given
p(H|E)
this evidence is 1 in 100
The ‘prosecution fallacy’ is to treat
the second statement as equivalent to
the first
p(H|E) =p(E|H)=1/100 ??
Fred has size 13
Fred has size 13
Imagine 1,000
other people
also at scene
Fred has size 13
About 10
out of the
1,000 people
have size 13
Fred is one of
11 with
size 13
So there is
a 10/11
chance that
Fred
is NOT
guilty
That’s very
different
from
the
prosecution
claim of 1%
How the fallacy is also stated
“The chances of finding this
evidence in an innocent man are
so small that you can safely
disregard the possibility that this
man is innocent”
Ahh.. but DNA evidence is different?
Very low random match probabilities … but
same error
Low template DNA ‘matches’ have high random
match probabilities
Probability of testing/handling errors not
considered: startling new research shows DNA
‘statistics’ are meaningless without this
Principle applies to ALL types of forensic match
evidence
Tip of the Fallacies Iceberg
•
•
•
•
•
•
•
•
Defendant fallacy
Confirmation bias fallacy
Base rate neglect
Treating dependent evidence as independent
Coincidences fallacy
Various evidence utility fallacies
Cross admissibility fallacy
‘Crimewatch UK’ fallacy
Fenton, N.E. and Neil, M., 'Avoiding Legal Fallacies
in Practice Using Bayesian Networks', Australian Journal of
Legal Philosophy 36, 114-151, 2011
The Likelihood Ratio approach
Prosecution likelihood (The probability of seeing the
evidence if the prosecution hypothesis is true)
(=1 in example)
Defence likelihood (The probability of seeing the
evidence if the defence hypothesis is true)
(=1/100 in example)
Likelihood ratio = Prosecutor likelihood
Defence likelihood
Bayes Theorem:
(=100 in example)
Posterior odds = LR x Prior odds
LR > 1 supports prosecution;
LR <1 supports defence
LR = 1 means evidence has no probative value
Likelihood Ratio approach may be
making things worse
Contrary to what is claimed LR does depend on
Bayes (and cannot avoid priors)
Only works if prosecution hypothesis is
negation of defence hypotheses
LR for source-level hypotheses tell us nothing
about offence-level hypotheses
To ensure ‘manual calculations’ LR forces
over-simplification that can lead to
erroneous conclusions
The basic legal argument
H
(hypothesis)
E
(evidence)
• Easy to do Bayes, LR calculation manually
• But in practice this is not the correct model
Even single piece of forensic match
evidence is NOT a 2-node BN
Target is
type X
Target is
source
Target
tested X
Source is
type X
• Calculating correct LR for the
evidence cannot be done
manually
• …But there are standard tools
to do it
Source
tested X
..beware mathematicians who say
“The correct computations needed to
understand statistical evidence such as DNA
profile matches are not mathematically difficult,
it's simply arithmetic”.
+Plus Magazine (Phil Dawid and Rachel Thomas)
https://plus.maths.org/content/os/issue55/features/dnacourt/index
In an otherwise excellent article
Trying to do it manually helped to ‘kill
Bayes’
If you want accurate probabilities
you need to do it properly
Cannot be done manually with
simple LR
Inevitably requires causal
Bayesian Network model
Calculator analogy: Once
assumptions made clear, no
need to explain or understand
how results obtained
Slide 22
Bayesian nets in practice
Exposed problems of Barry
George appeal
Improved understanding of
impact of complex forensic
evidence in several cases
Exposed errors in numerous
cases
All done in ‘background’
Slide 23
‘Valid’ statistical evidence
The elevation of DNA evidence to a
uniquely privileged position is
irrational
DNA evidence, like many other forms
of forensic/trait evidence, contains
much uncertainty and subjective
judgement.
Statistical evaluation of all such
evidence can be done using BNs
Slide 24
Conclusions and way forward
Basic misunderstandings of probability
and stats is compromising justice
Banning Bayes makes things worse
The use of the Likelihood ratio has
massively confused things
Most statistical claims about DNA
evidence are wrong/misleading
Way forward is proper integration of
relevant evidence in causal models with
computations done ‘out of sight’.
Slide 25
http://bayesknowledge.blogspot.co.uk/
https://www.newton.ac.uk/event/fos
http://www.probabilityandlaw.blogspot.co.uk/
Slide 26