Transcript 20160902113012001

The Challenges of
Bayes in the Law
2 September 2016
Norman Fenton
Queen Mary University of London and Agena Ltd
Overview
1. Barriers to Bayes in the Law
2. Misunderstanding about probative value of
evidence
3. Likelihood ratio: the good, bad, ugly
4. Overcoming the limitations
www.eecs.qmul.ac.uk/~norman
•
•
•
•
Fenton N.E, Neil M, Berger D, “Bayes and the Law”, Annual Review of Statistics and Its
Application, Volume 3, 2016 (June), pp 51-77
Fenton, N. E. (2014). Assessing evidence and testing appropriate hypotheses. Science & Justice,
54(6), 502-504
Fenton, N. E., Neil, M., & Hsu, A. (2014). "Calculating and understanding the value of any type of
match evidence when there are potential testing errors". Artificial Intelligence and Law, 22. 1-28
Fenton, N. E., D. Berger, D. Lagnado, M. Neil and A. Hsu, (2014). "When ‘neutral’ evidence still has
probative value (with implications from the Barry George Case)", Science and Justice, 54(4), 274287
Legal reasoning: perfect fit for Bayes
We have a prior P(H)
We now get some evidence E.
H
(hypothesis)
E
(evidence)
We know P(E|H)
--But we really want to know the posterior P(H|E)
U S Federal Rules of Evidence 401
Evidence is relevant if: a) it has any tendency to make a fact more or
less probable than it would be without the evidence…
Kilbourne v DPP: . . relevant (i.e. logically probative or disprobative)
evidence is evidence which makes the matter which requires proof more or
less probable.
Court of Appeal Rulings
“The task of the jury is to evaluate evidence and
reach a conclusion not by means of a formula,
mathematical or otherwise, but by the joint
application of their individual common sense and
knowledge of the world to the evidence before
them” (R v Adams, 1995)
“..no attempt can realistically be made in the
generality of cases to use a formula to calculate the
probabilities. .. it is quite clear that outside the field
of DNA (and possibly other areas where there is a
firm statistical base) this court has made it clear
that Bayes theorem and likelihood ratios should
not be used” (R v T, 2010)
Ramifications of R v T
Fairly rigorous and ‘correct’ forensic analyses
withdrawn or rewritten
Resulting analyses misleading, meaningless and
even ‘wrong’
Simple Bayes Example
Some of those who were at
the scene of the crime
Fred
Police discover
shoeprint of person
who committed the
crime – it’s size 13
Nationally only about 1 in a 100
men are size 13
Fred is size 13
What is the probability it
was Fred’s shoeprint on the
victim?
What we know
1. the probability of finding
this evidence (matching size
13) given it was Fred’s shoe
is 1
2. the probability of finding
this evidence (matching size
13) given it was not Fred’s
shoe is 1 in 100
P(E|H)
P(E|not H)
Prosecutor fallacy assumes
P(H| not E) = P(E| not H)
For Bayes we also need the prior P(H)
Assume Fred and 1000 others were at the scene. Then P(H) = 1/1001
Bayes Theorem
P(E|H)*P(H)
P(H|E) = P(E|H)*P(H) =
P(E)
P(E|H)*P(H) + P(E|not H)*P(not H)
P(H|E)
=
1*1/1001
1*1/1001+ 1/100*1000/10001
=
0.000999
 0.091
0.000999 + 0.00999
Hopeless trying to
present this to lawyers
Fred has size 13
Fred has size 13
Imagine 1,000
other people
also at scene
Fred has size 13
About 10
out of the
1,000 people
have size 13
Fred is one of
11 with
size 13
So there is
a 10/11
chance that
Fred
is NOT
guilty
That’s very
different
from
the
prosecution
claim of 1%
Classic objections to Bayes (1)
‘No such thing as probability’
• “He either did it or didn’t do it, so the only valid
probabilities for guilt are 0 or 1”
• “I would reject that approach. It is not only over
formulaic but it is intrinsically unsound. The chances
of something happening in the future may be
expressed in terms of percentage. Epidemiological
evidence may enable doctors to say that on average
smokers increase their risk of lung cancer by X%. But
you cannot properly say that there is a 25 per cent
chance that something has happened . . . Either it
has or it has not.”
Classic objections to Bayes (2)
• Accurate/nonoverpowering prior cannot be devised.
• In using statistical evidence to formulate priors,
jurors might use it twice in reaching a posterior.
• Not all evidence can be considered in probabilistic
terms.
• No probability value can ever be reconciled with
“beyond a reasonable doubt.”
• Owing to complexity of cases any application of
Bayes too cumbersome for a jury to use
• Probabilistic reasoning is not compatible with the
law, for policy reasons.
‘Probative value’: When does
evidence E “support” hypothesis H?
U S Federal Rules of Evidence 401
Evidence is relevant if:
a) it has any tendency to make a fact more or less probable
than it would be without the evidence…
Kilbourne v DPP: . . relevant (i.e. logically probative or disprobative)
evidence is evidence which makes the matter which requires proof
more or less probable.
when the probability of H being true increases
after we find E
i.e. P(H | E) > P(H)
(‘posterior odds’ of H increase over the ‘prior
odds’ of H)
A simple formal definition of probative
value of evidence
𝑃(𝐻|𝐸)
The ratio R:
𝑃(𝐻)
• R > 1 means E supports H
• R < 1 means E supports not H
• R = 1 means E is neutral for H
Why do we never see this definition used?
Because of obsessive and irrational
fear of the explicit PRIOR P(H)
Instead Likelihood Ratio (LR) used
𝑃(𝐸|𝐻)
LR =
𝑃(𝐸|𝒏𝒐𝒕 𝐻)
Bayes Theorem:
Posterior odds of H = LR x Prior odds of H
LR > 1: means E supports prosecution
hypothesis (as P(H|E)>P(H) in this case)
LR <1: means E supports defence hypothesis
(as P(not H|E) > P(not H) in this case)
LR = 1: means E has no probative value (as
P(H|E)=P(H) in this case)
Likelihood Ratio (The Good)
Simple formula for probative value of
evidence
No need to explicitly consider prior for H
Forces forensic experts to consider the
likelihood of both the prosecution
hypothesis and the defence hypothesis
…..unfortunately there are problems
with each of the above
LIMITATION 1
As a measure of probative value
LR only works when the defence
hypothesis is the negation of the
prosecution hypothesis
…in practice this is RARELY adhered to
Limitation 1: Example
H: “Mrs Peacock guilty”
E: “The murderer was
female
P(E | H) = 1
P(E | not H) = 2/5
LR= 2.5
But
if H’: “Miss Scarlet guilty”
P(E | H’) = 1
LR=1
So when the hypotheses H, H’ not exhaustive
• LR>1 only tell us E supports H more than it
supports H’. But E may strongly support not H
over H
• LR=1 only tells us E supports both H and H’
equally. That does NOT mean E is neutral. E
may strongly support H over not H or vice
versa
But in practice it is difficult to work with
exhaustive pairs of hypotheses
LIMITATION 2
When using LR you can never
totally ignore the prior
LIMITATION 3
In general the LR calculation is
NOT simple
LR is only simple in this case
H
(hypothesis)
E
(evidence)
LR calculation does not need Bayes in this
case
Extends to case of multiple independent
evidence
No evidence is ever so simple
H1:
Guilty (y/n)
H2:
Defendant is
source of DNA
at scene (y/n)
E’: Defendant
credibility
H3:
E:
Defendant DNA
‘matches’ DNA
at scene (y/n)
E:
Expert testifies
defendant DNA
‘matches’ DNA
at scene (y/n)
Even single piece of forensic match
evidence is NOT a 2-node model
Target is
type X
Target is
source
Target
tested X
Source is
type X
This is a causal Bayesian network
Calculating correct LR manually
extremely complex
…But there are standard tools to do it
Source
tested X
Fenton, N. E., Neil, M., & Hsu, A. (2014). "Calculating and understanding the value of any type of
match evidence when there are potential testing errors". Artificial Intelligence and Law, 22. 1-28
Correct calculation of LR for DNA
match evidence reveals exaggerated
claims for its probative value
Fenton, N. E., Neil, M., & Hsu, A. (2016) “Why increasingly small DNA match
probabilities have exactly the opposite impact to what is expected”
Fenton, N. E., Neil, M., & Hsu, A. (2014). "Calculating and understanding the value of
any type of match evidence when there are potential testing errors". Artificial
Intelligence and Law, 22. 1-28
LIMITATION 4
LR for source level hypothesis
tells us nothing about offence
level hypothesis
Limitation 4: Example
LR=1 for source level hypotheses…..
but E has real probative value on Hp
R v Barry George
Fenton, N. E., D. Berger, D. Lagnado, M. Neil and A. Hsu, (2014). "When ‘neutral’ evidence
still has probative value (with implications from the Barry George Case)", Science and
Justice, 54(4), 274-287
The Barry George case
George fired gun
Evidence George
fired gun
LIMITATION 5
Any serious LR calculation
requires an automated BN model
Do not attempt a formulaic
calculation from first principles
..but this is a
typical real BN
Hence the Calculator Analogy
Summary
• Bayes is natural fit for legal reasoning
• Reliance on priors and maths are fundamental
impedements
• LR and probative value of evidence may not be what
people think it is
• In isolation the LR may be highly misleading
• LR does depend on Bayes (and cannot avoid priors)
• LR only works if prosecution hypothesis is negation
of defence hypotheses (difficult in practice)
• LR of source level hypotheses tells us NOTHING
about LR of offense level hypotheses
• To ensure ‘manual calculations’ LR forces oversimplifications that lead to erroneous conclusions
• Doing things correctly requires fuller models - BNs