Transcript Interesting Examples of the Use and Misuse of Statistics from the Law

Interesting Examples of the Use &
Misuse of Statistics from the Law
H. James Norton, PhD
[email protected]
Website:
www.jimnortonphd.com
Oliver Wendell Holmes, Jr.
Oliver Wendell Holmes, Jr.
The Path of the Law
10 Harvard Law Review
(1897): 457-469.
“For the rational study of the law the black
letter man may be the man of the present,
but the man of the future is the man of
statistics and the master of economics.”
People v. Collins (1968)
Crim. No. 11176
Supreme Court of California
March 11, 1968
• A woman had her purse stolen.
• The witnesses did not get a good look at the
robber’s face.
• Witnesses were able to describe some
characteristics of the robber, the get-away
car, and the driver.
• Prosecution calls an Instructor of
Mathematics to testify.
• Instructor explains the product rule for
multiplying probabilities of independent
events.
Prosecutor suggests these probabilities:
•
•
•
•
•
•
Black man with a beard
Man with a moustache
White woman with pony tail
White woman with blonde hair
Yellow automobile
Interracial couple in car
1 in 10
1 in 4
1 in 10
1 in 3
1 in 10
1 in 1000
• Asks instructor what the probability would be under these
estimates.
• 1 in 12,000,000.
• Prosecutor claims these estimates are conservative.
• “Chances of having every similarity … something like
1 in a billion.”
• Jury finds defendant guilty.
The ruling of the appeal’s court:
• “It is a curious circumstance of this
adventure in proof that the prosecutor not
only made his own assertions of these
factors in the hope that they were
conservative… but invited the jury to
substitute their estimates.”
• “There was another glaring defect in the
prosecution’s technique, namely an
inadequate proof of the statistical
independence of the six factors.”
The final ruling of the appeals court:
“Mathematics, a veritable sorcerer in our computerized world,
while assisting the trier of fact in the search for truth,
must not cast a spell over him. We reverse the judgment.”
The Sally Clark Case
• Sally Clark was a solicitor in
Cheshire, England.
• Her son, Harry Clark, born 3 weeks
premature, died 8 weeks after birth.
• In addition, her first child had died less than 3
weeks after birth. His autopsy concluded he
had died of natural causes. He had signs
of a respiratory infection.
• She was arrested for 2 counts of murder,
despite the fact that there was very little
evidence against her.
• Sally had no history of violent or unusual
behavior. Harry had some evidence of
being shaken but this was consistent
with her report to the police that she
had shaken the baby when she noticed
that he was not breathing.
• Prosecutor’s main argument was that it
would be very unlikely that 2 babies in
same family would die of cot death. In the
U.S. we would use the term Sudden Infant
Death Syndrome (SIDS).
Prosecution calls Sir Roy Meadow
Professor of Paediatrics
St. James University Hospital
President British Paediatric Association
1994-1997
His testimony was based on the
Confidential Enquiry for Stillbirths and Deaths,
a study of deaths of babies in infancy,
in 5 regions of England from 1993 to 1996.
• Probability random baby dies of a cot death
= 1 in 1303.
• Probability random baby dies of a cot death if the
mother is > 26 years old, affluent, and a non smoker
= 1 in 8543.
• Probability two children from such a family both die
from a cot death = (1 in 8543) x (1 in 8543)
= 1 chance in 73 million.
• Judge’s summary to jury, “Although we
do not convict people in these courts on
statistics, … the statistics in this case
are compelling.”
• Jury convicts on a 10 to 2 vote.
• One juror said, “Whatever you say about
Sally Clark, you can’t get round the 1 in 73
million figure.”
• Sally’s conviction upheld on appeal.
• 2001, Royal Statistical Society issues a news
brief condemning the use of the multiplication
rule for independence in this case.
• “This approach is statistically invalid. … The
well publicized figure of 1 in 73 million has no
statistical basis.”
• 2002, Ray Hill, Professor of Mathematics at the
University of Salford, analyses other
published data. He concludes the probability
of having a second child die a cot death, given
a first child in a family died a cot death, may
be as high as 1 in 60.
• In 2003, after spending 3 years in jail, Sally’s second
appeal was upheld, and she was released from jail.
This was only after a new pro bono lawyer, while
reviewing the evidence, discovered a pathology
report revealing that Harry was infected with
staphylococcus aureus and that this fact had been
hidden from her defense team.
• Two other women whom Meadow had testified
against at the murder trial of their children were
released upon appeal.
• In 2007, Sally Clark died, of apparently natural
causes, due to acute alcohol intoxication.
New Evidence on S. aureus & SIDS
• “Infection and sudden unexpected death in infancy
(SUDI): a systematic retrospective case review.
M.A. Weber. Lancet May 31 2008;371:1848-53.
• “Significantly more cultures from infants whose
death was unexplained contained S. aureus
(262/1628, 16%) than did those from infants
whose deaths were of a non infective cause
(19/211, 9%, p=0.005).
• From editorial by Morris, “but this work …
provides support for the idea that S. aureus and
E. coli could have a causal role in some cases of
unexplained SUDI.”
The Case of Lucia de Berk
Lucia de Berk, a nurse, was sentenced to life imprisonment in a Dutch court in
2003 on 7 counts of murder and 3 counts of attempted murder in 3 hospitals
where she worked. The circumstantial evidence against her was that she was on
duty when several unexplained deaths had occurred. A law psychologist stated
at the trial that the chance of a nurse working at the 3 hospitals and being
present for so many unexplained deaths was 1 in 342 million. The probability
was based on the number of suspicious deaths on the shifts she worked
versus the number of suspicious deaths on the shifts she did not work.
Her conviction was upheld on appeal in 2004.
• After her incarceration, experts in statistics, toxicology, and
medicine reviewed the evidence used to convict her.
• A report requested by the prosecution after the verdict but prior
to the appeal shows the level of digoxin in the first unexplained
death was not a lethal concentration.
• The way decisions were made concerning the gathering of the
evidence are criticized. For instance in the case of one of the
deaths, 5 experts reported that the death was not suspicious
but 1 expert disagreed and his opinion was used to classify the
death. In another, an expert changed his opinion after
becoming aware of the controversy.
• Statisticians report that the figure 1 in 342 million was
incorrectly calculated and was meaningless. Several do their
own analysis and report estimates of 1 in 50 or 1 in 9.
• A witness against Lucia was a detainee in the criminal
psychological observation unit who testified she told him that
“I released these 13 people from their suffering.” He recanted
his testimony.
• A key toxicologist who testified for the prosecution decided ,
based upon new medical information, that his original
conclusions may not be correct.
• It has been proven that a medical specialist and his assistant
were with the first baby that died and her statement that she
was not near the baby at the time it died was true.
• It has been noted that in one unit 7 suspicious deaths occurred
prior to her working on the unit and only 6 occurred in a similar
time-period after she joined.
• In 2008 the Dutch Supreme Court freed Lucia until a new trial
could take place.
• In December 2009 a court ruled that recently gathered evidence
confirms that 3 of the deaths were due to natural causes.
• Further court hearings took place March 17, 2010.
• Prosecutor says that the state now believes that
Lucia de Berk did not murder or attempt to murder
anyone.
• However the Prosecutor felt obliged to remind the
court that during the investigation they found
evidence that Lucia had committed the following
crimes:
1. Exaggerated her high school grades.
2. Stole a tube of disinfectant gel.
3. Did not return 2 library books.
• While Lucia was in jail for approximately 5 years she
had a stroke.
The Case of Raymond Eason
• In 1999, Raymond Eason, a 49 year old
man living in Swindon England, was
arrested for burglary.
• His DNA had a 6 loci match with evidence
left at the scene.
• Forensic scientists reported to the
prosecutor that this DNA pattern would
only occur in 1 in 37 million people.
• Pretty convincing evidence? But how did
his DNA come to match the evidence at the
crime scene?
• His blood sample had been taken when he was
arrested, but later released, in a domestic dispute.
• He had an alibi that he was home taking care of a
sick daughter. The crime was committed 200 miles
from his home.
• The results of the DNA analysis was entered into
a database with 700,000 suspects.
• Comparing one sample to many others in a
database increases the likelihood of finding
a match by chance.
• The complexity of this statistical calculation
is explained in “The Evaluation of Forensic
DNA Evidence.”
• A rule of thumb, that some forensic scientists use,
is to divide the match probability by the number of
subjects in the database.
• If this is done the match probability is 0.0027
• His lawyer asked for the DNA sample to be
reanalyzed by another laboratory.
• The laboratory found the same 6 loci match,
but found his DNA did not match on 4 newly
tested loci.
• He was released from jail.
• Oh, by the way, Mr. Eason had severe
Parkinson’s disease. He could not drive a
car, and could barely dress himself.
• As the appeals court judge in the Collins
case had warned, the prosecutor had let the
numbers cast a spell over his common
sense!
228 Conn. 610
State of Connecticut v. Roy E. Skipper.
No. 14744. Supreme Court of Connecticut.
Argued Dept 23, 1993. Decided Feb. 22, 1994
Defendant was convicted in the Superior Court of sexual
assault in the second degree and risk to injury of child.
Defendant appealed.
The Appellate Court transferred the case to
The State of Connecticut Supreme Court.
A victim (a young girl) gave a statement to the police. The next
day, the victim’s mother took her for a medical examination that
revealed she was pregnant. The victim had an abortion. One of
the key points for the prosecution at the defendant’s trial was
evidence comparing the DNA of the defendant to the DNA of the
fetus.
An expert witness testified at the trial regarding the defendant’s
paternity index (PI). He calculated this to be 3496, “indicating
that only 1 out of 3497 randomly selected males would have
phenotypes compatible with the fetus in question.” “McElfresh
further testified that the PI could be converted into a statistics
indicating the percentage of the defendant’s probability of
paternity.” He did this using Bayes’ Theorem and derived a
value of 99.97 percent. However, when Bayes’ Theorem is used
in this calculation a prior probability of paternity is required.
Bayes’ Theorem in and of itself does not cast this prior as any
particular number.
McElfresh set the prior odds to 50%. “Generally experts..
choose this number.”
The ruling of the court:
“Although the probability of paternity is the
mathematically equivalent of the PI if a 50% prior
probability is used, the two statistics have different
connotations. In the present case, the paternity index
of 3496 means that if 3497 random males were
chosen, 3496 would not have a DNA profile
compatible with producing the fetus in question
. … It is the application of Bayes’ Theorem that says the
likelihood that this defendant is the father of the fetus in
question is 99.97%.”
“If we assume the presumption of innocence stand
would require the prior probability to be zero, the
probability of paternity in a criminal case would always
be 0 …”
”We conclude that the trial court should not
have admitted the expert testimony stating
a probability of paternity statistic.”
Supreme Court held that the admission of
probability of paternity calculated based on
Bayes’ Theorem was reversible error.
… Reversed and remanded for new trial.
637 Atlantic Reporter, 2d Series, pages 1101-1110
The purported first courtroom case
using statistical evidence reported by
Meier & Zabell (1980)
• The contested will of Silvia Ann Howland by
her niece Hetty Robinson in 1865.
• She claimed an earlier will in 1862 left the
entire estate to her and the will stated no
later will should be honored.
• The attorney for the estate claimed the
signature on the earlier will was a forgery.
• Her attorney called Oliver Wendell Holmes,
Sr., Professor of Anatomy and Physiology at
Harvard, to testify about the authenticity of
the signature.
• He examined the signature under a
microscope and found no evidence of a
forgery.
• The attorney for the executor called
Benjamin Peirce, Professor of Mathematics
at Harvard, to testify.
• Peirce used statistical techniques to decide if
the signature was “too similar” to another
known signature of Silvia Howland.
• His concluded that the “probability of finding
30 matches in a pair of signatures was once
in 2,666 millions of millions of millions.”
• Meier and Zabell criticize Peirce’s methods of
supporting his model by means of a
graphical test of goodness-of-fit.
• They discuss the “use and abuse of the
product rule for multiplying probabilities of
independent events.”
• Court ruled that there was lack of evidence to
support Hatty’s claim that “the 1862 will was
in fulfillment of a contract.”
U.S. v. Burton
The case involved a Sergeant who was
accused of obtaining an illegal copy of an
exam that was used as a basis for
promotions. However, no copy of the exam
was found on the defendant, at his home, or
workplace. The prosecution presented a
statistical argument that they claimed was
strong evidence that the defendant must
have had a copy of the test prior to the exam.
This statistical argument had been used in 4
previous court-martials. I (hjn) was asked to
help refute this assertion.
Evidence against Sergeant Burton
•
•
The testimony of Sergeant X.
The fact that Sergeant Burton was a friend of
another Air Force Sergeant at the Pentagon who
had access to the exam. No charges were ever
brought up against this person.
•
Sergeant Burton scored high on the test,
approximately at the 95th percentile. The
information the test is based upon is from a 120page manual that everyone taking the test has
access to and they are encouraged to study from.
However, Sergeant Burton’s greatest improvement
came the year before when he improved from the
40th percentile to the 85th percentile. There was
no evidence that he had cheated in the prior exam.
•
The statistical procedure developed by a Colonel
in the Mathematics Department at the Air Force
Academy.
The statistical procedure consisted of:
•
For each wrong answer the percent of soldiers
choosing this answer was found. For instance if
Sergeant Burton answered question #1 incorrectly,
for example he choose answer B while answer A
was correct, and 20% of the soldiers answered B,
then 0.20 was recorded.
•
This was done for each of his 11 incorrect
answers.
•
These probabilities were multiplied together.
•
Their conclusion was since the product of these
numbers was very small, he choose an unusual set
of incorrect answers. This implied he had a copy
of the exam prior to the test.
The following are some of the points that I brought up
with the defense lawyers:
•
•
•
•
The method employed seemed to have nothing to
do with proving cheating or even with choosing
unusual answers. Was there any published
reference that justified this procedure?
There was a paradox in their procedure, as the
more questions one got incorrect, the smaller the
probability. Did this make sense that the more
questions one got wrong the more evidence of
cheating?
The simple product rule for probability requires
independence. What evidence was there that the
questions were independent?
How many soldiers had a lower product than
Sergeant Burton?
• A Major from the Air Force Academy was brought in to
defend the Colonel’s statistical method.
• During the pretrial motion to suppress the “statistical
evidence” the Major presented an “ad hoc” method to
show independence of the test questions.
• Military law states that, “the test for determining whether
to admit expert scientific testimony does not apply to
admissibility of expert testimony based on technical or
specialized knowledge.”
• The judge ruled that statistics is not a science but
specialized knowledge.
• “Military justice is to justice what military music is to
music.”
• The Major was rigorously cross-examined at the trial.
• Defense calls only 3 witnesses to testify against Sgt. X.
• Jury deliberates 40 minutes and returns a not-guilty
verdict.
Comparing bullets using Compositional Analysis of Bullet Lead
(CABL)
•
•
“Since the 1960s, FBI testimony in
thousands of criminal cases has relied
on evidence from Compositional
Analysis of Bullet Lead (CABL), a
forensic technique that compares the
elemental composition of bullets found
at a crime scene to the elemental
composition of bullets found in a
suspect's possession. The report
assesses the scientific validity of CABL,
finding that the FBI should use a
different statistical analysis for the
technique, and that, given variations in
bullets manufacturing processes, expert
witnesses should make clear the very
limited conclusions that CABL results
can support. “
Authors: Committee on Scientific
Assessment of Bullet Lead Elemental
Composition Comparison, National
Research Council
Bullet from
Crime Scene
=?
Bullet in
Subject’s Possession