Transcript 20160902100010301

Statistics and the Law
Varieties of Statistical Challenges
Varieties of Challenges to Statistics
Conforming Statistics to the Law
• The law sets parameters of relevance and
poses the questions the statistician must
answer.
– May not be questions the statistical expert is best
positioned to answer.
– Standards of proof may not be statistical
standards.
Legal and Statistical Models: Best Case
• Jury Discrimination:
– Statistical and legal issues almost perfectly aligned.
• Jury selection is supposed to be random.
• Easy to model the extent to which a given distribution is
consistent with a random draw from a given population.
– Challenges:
• Determining the characteristics of the relevant population.
• Determining how large a deviation is required to trigger a
legal finding.
Castenada v. Partida, 430 U.S. 432
(1977)
• “As a general rule for such large samples, if the
difference between the expected value and the
observed number is greater than two or three
standard deviation, then the hypothesis that the
jury drawing was random would be suspect to a
social scientist.”
• Grand Jury case, not a petit jury
– “Key Man” system was permitted.
– Mexican-Americans in county dominated politically by
Mexican (79% of county; 50% of grand jury)
• Married women with changed names.
• Non-citizen immigrants.
Legal and Statistical Models: A Not So Good Case
• Regression Models
– Employment discrimination, antitrust, etc.
• Challenges
– The right legal model Sobel v. Yeshiva University,
839 F2d 18 (1988)
– The right scientific model
• Difficult to determine
– Functional Form (OLS is default)
– Included Variables
» Whose burden? Smith v. VCU, 84 F.3d 672 (1995)
Model Mischief Making
• Challenge: Ensure fair, transparent, relevant and
honest statistics. The law does not make this easy.
– Choosing models by their outcomes.
• Variable choices
– E.g. minimizing effects by adding proxies
– P Hacking
• E.g. Concealing approaches tried.
– Avoiding robustness checks
• Love the results – top there
– Ignoring effect sizes
• Increase sample size – get significance.
• Pretend that only significance matters.
– Unjustified assumptions
Answering the Law’s Question
•
DNA evidence.
– Science question: how likely is it that the DNA of a “random person”
will, like the defendant’s DNA, match the crime scene sample.
• In usual case, easy to answer with conservative adjustments.
– Law’s question: how likely is testimony reporting a DNA match if the
defendant did not leave the crime scene sample. This requires
accounting for:
• Random match probability given suspect population
• Likelihood of laboratory error
–
–
–
–
Sample contamination
Mislabeling
Subjective judgments
Perjury
• Complication: .001 probability of receiving testimony of DNA match not the
same as .001 probability that a random person would have matching DNA.
– Latter implies other matching people. (There is something to the “Defense Attorney’s
Fallacy.”)
Other Identification Evidence
• Similar analysis applies to other evidence that
seeks to associate individuals with trace
evidence. (e.g. Fingerprints, Handwriting, Bite
Marks, Shoe or Tire Marks, Tool Marks, Etc.)
– Objective other match probabilities higher.
– Uncertainty greater.
– Subjectivity greater.
Acquiring Necessary Statistics (Data)
• Forensic evidence in general
– Need to estimate error rates (even for most
reliable; e.g., DNA, Chemical Analysis)
• Proxies (lab acredidation, individual certification,
training, experience) of uncertain value.
• Proficiency testing.
– Open: at best sets lower error bound.
– Blind: expensive and hard to do.
• History of error.
– Claimed or real correction.
• Perjury almost impossible to estimate.
Data Needs
• Many forensic technologies
– Need to validate (set bounds on) assumptions.
• Are fingerprints unique.
– How much information is needed for unique identification
» Partial prints
» Smudged prints
– How unique are bite marks?
– What do we know about blood splatter.
» Arson: A cautionary tale.
• Need to assess utility.
– How much better do handwriting experts do than lay jurors?
Communication Challenges
• Sweet spot: Fair and appropriate persuasiveness.
– Incentives run contrary.
• Offeror wants to exaggerate probative value.
• Opponent wants to denigrate probative value.
• Making statistics understandable.
– Difficult to do, especially when both objective match
rates and false reporting rates should both be made
available.
• Jay Koehler shoe print study
• Bayesian dreams.