Transcript Workshop on Current Firearms and Toolmark Research

CCI Firearms and Toolmark Examiner Academy
Workshop on Current Firearms and Toolmark
Research
3
2
1
0
1
2
3
Pushing Out the Frontiers of Forensic Science
Outline
• Morning-ish
• Introduction and the Daubert Standard
• Confocal Microscopy
• Focus Variation Microscopy
• Interferometric Microscopy
• Surface Data/Filtering
Outline
• Afternoon-ish
• Similarity scores and Cross-correlation functions
• Known Match/Known Non-Match Similarity
Score histograms. False Positives/False
Negatives/Error Rates
• Multivariate Discrimination of Toolmarks
• Measures of “Match Quality”
• Confidence
• Posterior Error Rate/Random Match
Probability
• Lessons learned in conducting a successful
research project
Introduction
• DNA profiling the most successful application of
statistics in forensic science.
• Responsible for current interest in “raising standards” of
other branches in forensics…??
• No protocols for the application of statistics to
comparison of tool marks.
• Our goal: application of objective, numerical
computational pattern comparison to tool marks
Caution: Statistics is not a panacea!!!!
The Daubert Standard
• Daubert (1993)- Judges are the
“gatekeepers” of scientific evidence.
• Must determine if the science is reliable
• Has empirical testing been done?
• Falsifiability
• Has the science been subject to peer review?
• Are there known error rates?
• Is there general acceptance?
• Federal Government and 26(-ish) States
are “Daubert States”
Tool Mark Comparison Microscope
G. Petillo
4 mm
G. Petillo
5/8” Consecutively manufactured chisels
Known Match Comparisons
G. Petillo
5/8” Consecutively manufactured chisels
Known NON Match Comparisons
G. Petillo
5/8” Consecutively manufactured chisels
4 mm
4 mm
600 um
Confocal Microscope
Marvin Minsky
First confocal microscope
Confocal Microscopes
Source
Illumination aperture
Detector
Confocal pinhole
Out of focus light
In focus light
Objective lens
Tool mark surface
(profile of a striation pattern)
Focal plane
for objective
Sample stage
Rastering pattern of
laser confocal
Nipkow disk sweeps
many pinholes
Programmable array Illumination/Detection
Get any illumination/detection pattern
Objective’s focal plane
Sample stage
Scan stage in
“z”-direction
Detector
Objective’s focal plane
Sample stage
Scan stage in
“z”-direction
Detector
Objective’s focal plane
Sample stage
Scan stage in
“z”-direction
Detector
Objective’s focal plane
Sample stage
Scan stage in
“z”-direction
Detector
Objective’s focal plane
Sample stage
Scan stage in
“z”-direction
Detector
Objective’s focal plane
Scan stage in
“z”-direction
Sample stage
Detector
Objective’s focal plane
Scan stage in
“z”-direction
Sample stage
Detector
Objective’s focal plane
Scan stage in
“z”-direction
Sample stage
Detector
Objective’s focal plane
Scan stage in
“z”-direction
Sample stage
Detector
Objective’s focal plane
Scan stage in
“z”-direction
Sample stage
Detector
Objective’s focal plane
Scan stage in
“z”-direction
Sample stage
Detector
Objective’s focal plane
Scan stage in
“z”-direction
Sample stage
Detector
Objective’s focal plane
Scan stage in
“z”-direction
Sample stage
Detector
Objective’s focal plane
Scan stage in
“z”-direction
Sample stage
Detector
Objective’s focal plane
Scan stage in
“z”-direction
Sample stage
Detector
Objective’s focal plane
Scan stage in
“z”-direction
Sample stage
Detector
Objective’s focal plane
Scan stage in
“z”-direction
Sample stage
Detector
Objective’s focal plane
Scan stage in
“z”-direction
Sample stage
Detector
Objective’s focal plane
Scan stage in
“z”-direction
Sample stage
Detector
Objective’s focal plane
Scan stage in
“z”-direction
Sample stage
Detector
Objective’s focal plane
Scan stage in
“z”-direction
Sample stage
Detector
Objective’s focal plane
Scan stage in
“z”-direction
Sample stage
Detector
Objective’s focal plane
Scan stage in
“z”-direction
Sample stage
Detector
For Each Detector Pixel:
Intensity
Axial Response Function
Point on surface
corresponding to
pixel’s is in maximum
focus here
z
Record the “axial response” as stage is moved along the z-direction
Increasing surface height
All-in-Focus 2D Image
Overlay confocal “z-stack”
• 3D confocal image of portion of chisel striation pattern
Confocal Microscope Trivia
• Use high NA objectives for best results
Optical slice thickness =
• Small working distances
• Flanks up to ~ 70o
• Cost ~150K – 250K (FTI IBIS ~1M)
• Get a vibration isolation table for your instrument ~7K
• Set up in a (dry) basement if possible
• Accuracy down to +/- 10 nm
Confocal Microscope Trivia
• Some manufactures:
• Olympus
• LEXT (Laser)
• Zeiss
• CSM (White Light)
• LSM (Laser)
• Nanofocus
• msurf series (White Light)
• Sensofar/Leica
• Plu series/DCM (White Light)
Focus Variation Microscope
“Low res” common
Focus variation mic
~ +/- 1mm
Scherer and Prantl
Detector
Source
Out of focus light
In focus light
Objective lens
Tool mark surface
(profile of a striation pattern)
Focal plane
for objective
Sample stage
Cutaway
Alicona, GMBH
Objective’s focal plane
Sample stage
Scan stage in
“z”-direction
For Each Detector Pixel:
Detector
Focus Measure
Axial Focus Function
Point on surface
corresponding to
pixel is in maximum
focus here
z
Record the “axial response” as stage is moved along the z-direction
Focus Determination:
Detector
Pixel of interest
Compute standard deviation (sd) of pixels grey values
in the neighborhood
A pixel in focus sits in a neighborhood with a large sd
Focus Variation Microscope Trivia
• Use high NA objectives for best results
• Can use external light
• Large working distances
• Flanks up to ~75o
• Cost ~200K – 250K.
• 80K models WON’T have the vertical resolution
needed for forensic work
• Get a vibration isolation table for your instrument ~7K
• Set up in a (dry) basement if possible
• Accuracy down to +/- 10nm
Focus Variation Microscope Trivia
• Some manufactures:
• Alicona
• IFM
• Can get optional rotational stage
• Sensofar/Leica
• S neox/DCM
Interferometer
Fixed mirror
recombine
split
Incoming wave
Path lengths equal
Recombine in-phase
Movable mirror
Interferometer
Fixed mirror
recombine
split
Incoming wave
Path lengths NOT equal
Recombine out-of-phase
Movable mirror
Interferometric Height Measurement
• The basic idea:
•
•
•
•
Each surface point is a “fixed mirror”
Move a reference mirror in objective
Split beams recombine in and out of phase
Constructive interference occurs when
surface points in focal plane
• Infer the surface heights from where
constructive interference occurs
Interferometric Microscope
Wyant
Early Interferometric
Microscope for Surafce Metrology
James Wyant
Wyant
Early Interferometric
Microscope
Modern Interferometric
Microscope for Surafce Metrology
Source
Camera (Detector)
Microscope
Configuration
Reference mirror
Objective lens
Piezo
Scan objective for
Interference in
“z”-direction
Beam-splitter
Tool mark surface
(profile of a striation pattern)
Path lengths equal
Point in focus
Focal plane
for objective
Sample stage
Source
Camera (Detector)
Microscope
Configuration
Reference mirror
Objective lens
Piezo
Scan objective for
Interference in
“z”-direction
Beam-splitter
Tool mark surface
(profile of a striation pattern)
Path lengths un-equal
Point in out of focus
Focal plane
for objective
Sample stage
Interference Objectives
Michelson objective
~ 2X – 10X
Mirau objective
~ 10X – 100X
Linnik objective
+ 100X
For Each Detector Pixel:
Detector
Intensity
Pixel Interference Pattern
Point on surface
corresponding
To pixel’s is in
maximum focus
here
z
Record each pixels interference pattern as objective is scanned
Inference patterns:
Scan objective for
Interference in
“z”-direction
Sample stage
Fringes
Bruker NSD
Fringe Pattern
Bruker NSD
Surface
Turn Fringes Into A Surface
Intensity for each detector pixel:
Fourier transform I(z) to get q(k)
with:
arg[q(k)]
Compute surface heights
A
q
k0
k
deGroot
Interferometry Trivia
• Use high NA objectives for best results
• Small working distances
• Flanks up to ~25o
• Cost ~200K – 250K.
• Get a vibration isolation table for your instrument ~7K
• Set up in a (dry) basement if possible
• Comes in two modes
• VSI: Accuracy +/- 10nm
• PSI: Accuracy below 1nm
Interferometry Trivia
• Some manufactures:
• Bruker (Acquired WYKO/Veeco)
• Taylor Hobson
• Sensofar/Leica
• S neox/DCM
Surface Data
Land Engraved Area:
Surface heights (mm)
37.88
37.88
37.86
37.84
37.81
37.81
37.82
37.84
37.84
37.80
37.89
37.89
37.85
37.82
37.80
37.78
37.80
37.81
37.80
37.77
37.89
37.87
37.84
37.81
37.80
37.79
37.80
37.80
37.76
37.76
37.90
37.87
37.84
37.81
37.82
37.82
37.83
37.81
37.77
37.74
37.92
37.87
37.84
37.83
37.84
37.85
37.87
37.84
37.78
37.79
37.91
37.85
37.85
37.85
37.86
37.89
37.91
37.89
37.86
37.84
Point are “double precision”: 64-bits/point
37.93
37.89
37.85
37.88
37.89
37.94
37.98
37.95
37.92
37.90
37.93
37.92
37.92
37.92
37.94
37.96
37.99
37.99
37.96
37.93
37.94
37.97
37.98
37.97
37.98
38.00
38.02
38.01
38.00
37.98
BIG FILES!
37.99
38.02
38.03
38.04
38.05
38.04
38.05
38.06
38.03
38.00
Surface Data
Land Engraved Area:
Detector levels (16-bit values):
16617
16618
16606
16600
16585
16587
16591
16597
16599
16581
16622
16620
16602
16589
16583
16572
16581
16586
16581
16567
16622
16613
16600
16587
16583
16579
16583
16583
16566
16562
16625
16613
16597
16587
16588
16590
16594
16585
16569
16556
16632
16610
16597
16594
16599
16604
16610
16597
16574
16575
16629
16605
16603
16603
16608
16622
16630
16623
16607
16597
Point are detector grey levels: 16-bits/point
16638
16622
16604
16616
16619
16641
16661
16646
16634
16625
16639
16632
16632
16632
16643
16652
16663
16666
16651
16640
16645
16656
16662
16658
16662
16669
16679
16674
16669
16660
16665
16676
16684
16686
16689
16688
16692
16695
16683
16671
Smaller files. Convert to mm in RAM
Surface Data Trivia
• Different systems use different storage formats
• Be aware if writing custom apps. ASK COMPANY
FOR FILE FORMAT!
• Alicona: Saves surface data as doubles. HUGE
FILES!
• Zeiss: Saves surface data as 16-bit grey levels with
conversion factor
• Other?? 24, 32-bit detectors now??
• Need to standardize file format!
• X3DZhang,Brubaker
• Digital-Surf .surPetraco
Surface Filtering
• Think of a toolmark surface as being made up of a
series of waves
Amplitude (mm)
−1.0
0.0
1.0
Sinusoid 1
0
2
4
x (mm)
6
8
Amplitude (mm)
−0.4
0.0
0.4
Sinusoid 2
Amplitude (mm)
−1.5 −0.5 0.5 1.5
(Fake) Profile = Sinusoid 1 + Sinusoid 2
0
0
2
4
x (mm)
6
8
2
4
x (mm)
6
8
Surface Filtering
• Examine different scales by “blocking out” (filtering)
some of the sinusoids
Filter Transmission Characteristic
Amplitude (%)
0 20
60
Amplitude (mm)
−1.5 −0.5 0.5 1.5
(Fake) Profile = Sinusoid 1 + Sinusoid 2
0.0
0.4
4
x (mm)
−0.4
2
Re(ifft(zf * fs))
0
6
8
“Low Pass” filter blocks high frequencies
and passes low frequencies
(long wavelengths)
5e−03
5e−02
5e−01
wavelength (mm)
5e+00
Surface Filtering
• Examine different scales by “blocking out” (filtering)
some of the sinusoids
Filter Transmission Characteristic
“High Pass” filter blocks low frequencies
and passes high frequencies
(short wavelengths)
0
Amplitude (%)
40
80
Amplitude (mm)
−1.5 −0.5 0.5 1.5
(Fake) Profile = Sinusoid 1 + Sinusoid 2
6
5e−03
8
5e−02
5e−01
wavelength (mm)
0.0
0.2
4
x (mm)
−0.3
2
Re(ifft(zf * fs))
0
0
2000
4000
6000
8000
5e+00
Surface Filtering Trivia
• Wavelength “cutoffs”
lcut
Amplitude (%)
40
80
A “Low Pass” filter
Filter Transmission Characteristic
A “High Pass” filter
lcut
0
Amplitude (%)
0 20
60
Filter Transmission Characteristic
5e−03
5e−02
5e−01
wavelength (mm)
5e+00
5e−03
• Wavelength ranges
• Short wavelengths passed: roughness
• Medium wavelengths passed: waviness
• Long wavelengths passed: form
5e−02
5e−01
wavelength (mm)
5e+00
Surface Filtering
• Band-pass filter: Select narrow wavelength bands to
keep.
• High-pass/Low-pass combinations (Filter banks)
• Wavelets are great at doing this
Statistics
Weapon
Mark Association
– What measurement techniques can be used to obtain
data for toolmarks?
– What statistical methods should be used?
• How do we measure a degree of confidence for an
association, i.e. a “match”?
• What are the identification error rates for different methods
of identification?
Why
?
• R is not a black box!
• Codes available for review; totally transparent!
• R maintained by a professional group of
statisticians, and computational scientists
• From very simple to state-of-the-art procedures
available
• Very good graphics for exhibits and papers
• R is extensible (it is a full scripting language)
• Coding/syntax similar to MATLAB
• Easy to link to C/C++ routines
Why
?
• Where to get information on R :
• R: http://www.r-project.org/
• Just need the base
• RStudio: http://rstudio.org/
• A great IDE for R
• Work on all platforms
• Sometimes slows down performance…
• CRAN: http://cran.r-project.org/
• Library repository for R
• Click on Search on the left of the website to search for
package/info on packages
Finding our way around R/RStudio
Common Computational Practice
• Gauge similarity between tool marks with
one number
• Similarity “metric” is a function which
measures “sameness”
• Only requirement: s(A,B) = s(B,A)
• There are an INFINITE number of ways to
measure similarity!!
• Often max CCF is used.
Cross-correlation
0.0 1.0
−1.5
z (mm)
NIST LEA 1
0
200
400
600
800
1000
1400
−1.5 0.0 1.0
z (mm)
NIST
LEA 2
x (mm)
0
200
400
600
800
1000
NIST
1 vs.
NIST
2
1400
0.0
−0.2
CCF
x (mm)
−1000
−500
0
Lag
500
1000
Cross-correlation
R2 = - 0.04
−1.5
0.0 1.0
Lag = 0
−1.5
−1.0
−0.5
0.5
1.0
R2 = 0.17
−1.5
0.0 1.0
Lag = 45
0.0
−1.5
−1.0
−0.5
0.0
0.5
1.0
−0.5 0.0 0.5 1.0
KNM canRef=Blue,
sometimesSliding=Red
have high max-ccf…
0
2000
4000
6000
8000
Index
max-ccf:
0.751
0.0
0.4
0.8
Ref=Blue, Sliding=Red
0
2000
4000
6000
8000
Cross-Correlation
• Glock primer shear: Each profile ~2+ mm
• Lag over 2000 units (~0.8 mm)
• Max CCF distributions
Scores from
“Known Matches”
Scores from
“Known Non-Matches”
2.0
1.5
1.0
0.5
0.0
Density
2.5
3.0
3.5
Max CCF Similarity Score Distributions
0.0
0.2
0.4
0.6
0.8
1.0
Max CCF Score
We thought: Ehhhhhh…….
Multivariate Feature Vectors
• Random variables - All measurements have an
associated “randomness” component
• Randomness –patternless, unstructured,
typical, total ignoranceChaitin, Claude
• For an experiment/observation, put many
measurements together into a list
• Collection random variables into a list called a
random vector
1. Also called: observation vectors
feature vectors
Multivariate Feature Vectors
• Potential feature vectors for surface
metrology
•
•
•
•
Entire surfaces
*Surface profiles
Surface/profile parameters
Surface/profile Fourier transform or wavelet
coefficients
• Translation/rotation/scale invariant surface (image)
moments
Mean total profile:
0.0 1
-0.5 0
rep(0pr, olefingtlbar[dprerhco(iodedvfier.lzeriv.ozs]eros))
Barcode
representation
0.5 2
1.0
Mean waviness
profile:
-1.0-1
Waviness profile
0
1000
2000
3000
1:length(barcode)
1:length(profil)
deriv.zeros
4000
5000
Consecutive Matching Striae (CMS)-Space
Toolmarks (screwdriver striation profiles) form database
Biasotti-Murdock Dictionary
Some Important Terms
• Latent Variable: weighted combination of
experimental variables into a new “synthetic”
variable
• Also called: scores, components or factors
• The weights are called loadings
• Most latent variables we will study are
linear combinations between experimental
variables and loadings:
• Dot prod. between obs. vect. and loading vect.
gives a score: z = x · a
Principal Component Analysis
• PCA:
• Is a rotation of reference frame
• Gives new PC directions’ relative importance
• PC variance
Principal Component Analysis
• Technically, PCA is an eigenvalue-problem
• Diagonalize some version of S or R to get a PCs
• Typically SAPC = APCs
covariance
matrix of PC
matrix
“loadings”
matrix of PC
variances
• For a data frame of p variables, there are p
possible PCs.
• s ≅ PC importance, dimension reduction
• Scores are data projected into space of
PCs retained ZPC = XAPC
• Scores plots, either 2D or 3D
Setup for Multivariate Analysis
• Need a data matrix to do machine learning
é X11 .. X1 j
ê
:
ê :
ê
X = Xi1 .. Xij
ê
ê :
:
ê X
êë n1 .. X nj
.. X1 p ù
ú
: ú
.. Xip ú
ú
: ú
.. X np úú
û
Represent as a
vector of values
{-4.62, -4.60, -4.58, ...} • Each profile or surface is a row in the data
matrix
• Typical length is ~4000 points/profile
• PCA can:
• 2D surfaces are far longer
• Remove much of the redundancy
• HIGHLY REDUNDANT representation
• Make discrimination computations
of surface data
far more tractable
• 3D PCA of 1740 real and simulated mean profiles of striation
patterns from 58 screwdrivers:
• How many PCs should we
use to represent the data??
•
•
•
~45% variance retained
No unique answer
FIRST we need an algorithm
to I.D. a toolmark to a tool
Support Vector Machines
• Support Vector Machines (SVM) determine
efficient association rules
• In the absence of specific knowledge of probability
densities
SVM decision boundary
Support Vector Machines
• SVM computed as optimization of “Lagrange
multipliers”
1
max å l - å å l l y y k(x , x ) 0 £ l £ C
i
2
n
l
n
n
i
i =1
i
j i
j
i
j
i =1 j =1
• Quadratic optimization problem
• Convex => SVMs unique unlike NNs
• k(xi,xj) kernel function
• “Warps” data space and helps to find separations
• Many forms depending on application: linear, rbf usually
• C: penalty parameter
• control the margin of error between groups that are not
perfectly separable: 0.1 to 10 usually
Support Vector Machines
• The SVM decision rule is given as:
æ n
ö
g(xi ) = sign ç å l j y j k(xi , x j ) + b÷
è j =1
ø
• Equation for a plane in “kernel space”
• Multi group classification handled by “voting”
PCA-SVM
• How many Principal Components should we
use?
With 7 PCs, expect ~3% error rate
With 13 PCs, expect ~1% error
rate
Canonical Variate Analysis
• This supervised technique is called Linear
Discriminant Analysis (LDA) in R
• Also called Fisher linear discriminant analysis
• CVA is closely related to linear Bayes-Gaussian
discriminant analysis
• Works on a principle similar to PCA: Look for
“interesting directions in data space”
• CVA: Find directions in space which best separate
groups.
• Technically: find directions which maximize ratio of
between group to within variation
Canonical Variate Analysis
Project on PC1:
Not necessarily good
group separation!
Project on CV1:
Good group separation!
Note: There are #groups -1 or p CVs
which ever is smaller
Canonical Variate Analysis
• Use between-group to within-group covariance
matrix, W-1B to find directions of best group
separation (CVA loadings, Acv):
(
)
W-1B A cv = A cvs
• CVA can be used for dimension reduction.
• Caution! These “dimensions” are not at right
angles (i.e. not orthogonal)
• CVA plots can thus be distorted from reality
• Always check loading angles!
• Caution! CVA will not work well with very
correlated data
Canonical Variate Analysis
• Distance metric used in CVA to assign group
i.d. of an unknown data point: arg min éëd ( x ) ùû
j =1,...k
T
j
(
d j (xunk ) = -x A cvA
T
cv
)x
unk
(
grp j
unk
)
1 T
T
+ x j A cvA cv
x j - ln [ Pr(grp j)]
2
• If data is Gaussian and group covariance structures
are the same then CVA classification is the same as
Bayes-Gaussian classification.
Canonical Variate Analysis
• 2D/3D-CVA scores plots of RB screwdrivers
2D CVA
3D CVA
PCA vs. CVA
• 2D scores plots of RB screwdrivers:
2D PCA of striation pattern mean profiles
2D CVA of striation pattern mean profiles
Error Rate Estimation
• Discriminant functions are trained on a finite
set of data
• How much fitting should we do?
• What should the model’s dimension be?
• Model must be used to identify a piece of
evidence (data) it was not trained with.
• Accurate estimates for error rates of decision
model are critical in forensic science applications.
• The simplest is apparent error rate:
• Error rate on training set
• Lousy estimate, but better than nothing
Error Rate Estimation
• Cross-Validation: hold-out chunks of data set
for testing
• Known since 1940s
• Most common: Hold-one-out
• Bootstrap: Randomly selection of observed
data (with replacement)
• Known since the 1970s
• Can yield confidence intervals around error rate
estimate
• The Best: Small training set, BIG test set
18D PCA-SVM Primer Shear I.D. Model, 2000 Bootstrap Resamples
Refined bootstrapped I.D. error rate for primer shear striation patterns= 0.35%
95% C.I. = [0%, 0.83%]
(sample size = 720 real and simulated profiles)
How good of a “match” is it?
Conformal PredictionVovk
• Confidence on a scale of 0%-100%
• Testable claim: Long run I.D. errorrate should be the chosen
significance level
• This is an orthodox “frequentist”
approach
• Roots in Algorithmic Information
Theory
• Data should be IID but that’s
it
Cumulative # of Errors
• Can give a judge or jury an easy to understand measure of
reliability of classification result
80% confidence
20% error
Slope = 0.2
95% confidence
5% error
Slope = 0.05
99% confidence
1% error
Slope = 0.01
Sequence of Unk Obs Vects
How Conformal Prediction works for us
• Given a “bag” of obs with known identities and one obs of
unknown identityVovk
• Estimate how “wrong” labelings are for each observation with a nonconformity score (“wrong-iness”)
• For us, one-vs-one SVMs:
1 k ( k -1)/2
ti =
å l
k - 1 j=1 i, j
• Looking at the “wrong-iness” of known observations in the bag:
• Does labeling-i for the unknown have an unusual amount of
“wrong-iness”??:
ppossible-ID =
i
{
# j Î{1,2,..., n}: t j
possible-IDi
n
³ ttest-pattern i
possible-ID
}
i Î{1,2,..., k I.D.s}
• If not:
• ppossible-ID ≥ chosen level of significance a
i
• Put IDi in the (1 - a )*100% confidence interval
Conformal Prediction
• For 95%-CPT (PCA-SVM) confidence intervals will not
contain the correct I.D. 5% of the time in the long run
• Straight-forward validation/explanation picture for
court
Empirical Error Rate: 5.3%
Theoretical (Long Run)
Error Rate: 5%
14D PCA-SVM Decision Model
for screwdriver striation patterns
Conformal Prediction Drawbacks
• CPT is an interval method
• Can (and does) produce multi-label I.D. intervals
• A “correct” I.D. is an interval with all labels
• Doesn’t happen often in practice…
• Empty intervals count as “errors”
• Well…, what if the “correct” answer isn’t in the database
• An “Open-set” problem which Champod, Gantz and
Saunders have pointed out
• Must be run in “on-line” mode for LRG Pr(error) £ a
• After 500+ I.D. attempts run in “off-line” mode we noticed
in practice Pr(error) » a
How good of a “match” is it?
Efron Empirical Bayes’
• An I.D. is output for each questioned
toolmark
• This is a computer “match”
• What’s the probability it is truly not a
“match”?
• Similar problem in genomics for detecting
disease from microarray data
• They use data and Bayes’ theorem to get an
estimate
No diseasegenomics = Not a true “match”toolmarks
Empirical Bayes’
• We use Efron’s machinery for “empirical
Bayes’ two-groups model”Efron
• Surprisingly simple!
• Use binned data to do a Poisson regression
• Some notation:
• S-, truly no association, Null hypothesis
• S+, truly an association, Non-null hypothesis
• z, a score derived from a machine learning task
to I.D. an unknown pattern with a group
• z is a Gaussian random variate for the Null
Empirical Bayes’
• From Bayes’ Theorem we can getEfron:
Estimated probability of not a true
“match” given the algorithms'
output z-score associated with its
“match”
(
) ( )
ˆ
p
z
|
S
ˆ S- | z =
ˆ SPr
Pr
fˆ ( z )
(
)
Names: Posterior error probability (PEP)Kall
Local false discovery rate (lfdr)Efron
• Suggested interpretation for casework:
• We agree with Gelaman and ShaliziGelman:
“…posterior model probabilities …[are]… useful as tools for prediction and for
understanding structure in data, as long as these probabilities are not taken too seriously.”
(
)
ˆ S- | z = Estimated “believability” of machine made association
1- Pr
Empirical Bayes’
• Bootstrap procedure to get estimate of the KNM distribution of
“Platt-scores”Platt,e1071
• Use a “Training” set
• Use this to get p-values/z-values on a “Validation” set
• Inspired by Storey and Tibshirani’s Null estimation methodStorey
• Use SVM to get KM and KNM “Platt-score” distributions
• Use a “Validation” set
From fit histogram by Efron’s method get:
(
fˆ ( z ) “mixture” density
)
pˆ z | S- z-density given KNM => Should be Gaussian
( )
ˆ S- Estimate of prior for KNM
Pr
What’s the point??
We can test the fits to
z-score
(
)
fˆ ( z ) and pˆ z | S !
Posterior Association Probability: Believability Curve
12D PCA-SVM locfdr fit for
Glock primer shear patterns
+/- 2 standard errors
Poisson (Efron) on test set
Bayesian Poisson with intercept on test set
Bayesian Poisson on test set
Bayesian over-dispersed Poisson with intercept on test set
Bayes Factors/Likelihood Ratios
• In the “Forensic Bayesian Framework”, the Likelihood
Ratio is the measure of the weight of evidence.
• LRs are called Bayes Factors by most statistician
• LRs give the measure of support the “evidence” lends to
the “prosecution hypothesis” vs. the “defense hypothesis”
• From Bayes Theorem:
LR =
(
Pr E | H p
Pr ( E | H d
(
Pr H p | E
)
) = Pr ( H | E ) = Posterior Odds
Prior Odds
) Pr ( H )
d
p
Pr ( H d )
Bayes Factors/Likelihood Ratios
• Once the “fits” for the Empirical Bayes method are
obtained, it is easy to compute the corresponding
likelihood ratios.
o Using the identity:
(
)
ˆ S- | z
Pr(H p | E) = 1- Pr
the likelihood ratio can be computed as:
LR(z) =
(
ˆ S- | z
1- Pr
ˆ S- | z
Pr
(
)
ˆ (S )
1 - Pr
ˆ (S )
Pr
-
-
)
Bayes Factors/Likelihood Ratios
• Using the fit posteriors and priors we can obtain the likelihood ratiosTippett, Ramos
Known match LR values
Known non-match LR values
Empirical Bayes’: Some Things That Bother Me
• Need a lot of z-scores
• Big data sets in forensic science largely don’t exist
• z-scores should be fairly independent
• Especially necessary for interval estimates around
lfdrEfron
• Requires “binning” in arbitrary number of intervals
• Also suffers from the “Open-set” problem
• Interpretation of the prior probability for this
application
• Should Pr(S-) be 1 or very close to it? How close?
How to Carry Out a “Successful” Research Project
The Synergy Between Practitioners and Academia
Collaboration
• Practitioners:
• Think about what questions you want to be able to
answer with data BEFORE experimentation
• Write down proposed questions/design
• Be aware that the questions you want answers too
MAY NOT have answers
• What can you answer??
• Be aware that a typical research project takes 1-2
years to complete
Collaboration
• Practitioners:
• Research projects are NOT just for interns!
• Interns typically need tremendous supervision
for scientific/applied statistical research
• Take a college course on statistics/experimental
design
• Rate-my-professor is your friend!
• Visit local university/company websites to look
for the outside expertise you may need.
• Visit the department, go to some seminars
Collaboration
• Academics/Research consultants:
• Be aware practitioners cannot just publish
whenever and whatever they want
• Long internal review processes!
• COMMUNICATION!!!!!
• Listen carefully to the needs/questions of
collaborating practitioners
• Negotiate the project design
• What kind of results can be achieved within a
reasonable amount of time?
• Hold regular face to face meetings if possible
Collaboration
• Academics/Research consultants:
• Applied research is not just for
undergraduates/high-school interns!
• Visit the crime lab!!!!!
• Watch the practitioners do their job.
• Learn the tools they use day to day!
• Microscopy!!!!!
• Use their accumulated experience to help guide
your design/desired outcomes
• What do they focus on??
Fire Debris Analysis Casework
• Liquid gasoline samples recovered during
investigation:
• Unknown history
• Subjected to various real world conditions.
• If an individual sample can be discriminated from the
larger group, this can be of forensic interest.
• Gas-Chromatography Commonly Used to ID gas.
• Peak comparisons of chromatograms difficult and time
consuming.
• Does “eye-balling” satisfy Daubert, or even Frye .....????
• 2D PCA
• 97.3% variance retained
• Avg. LDA HOO correct classification rate: 83%
PC 2
0.5
6
6
66
6
6
0.4
6
0.3
4 19
1
18
18
0.2
1
9
9
9
1 1
4
18
4 4
-2.5
5
5
5
5 55
4 4 4 15
16
15 158 1616
8
11
11
11
1
0.1
1
19
5
14
14
7
17
17
10
7
7
7 7
-1.5
10
7
12
12
88
12
2
22
333 3
3
3
2
3
2 2
-1
-0.1
-0.2
10
2
7
1
13
13
13
8
8 20 20
20
19
14
17
8
PC 1
• 2D CVA
• Avg. LDA HOO correct classification rate: 92%
CV 2
55
5 55
5
0.04
0.02
-0.1
1818
-0.08
-0.06
18
13
13
-0.0413
CV 1
-0.02
16
1616
14
14 14
11
11
11
22
2222
444444 8
88888
4
6 666
9
6 96
9
1
11
11 1
77
77 77
1010
10
33 3
33
3
2019
20
19
20
19
-0.02
0.02
15 15
17 15
17 17
-0.04
-0.06
12
12
12
Accidental Patterns on Footwear
• Shoe prints contain marks and
patterns due to various
circumstances that can be
used to distinguish one shoe
print from another.
• How reliable are the
accidental patterns for
identifying particular shoes?
Facial Recognition Approach to
Accidental Pattern Identification
5
y axis
6
0
66
6
6
6
8 6X 8
8
8
8
8
8
-5
X
4
2
4
4
X
0
4 4
4
4
z axis
4
X
9
99
9
9
4
-2
4
-4
3D PCA
59.7% of variance
-6
-7.5
-5
-2.5
x axis
0
9
66688
4888
277
6896
11
9 55
31
73
X775
1
7
77 5
7 3X
X3
777
33 33
3 X55
5
2
1
2 X
5
5
11
55
11
2
22
2
2
Tool marks
• Like shoes, tools can leave marks
which can be used in identification
• Class characteristics
• Subclass characteristics
• Individual characteristics
Standard Striation Patterns
Made with ¼’’ Slotted Screwdriver
Measure lines and grooves
with ImageJ
Translate ImageJ data to a
feature vector that can be processed
LEA Striations
A, 2, #2
Bromberg, Lucky
C, 8, #4
Bromberg, Lucky
Questioned Documents: Photocopier Identification
• Mordente, Gestring, Tytell
3000
2500
2000
1500
1000
500
3000
2500
2000
1500
1000
500
0500
1000
1500
2000
2500
3000
2500
2000
1500
1000
500
0500
1000
1500
2000
2500
3000
2500
2000
1500
1000
500
0500
1000
1500
2000
2500
3000
2500
2000
1500
1000
500
3000
2500
2000
1500
1000
500
0500
1000
1500
2000
2500
3000
2500
2000
1500
1000
500
0500
1000
1500
2000
2500
3000
2500
2000
1500
1000
500
0500
1000
1500
2000
2500
3000
2500
2000
1500
1000
500
0500
1000
1500
2000
2500
Photocopy of a
blank sheet of paper
3000
2500
2000
1500
1000
500
0500
1000
1500
2000
2500
0500
1000
1500
2000
2500
0500
1000
1500
2000
2500
Dust: Where does it come from?
Our Environments!
Any matter or substance:
• both natural and synthetic
• reduces into minute bits, pieces, smears, and residues
• encountered as trace aggregates
Evidence!
N. Petraco
Where can you find it?
House
Work
Everywhere
Outdoors
Vehicle
N. Petraco
Analyze Results
3D PCA-Clustering can show potential for discrimination
Bayes Net for Dust in Authentication Case
References
• Bolton-King, Evans, Smith, Painter, Allsop, Cranton. AFTE J 42(1),23 2010
• Artigas. In: Optical Measurement of Surface Topography. Leach ed. Springer, 201l
• Helmli. In: Optical Measurement of Surface Topography. Leach ed. Springer, 2011
• deGroot. In: Optical Measurement of Surface Topography. Leach ed. Springer, 201l
• Efron, B. (2010). Large-Scale Inference: Empirical Bayes Methods for Estimation,
Testing, and Prediction. New York: Cambridge University Press.
• Gambino C., McLaughlin P., Kuo L., Kammerman F., Shenkin S., Diaczuk P.,
Petraco N., Hamby J. and Petraco N.D.K., “Forensic Surface Metrology: Tool Mark
Evidence", Scanning 27(1-3), 1-7 (2011).
• JAGS “A program for analysis of Bayesian hierarchical models using Markov Chain
Monte Carlo simulation”, Version 3.3.0. http://mcmc-jags.sourceforge.net/
• Kall L., Storey J. D., MacCross M. J. and Noble W. S. (2008). Posterior error
probabilities and false discovery rates: two sides of the same coin. J Proteome
Research, 7(1), 40-44.
References
•
locfdr R package. 2011. locfdr: “Computation of local false discovery rates”, Version 1.17.http://cran.r-project.org/web/packages/locfdr/index.html
•
Moran B., "A Report on the AFTE Theory of Identification and Range of Conclusions for Tool Mark
Identification and Resulting Approaches To Casework," AFTE Journal, Vol. 34, No. 2, 2002, pp. 227-35.
Petracoa N. D. K., Chan H., De Forest P. R., Diaczuk P., Gambino C., Hamby J., Kammerman F., Kammrath B.
W., Kubic T. A., Kuo L., Mc Laughlin P., Petillo G., Petraco N., Phelps E., Pizzola P. A., Purcell D. K. and
Shenkin P. “Final Report: Application of Machine Learning to Toolmarks: Statistically Based Methods for
Impression Pattern Comparisons”. National Institute of Justice, Grant Report: 2009-DN-BX-K041; 2012.
Petraco N. D. K., Kuo L., Chan H., Phelps E., Gambino C., McLaughlin P., Kammerman F., Diaczuk P.,
Shenkin P., Petraco N. and Hamby J. “Estimates of Striation Pattern Identification Error Rates by
Algorithmic Methods”, AFTE J., In Press, 2013.
Petraco N. D. K., Zoon P., Baiker M., Kammerman F., Gambino C. “Stochastic and Deterministic Striation
Pattern Simulation”. In preparation 2013.
Platt J. C. “Probabilities for SV Machines”. In: Advances in Large Margin Classifiers Eds: Smola A. J.,
Bartlett P., Scholkopf B., and Schuurmans D. MIT Press, 2000.
Plummer M. “JAGS: A Program for Analysis of Bayesian Graphical Models Using Gibbs Sampling”,
Proceedings of the 3rd International Workshop on Distributed Statistical Computing (DSC 2003), March 20–
22, Vienna, Austria.
Stan Development Team. 2013. “Stan: A C++ Library for Probability and Sampling”, Version 1.3. http://mcstan.org/
Storey J. D. and Tibshirani R. “Statistical significance for genome wide studies”. PNAS 2003;100(16):94409445.
Vovk V., Gammerman A., and Shafer G. (2005). Algorithmic learning in a random world. 1st ed. Springer,
New York.
•
•
•
•
•
•
•
•
References
20. Tippett CF, Emerson VJ, Fereday MJ, Lawton F, Richardson A, Jones LT, Lampert SM., “The
Evidential Value of the Comparison of Paint Flakes from Sources other than Vehicles”, J Forensic Soc
Soc 1968;8(2-3):61-65.
21. Ramos D, Gonzalez-Rodriguez J, Zadora G, Aitken C. “Information-Theoretical Assessment of the
Performance of Likelihood Ratio Computation Methods”, J Forensic Sci 2013;58(6):1503-1518.
Acknowledgements
• Professor Chris Saunders (SDSU)
• Professor Christoph Champod (Lausanne)
• Alan Zheng (NIST)
• Research Team:
• Dr. Martin Baiker
•
• Ms. Helen Chan
•
• Ms. Julie Cohen
•
• Mr. Peter Diaczuk
•
• Dr. Peter De Forest
•
• Mr. Antonio Del Valle
•
• Ms. Carol Gambino
•
•
• Dr. James Hamby
• Ms. Alison Hartwell, Esq. •
• Dr. Thomas Kubic, Esq. •
Ms. Loretta Kuo
•
Ms. Frani Kammerman •
Dr. Brooke Kammrath •
Mr. Chris Lucky
•
Off. Patrick McLaughlin
•
Dr. Linton Mohammed
•
Mr. John Murdock
•
Mr. Nicholas Petraco
•
Dr. Dale Purcel
•
Ms. Stephanie Pollut
Dr. Peter Pizzola
Dr. Graham Rankin
Dr. Jacqueline Speir
Dr. Peter Shenkin
Mr. Chris Singh
Mr. Peter Tytell
Mr. Todd Weller
Ms. Elizabeth Willie
Dr. Peter Zoon
Website: Data, codes, reprints and preprints:
toolmarkstatistics.no-ip.org/
[email protected]