Transcript Accuracy and Effectiveness of a Product Quality Classifier

Presentation Outline









Introduction
Objective
Sorting Probability and Loss Matrices
The Proposed Model
Analysis Of Some Loss Functions
Case Study
Redundancy based methods
Illustrative example
Summary
1
Introduction
Accuracy and Precision
B. Types of data
C. Binary Situation
A.
2
A. Accuracy and Precision.
Accuracy
The closeness of agreement between the
result of measurement and the true
(reference) value of the product being sorted.
Precision
Estimate of both the variation in repeated
measurements obtained under the same
conditions (Repeatability) and the variation
of repeated measurements obtained under
different conditions (Reproducibility).
3
B. Two types of data characterizing
products or processes
Variables
(results of measurement,
Interval or Ratio Scales)
Attributes
(results of testing,
Nominal or Ordinal Scales ).
4
Four types of data
 The four levels were proposed by
Stanley Smith Stevens in his 1946
article.Different mathematical
operations on variables are
possible, depending on the level at
which a variable is measured.
5
Categorical Variables
1. Nominal scale:
 gender,
 race,
 religious affiliation,
 the number of a bus.
Possible operations:
, 
6
Categorical Variables(cont.)
2. Ordinal scale :









results of internet page rank,
alphabetic order,
Mohs hardness scale (10 levels from talc to diamond)
customer satisfaction grade ,
quality sort,
customer importance (QFD)
vendor’s priority,
severity of failure or RPN (FMECA),
the power of linkage (QFD)
Possible operations:
, , , 
7
Numerical data.
3. Interval scale:
 temperature in Celsius or Fahrenheit scale ,
 object coordinate,
 electric potential.
Possible operations:
, , , ,,
8
Numerical data (cont.)




4. Ratio scale:
most physical quantities, such as mass, or
energy,
temperature, when it is measured in kelvins,
amount of children in family,
age.
Possible operations:
, , , ,,,, /
9
C. The sorting probability matrix
for the binary situation
Accuracy is characterized by:
1. Type I Errors
(non-defective is
reported as defective) –
alfa risk
2.
Type II Errors
(defective is reported as
non-defective) – beta
risk
10
The sorting probability matrix
for the binary situation (cont.)
Actual
Factual
+
-
+
-
1-α
α
β
1- β
11
E. Bashkansky, S. Dror, R.
Ravid, P. Grabov
ICPR-18, Salerno
August, 2, Session 19
6
Objective
Developing a new statistical procedure
for evaluating the accuracy and
effectiveness of measurement systems
applicable to Attribute Data based on
the Taguchi approach.
12
Sorting Probability Matrix
sorting matrix P̂ is an 'm by m'
matrix.
 Its components Pi,j are the conditional
probabilities that an item will be
classified as quality level j, given its
quality level is i.
m
Pij  1, 1  i  m
 A stochastic matrix:
 The

j 1
13
Four Interesting Sorting Matrices

(a) The most exact
sorting:

(b) The uniform
sorting:
(designated as
MDS: most
disordered sorting):

(c) The “worst
case” sorting. For
example, if m = 4:
Pij   ij
1
Pij 
m
0

0
P
1

1

0 0 1

0 0 1
0 0 0

0 0 0 
14
Four Interesting Sorting Matrices (cont.)
(d) Absence of any sorting .For example, if m = 2:
 1 0 

P  
1 0 
15
Indicator of the classification system
inexactness
m
G
Dˆ
2
2m

m
 ( P  
i 1 j 1
ij
ij
)
2
2m
16
Loss Matrix Definition
Let Lij - be the loss
incurred by classifying
sort i as sort j.
17
The Proposed Model
Expected Loss Definition:
m
EL 
m
 p  P  L
i
ij
ij
i 1 j 1
Effectiveness Measure:
EL
Eff  1 
EL( for the MDS )
18
Analysis Of Some Loss Functions
 Equal loss:
Lij  1   ij
 Quadratic loss: Lij  ( j  i)2
 Entropy loss: L   log P
ij
2 ij
 Linear loss: Lij  ( j  i)
19
Equal loss
If there is no preliminary information about pi
Trace( Pˆ )
EL  1 
m
Trace( Pˆ )
1
m
Eff  1 
m 1
m
20
Equal loss (Cont.)
If any preliminary information about pi
is available :
m
i
m
EL 

i 1
 p  (1  P )
p i  (1  Pii )
Eff  1 
ii
i 1
m 1
m
21
Quadratic Loss
For ordinal data, the total accuracy of the rating
could be defined as the expected value of Lij.
m
Accuracy 
m
( j  i)
2
 pi  Pij
i 1 j 1
22
Quadratic Loss (Cont.)
If there is no preliminary information about pi:
ELmds 
1
m
2
m
m

i 1 j 1
m 1
( j  i) 
6
2
2
If there is any preliminary information about pi:
ELmds
(m  1)(2m  1)
 E ( X )  (m  1) E ( X ) 
6
2
23
Entropy loss
If level i is systematically related to level j
(Pij=1), there is no “entropy loss”.
m
EL  
m
 p
i
 Pij  log 2 Pij
i 1 j 1
The above loss function leads us directly to
Eff = Theil’s uncertainty index.
24
Linear Loss
Could be useful for bias evaluation:
m
Bias 
m

pi  Pij  ( j  i)
i 1 j 1
This measure characterizes the dominant
tendency [over grading, if Bias > 0 or under
grading, if Bias < 0 ]
25
Case Study (nectarines sorting)
Classification
Type1 Type2 Type3 Total
91
7
544
Act Type1 446
ual Type2
12
307
33
352
0
11
93
104
Type3
458
409
133
1000
Total
Type 1- 0.860, Type 2 - 0.098 , Type 3 -0.042
26
The Classification Matrix
P
ij
 446

 544

 12

 352

 0

 104
91
544
307
352
11
104
7 
  0.820
544  
 


33
   0.034
352  
 


93
  0
104 
0.167
0.872
0.106
0.013



0.094



0.894
27
The Loss matrix
 0



L   5.42
ij 


12.42
0.98
0
7.00
1.76



0.78



0 
28
Effectiveness Evaluations
According to proposed approach: Eff = 80%
Compare the above to the measures of effectiveness
for different loss functions :





Equal loss: 74%.
Taguchi loss: 87%.
Disorder entropy loss: 62%
Linear penalty function:79%
Traditional kappa measure : 79% .
It can be seen that the effectiveness estimate
strongly depends on the loss metric model.
29
REPEATED SORTING
x1  x2  ...  xn
x
n
x 
x
n
30
Independency vs. Correlation
 product
2

 product   measurement
2
2
31
Case A: Two Independent Repeated
Ratings

Real improvement may be obtained if, in the
case of disagreement, the final decision is made in
favor of the inferior sort (one rater can see a defect,
which the other has not detected).

Usually, the loss that results from overrating is
greater than the loss due to underrating, thus such a
redistribution of probabilities seems to be legitimate.
Nevertheless, to verify improvement in sorting
effectiveness, we need a new expected loss
calculation.
32
Case B: Three Repeated Ratings
We add a third rater only if the first two
raters do not agree. For most industrial
applications this means that a product is
passed through a scrupulous laboratory
inspection or, for the purpose of our analysis,
through an MRB board. The decision could
be considered as an etalon measurement.
The probability of correct decisions increases,
and the probability of wrong decisions,
decreases.
33
Conclusions
To decide whether a double or triple
rating procedure is expedient the
total expenditures
have to be compared
34
Case C: A Hierarchical Classification
System
 The classification procedure is built on more
than one level.
 To characterize such a hierarchical
classification system G can be utilized as a
“pure” indicator of the classification system’s
inexactness.
 Usually, the cost of classification (COC) has
an inverse relationship to the amount of G. In
contrast to the COC, the expected loss
usually decreases, as the exactness of the
judgment improves.
35
Optimization of HCS
If one decides to pass the hierarchical
classification subsystem from the lower
level (1) up to the K level, the total
expenditure can be optimized by
looking for the best level minimizing it .
36
A CASE STUDY AND
ILLUSTRATIVE EXAMPLE
The proportions of the sort types were:
Type 1- 0.53, Type 2 - 0.27 and Type 3 0.20. The same loss matrix was
considered. From an R&R study,
executed in relation to two independent
raters’ results, the joint probability
matrices were estimated.
37
Summary table
One rater
Two
independent
raters
Three raters
(case B)
( case A )
0.89, 0.08, 0.03
0.07, 0.85, 0.08
0.06, 0.14, 0.80
0.84, 0.10, 0.06
0.04, 0.80, 0.16
0.01, 0.06, 0.93
0.96, 0.04, 0.00
0.04, 0.96, 0.00
0.01, 0.92, 0.07
EL = 0.534
EL' = 0.309
EL'' = 0.132
Inexact. Ind. G = 0.0194
G' = 0.0192
38
G'' = 0.0013
Sorting
matrix
Exp. loss
Summary
 The proposed procedure for
evaluation of product quality
classifiers takes into account some
a priori knowledge about the
incoming product, errors of sorting
and losses due to under/over
graduation.
39
Summary (Cont.1)
 The appropriate choice of the loss function
(matrix) provides the opportunity to fit quality
sorting process model to the real situation.
 The effectiveness of quality classifying can be
improved by different redundancy based
methods. However, the advantages of
redundancy based methods are not
unequivocal, as is the case in the usual
measurement processes, and corresponding
calculations according to the technique being
used are required.
40
Summary (Cont.2)
 The conclusion concerning the selection of
the preferred case depends on the losses
due to misclassification, as well as on the
incoming quality sort distribution.
 Possible applications of the proposed
approach are not limited only to quality
sorting. The approach can be extended to
other QA processes concerned with
classification
41
Thank You
42