Transcript EFFECTIVENESS OF A PRODUCT QUALITY CLASSIFIER

Effectiveness of a
Product Quality
Classifier
Dr. E. Bashkansky, Dr. S. Dror, Dr. R. Ravid
Industrial Eng. & Management, ORT Braude College, Karmiel, Israel
Dr. P. Grabov
A.L.D. Ltd., Beit Dagan 50200, Israel
2
3
“Effectiveness” - definition

Extent to which planned activities
are realized and planned results
achieved (ISO 9000:2000)

The state of having produced a
decided upon or desired effect (ASQ
Glossary, 2006).
4
Presentation Outline
 Objective
 Background
 Basic definitions
 Proposed approach to effectiveness evaluation
 Measures resulting from proposed approach
 The basic of repeated sorting
 Case A – two raters
 Case B – two raters + supervisor
 Illustrative example and conclusions
 Summary
5
Objective
Developing a new statistical
procedure for evaluating the
effectiveness of measurement systems
applicable to Attribute Quality Data
based on the Taguchi approach.
6
Objective
(cont.)
When the loss incurred by quality sort
misclassifying is large, an improvement
of the sorting procedures can be
achieved by the help of repeated
classifications. The way it influences the
classifying effectiveness is also
analyzed.
7
Accuracy and Precision.
Accuracy
The closeness of agreement between the
result of measurement and the true
(reference) value of the product being sorted.
Precision
Estimate of both the variation in repeated
measurements obtained under the same
conditions (Repeatability) and the variation
of repeated measurements obtained under
different conditions (Reproducibility).
8
The most popular precision metrics

Signal to Noise (S/N) ratio compares the variations among
products to variations in the MS

Precision to Tolerance (P/T) ratio compares the latter to tolerance.
9
Four types of quality data
 The four levels were proposed by
Stanley Smith Stevens in his 1946
article. Different mathematical
operations on variables are
possible, depending on the level at
which a variable is measured.
10
Categorical Variables
1. Nominal scale:
Supplier: A,B,C….
Possible operations:
, 
11
Categorical Variables(cont.)
2. Ordinal scale :






customer satisfaction grade ,
quality sort,
customer importance (QFD)
vendor’s priority,
severity of failure or RPN (FMECA),
the power of linkage (QFD)
Possible operations:
, , , 
12
Numeric data.
3. Interval scale:
Image Quality
Possible operations:
 ,  ,  ,  , ,
13
Numeric data (cont.)
4. Ratio scale:
 Amount of defectives in a batch
 Deviation from a specification
Possible operations:
 ,  ,  ,  ,  ,  , , /
14
Two types of data characterizing
product/process quality
Variables
(results of measurement,
Interval or Ratio Scales)
Attributes
(results of testing,
Nominal or Ordinal Scales ).
15
Traditional accuracy metrics for
binomial situation
Accuracy is characterized by:
1. Type I Errors
(non-defective is
reported as defective) –
alfa risk
2.
Type II Errors
(defective is reported as
non-defective) – beta
risk
16
The sorting probability matrix
for the binary situation
Actual
Factual
+
-
+
-
1-α
α
β
1- β
17
Case Study (nectarines sorting)
Classification
Type1 Type2 Type3 Total
91
7
544
Act Type1 446
ual Type2
12
307
33
352
0
11
93
104
Type3
458
409
133
1000
Total
Type 1- 0.860, Type 2 - 0.098 , Type 3 -0.042
18
Introduction
The proposed method takes into account
the information available about:
1.Incoming product quality sort distribution,
2. Sorting errors rates,
3. Losses due to misclassification,
4. Additional organizational charges.
19
Basic definitions:
1. Incoming and outgoing quality sort distributions
pi - the probability that an item whose
quality is to be classified has a quality
level i
qj - the probability that an outgoing item
was classified as belonging to quality
level j
20
Basic Definitions:
2. Sorting Probability Rates
Sorting matrix: Pˆ
The sorting matrix is an 'm by m' matrix.
Its components Pi,j are the conditional
probabilities that an item will be classified
as quality level j, given its quality level is i.
21
Sorts Distribution Transformation
m
qj 

p i  Pij
i 1
22
Binary sorting matrix examples
(1)
1
ˆ
P  
0
0

1
(2)
 0 .5
ˆ
P  
 0 .5
0 .5 

0 .5 
(3)
0
ˆ
P  
1
23
1

0
Cheating
(4) Absence of any sorting
 1
P  
1
0

0
24
Three Interesting Sorting Matrices

(a) The most exact
sorting:

(b) The uniform
sorting:
(designated as
MDS: most
disordered sorting):

(c) The “worst
case” sorting. For
example, if m = 4:
Pij   ij
Pij 
0

0
P 
1

1

1
m
0
0
0
0
0
0
0
0
1

1

0

0 
25
The Classification Matrix
Estimation
P
ij
 446

 544

 12

 352


0

 104
91
544
307
352
11
104
7 
  0 . 820
544  
 


33
   0 . 034
352  
 
 
93
  0
104 
0 . 167
0 . 872
0 . 106
0 . 013 



0 . 094 




0 . 894 
26
Indicator of the classification system
inexactness
G 
Dˆ
2

2m
m
m
i 1
j 1
  (P   )
ij
2
ij
2m
Det ( Dˆ )  0
27
Loss Matrix
Let Lij - be the loss
incurred by classifying
sort i as sort j.
28
Losses due to misclassification
Loss matrix (in NIS/kg) :
L
ij

 0

 5 . 42
 12 . 42

0 . 98
0
7 . 00
1 . 76 

0 . 78 

0 
29
Additional Organizational
Charges
A - the cost (per product unit)
concerned with one additional
rating.
B - all expenditures (per product unit)
concerned with the supervisor
control.
30
The Proposed Measures
Expected Loss Definition:
m
EL 
m

p i  Pij  L ij
i 1 j 1
Effectiveness Measure:
Eff  1 
EL
EL ( for the MDS )
For exact sorting Eff = 1, for MDS Eff = 0
31
Measures resulting from proposed approach when:
1. Only information about the sorting matrix is
available
m
Eff  H 
 Pii  1
i
1
m 1

Trace ( Pˆ )  1
m 1
• H equals 1 for the exact sorting
• H equals 0 for the random sorting
32
Measures resulting from proposed approach when:
2. Information about the sorting matrix and the
incoming quality is available
2.1 Uncertainty reduction measure
L ij   log
m
Eff  U  1 
2
m

i 1
Pij
j 1
p
 P  (  log
i
ij
2
P )
ij
m
( 
q
j 1
j
 log
2
q
j
)
• U equals 1 for the exact sorting
• U equals 0 for the random sorting
33
Measures resulting from proposed approach:
2. Information about the sorting matrix and the
incoming quality is available
2.2 Modified kappa measure
L
ij
 const
1  i, j  m
i j
m
Eff
 G 1
1  p P
k 1
1
k
kk
1
m
• G equals 1 for the most exact sorting
• G equals 0 for the most disordered sorting
34
Quadratic Loss (Cont.)
m
6
Eff  1 
m
 ( j  i)
2
 Pij
i 1 j 1
m  ( m  1)
2
35
REPEATED (REDUNDANT) SORTING
Independence vs. Correlation
36
REPEATED (REDUNDANT) SORTING
Case A: Two Independent (but correlated)
Repeated Ratings
Assumptions:
 ij
1.
- conditional joint probability of
sorting i by the first rater, and j by the
second, given the actual sort – k.
2. The same capabilities for both stages/raters,
(
k)
 ij
(
k)

(
k)
ji
3. In the case of disagreement, the final
decision is made in favor of the inferior
sort (one rater can see a defect, which
the other has not detected).
37
REPEATED (REDUNDANT) SORTING
Case A: Two Independent Repeated Ratings
Results:
1. Redistribution of probabilities:
The probability of making a true decision about low
quality products increases, while the probability of
making a true decision about high quality products
decreases.
2. To verify improvement in sorting effectiveness, we
need a new expenditure calculation:
m
m


EL  A    p i  Pij  L ij  A
i 1
j 1
( A is the cost (per product unit) concerned with the
additional rating ).
38
REPEATED (REDUNDANT) SORTING
Case B: Three Repeated Ratings
Assumptions:
1. The same capabilities for the first two
stages/raters,
2. A third rater is added only if the first two
raters do not agree.
3. His/her decision could be considered as an
etalon measurement.
39
REPEATED (REDUNDANT) SORTING
Case B: Three Repeated Ratings
Results :
1. The probability of correct decisions,
increases, and the probability of
wrong decisions, decreases.
2. The probability of having to carry out the
etalon measurement is important.
3. The total expenditure concerning the
triple procedure has to be calculated.
EL

 A BP
D
40
ILLUSTRATIVE EXAMPLE
Consider a sorting line that classifies fruits
into the three levels of quality:
1. High
2. Medium,
3. Unacceptable.
The proportions of the above types are:
Type 1 - 53 %,
Type 2 - 27 %,
Type 3 - 20 %.
41
Losses due to misclassification
Loss matrix (in NIS/kg) :
L
ij

 0

 5 . 42
 12 . 42

0 . 98
0
7 . 00
1 . 76 

0 . 78 

0 
42
The joint probability matrices for two
repeated ratings
k
(k )
ˆ
 ij
Kappa
coefficient of
agreement
between two
raters
1
 0 . 84

 0 . 03
 0 . 02

0 . 03
0 . 04
0 . 01
0.40
2
0 . 02 

0 . 01 
0 . 00 
 0 . 04

 0 . 02
 0 . 01

0 . 02
0 . 76
0 . 07
0.25
3
0 . 01 

0 . 07 
0 . 00 
 0 . 01

 0 . 02
 0 . 03

0 . 02
0 . 02
0 . 10
0 . 03 

0 . 10 
0 . 67 
0.11
-k
indicates the quality level of the product
- i represent the first rater’s decision and j the second rater’s
decision.
The probability of disagreement equals 0.1776:
43
Results of calculations
Case
One rater
Two raters
Three raters
H
0.77
0.785
0.945
U
0.528
0.523
0.801
0.774
0.757
0.937
0.792
0.771
0.943
Expected Loss
EL = 0.534
EL' = 0.309
EL'' = 0.132
Effectiveness
Eff = 77%
Eff ' = 87%
Eff '' = 94%
G
44
Conclusions Concerning Case A
 The probability of making a true decision
concerning low quality products increases while
the probability of making a true decision
concerning good quality products decreases.
 All measures that do not take into account the real
losses of misclassification (H, U, kappa, G ) do not
differ significantly.
 Applying the two raters method is expedient, if the
cost of additional rating does not exceed:
EL - EL' = 0.534 - 0.309 = 0.225 (NIS/kg).
45
Conclusions Concerning Case B
 Accuracy of classification is much better, if all items
on which there is no agreement are passed to the
supervisor. It reflects in improvement of all
metrological parameters.
 Nevertheless, to decide whether applying this method
is expedient or not, the cost of the third additional
rating multiplied by the probability that it will be
required should not exceed
EL' - EL'' = 0.309 – 0.132 = 0.177 (NIS/kg)
or ,in other words, this cost should not exceed one
NIS/kg.
46
Summary - 1
The proposed procedure for
evaluation of product quality
classifiers takes into account some
a priori knowledge about the
incoming product, errors of sorting
and losses due to under/over
graduation.
47
Summary - 2
It is shown that when the loss
function - the major component of
the proposed measure - is chosen
appropriately, we arrive at already
known measures for quality
classification as well as to some
new measures.
48
Summary - 3
The appropriate choice of
the loss function matrix
provides the opportunity to fit
quality sorting process model
to the real situation.
49
Summary - 4
Repeated sorting procedures
could be expedient for
cases when the loss incurred
by quality sort misclassifying
is large.
50
Summary - 5
Nevertheless, to decide whether
applying this method is expedient
or not, the expected cost of the
additional rating/s should be
compared to the expected loss
resulting from misclassification.
51
Another possible applications
The approach can be extended to other QA
processes concerned with classification , for
example :

vendor's evaluation,

customer satisfaction survey,

FMECA analysis

quality estimation of multistage or
hierarchical
service systems
 etc…
52
Thank You
53