Transcript Document

Remarks on uncertainty calculations for key comparisons
with a few examples from CCEM key comparisons
Thomas J. Witt, Bureau International des Poids et Mesures (BIPM)
TJW EMCP 01
1
Purpose:
• Call attention to some common errors seen in the statistical analysis of
CCEM key comparison results
• Bad analysis slows the processing of key comparisons and can reduce the
credibility of the key comparison scheme
Intended for:
• Authors of key comparison reports
• Participants in key comparisons
• Reviewers of CCEM key comparison reports
TJW EMCP 01
2
Outline
• Motivation: Appendix B of the MRA; degrees of equivalence with respect to the
KCRV and between pairs of participants
• Underlying concept; covariance
• Application to key comparisons, KCRV is weighted mean
A.
general case with correlations among results
B.
Mutually independent results
C.
example CCEM-K4 (10 pF)
• KCRV is weighted mean with equal weights
• KCRV is unweighted mean with unequal variances
• Examples of treatment of correlations
A.
QHE-derived results in CCEM-K4
B.
Correlations via PTB calibrations in CCEM-K6a (ac/dc)
TJW EMCP 01
3
Degrees of equivalence
with respect to the KCRV
TJW EMCP 01
Pair wise degrees of
equivalence
4
Underlying concept: covariance
General formulas for covariance:
The operator E is the expectation operator; e.g.,
E ( x)  
cov( x, y)  E{[ x  E ( x)][ y  E ( y)]}
 E{[ y  E ( y )][ x  E ( x)]}
 cov( y, x)
cov( x, x)  E{[ x  E ( x)][ x  E ( x)]}
 var( x)
(1)
(2)
If a, b and c are constants and x, y and z are variables:
cov(ax  by  c, z )  a cov(x, z )  b cov( y, z )
TJW EMCP 01
(3)
5
var( y  z )  cov( y  z, y  z )
 cov( y, y  z )  cov( z, y  z )
 cov( y, y )  cov( y, z )  cov( z, y )  cov( z, z )
 var( y )  2cov( y, z )  var( z )
 var( y )  var( z )  2cov( y, z )
(4)
In GUM notation and from (13) of GUM section 5.2.2, if q = y - z = f
2
1
 f  2
u (q )     u ( xi )  2
i 1  xi 
i 1
2
2
c
f f
u ( xi , x j )

j i 1 xi x j
2
 u 2 ( y )  u 2 ( z )  2(1)(1)u ( y, z )
TJW EMCP 01
(5)
6
Applications to key comparisons; key comparison reference value (KCRV) is
the weighted mean
• Weights, gi, are proportional to the reciprocal of the variance and normalized.
• the experimental standard deviation is denoted by s
A.
General case: mutual correlations assumed
n
1

2
i 1 si
 1,
A
n
A
1
x

2 i
n
s
i 1 i
xw  n
  gi xi ,
1
i 1

2
i 1 si
TJW EMCP 01
n
i 1
1
 1 2
2
sw
si
(6)
This relation will be
described shortly.
1
1
si2 si2
gi 
 .
1
A
sw2
(7)
7
Illustration: Assume only two participants in a key comparison and
that their results are correlated
2
xw   gi xi  g1 x1  g 2 x2
i 1
var( xw )  var( g1 x1 )  var( g 2 x2 )  2 cov( g1 x1 , g 2 x2 )
 g12 var( x1 )  g 22 var( x2 )  2 g1 g 2 cov( x1 , x2 )
Illustration: for three participants in a key comparison and for correlated results
xw  g1 x1  g 2 x2  g3 x3
var( xw )  var[( g1 x1  g 2 x2 )  g3 x3 ]
 g12 var( x1 )  g 22 var( x2 )  2 g1 g 2 cov( x1 , x2 )
 g32 var( x3 )  2 g1 g3 cov( x1 , x3 )  2 g 2 g3 cov( x2 , x3 )
3
2
3
  g var( xi )  2 g j g k cov( x j , xk )
i 1
TJW EMCP 01
2
i
j 1 k  j
8
In general, for n participants in a key comparison with mutually
correlated results:
n
xw   gi xi
i 1
n 1 n
n
var( xw )   g var( xi )  2 g j g k cov( x j , xk )
i 1
TJW EMCP 01
2
i
j 1 k  j
Variance of the
weighted mean
of n mutually
correlated results
(8)
9
Applications to key comparisons (KCRV is weighted mean)
B. Next assume mutually independent results; i.e. cov(xi,xj)=0 for all i, j
2
 1/ si2 
2
2
2
var( xw )  sw   gi var( xi )   
  si
i 1
i 1  A 
n
n
n

1/ s
2
i
i 1
A2
var( xw ) 
TJW EMCP 01

1
n
1

2
i 1 si
A
1

A

A2
1
n
1

2
i 1 si
Variance of the weighted mean of
n mutually independent results
(9)
10
C.
Uncertainty in the degree of equivalence between the value, xi, of a participant
whose (independent) result contributed to KCRV and the KCRV itself.
KCRV is weighted mean, xw of the results from all participants having
independent reference standards.
Since laboratory i contributes to the KCRV, its value is correlated with KCRV.
Example: key comparison CCEM-K4 (10 pF capacitance).
var( xi  xw )  var( xi )  var( xw )  2cov( xi , xw )
n
n
j 1
j 1
j i
cov( xi , xw )  cov( xi ,  g j x j )  cov( xi , gi xi   g j x j ).
(10)
Since xi and xj are uncorrelated, cov(xi,xi) = 0 if i j . Then
1
si2
cov( xi , xw )  gi cov( xi , xi )  gi var( xi ) 
 si2  sw2  var( xw )
1
sw2
TJW EMCP 01
(11)
11
Variance of difference from weighted
var( xi  xw )  var( xi )  var( xw ). mean, contributor to weighted mean,
mutually independent results
(12)
Discussion:
• Simple, easy to remember
• cannot directly generate the variance of the pair wise degrees of equivalence
between two participants, var(xi- xj), from the variances of the degrees of
equivalence with respect to the KCRV.
• May be some contestation if one participant’s uncertainty is small enough to
dominate the KCRV. In that case possible solutions are:
(1) use that participant’s value to define the KCRV;
(2) set a “state of the art” uncertainty value defining the minimum
acceptable uncertainty.
• It is not always true that results from all participants in a key
comparison contribute to the KCRV (e.g., CCEM-K4)
TJW EMCP 01
12
• KCRV is weighted mean with equal weights:
For mutually independent results and assuming equal weights for all participants,
the mean is x , si = s for all i and (12) yields:
s2
var( xi  x )  var( xi )  var( x )  s 
n
2
Equal weights, mutual
independence.
(13)
since for equal weights:
n
1
1 1 n
n


1

.


sw2 j 1 s 2j s 2 j 1 s 2
TJW EMCP 01
13
• KCRV is unweighted mean with unequal weights:
It could possibly be decided to use an unweighted mean as the KCRV but to
calculate its variance with unequal weights.
x  ( x1  x2  ...  xn ) / n.
(14)
In general
var( y  z)  var( y)  var( z)  2cov( y, z)
(15)
and successive applications give:
var( x ) 
1
n
2
n
n 1 n
 var( xi )  n2  cov( x j , xk ).
i 1
2
(16)
j 1 k  j
If cov( x j , xk )  0 for j  k
var( x ) 
n
var( xi )
2 
n
1
i 1
TJW EMCP 01
Unweighted mean, unequal
weights, mutual independence.
(17)
14
Expressed as standard deviations, for an unweighted mean with unequal weights
(i.e., unequal standard deviations) and mutually independent results
s( x )  [s2 ( x1 )  s2 ( x2 )  ...  s2 ( xn )] 1/ 2 / n.
(18)
Note factor in denominator is n, not n1/2 !
A common error is to confuse the above expression with the familiar
expression for the standard deviation of the mean of n independent, identically
distributed observations for which s( x1 )  s( x2 )  ...  s( xn )  s . In that case
s( x )  [s2  s2  ...  s2 ] 1/ 2 / n  s / n1/ 2 .
TJW EMCP 01
(19)
15
Continuing with the case of an unweighted mean with unequal weights, if the
x
result from participant i contributes to the KCRV, , then
var( xi  x )  var( xi )  var( x )  2cov( xi , x ).
and
1 n
cov( xi , x )  cov( xi ,  x j ).
n j 1
(20)
The results from all participants are assumed to be mutually
independent so that
cov( xi , x j )  0
and
TJW EMCP 01
cov( xi , x ) 
1
1
cov( xi , xi )  var( xi )
n
n
(21)
16
so that
2
var( xi  x )  (1  ) var( xi )  var( x ),
n
if cov( xi , x j )  0 for i  j.
(22)
It was just shown that
var( x ) 
1
n
2
n
 var( xi ),
if cov( x j , xk )  0 for j  k .
i 1
(17)
so that, finally,
2
1 n
var( xi  x )  (1  ) var( xi )  2  var( xi ).
n
n i 1
Unweighted mean, unequal
weights, mutual independence.
(23)
For example, for n = 3, this gives
2
1 3
4
1
1
var( x1  x )  (1  ) var( x1 )   var( xi )  var( x1 )  var( x2 )  var( x3 ).
3
9 i 1
9
9
9
Check:
TJW EMCP 01
2
1
1
x1  x2  x3
3
3
3
4
1
1
var( x1  x )  var( x1 )  var( x2 )  var( x3 ).
9
9
9
x1  x  x1  [( x1  x2  x2 ) / 3] 
17
Examples of treatment of correlated results:
CCEM-K4 (10 pF); defined from weighted mean of participants having independent
link to calculable capacitor. BIPM, NPL and BNM had links through the QHR
combined with CODATA value of RK-90 in terms of the ohm, derived from the link
between the ohm and the farad.
• In calculating var( xi  xw ) for these participants, the uncertainty, u(RK-90) is included.
• When calculating var(xi-xj), the variance of the pair wise degree of equivalence for
any pair of these three participants, the uncertainty u(RK-90) is “removed from the
uncertainty budgets” of both i and j as is shown formally by
var( xi  x j )  var( xi )  var( x j )  2cov( xi , x j )
 var( xi )  var( x j )  2  r  u 2 ( RK 90 )
 var( xi )  var( x j )  2u 2 ( RK 90 )
Here the covariance is written in terms of the correlation coefficient, r (=1), and the
product of the standard deviations associated with the correlated terms.
TJW EMCP 01
18
Another example of correlated results:
Correlations are common in EUROMET comparisons because some participants may
have defined their reference standards via calibrations from a major NMI.
Example: CCEM-K6a (ac/dc difference). Consider uncertainty in pairwise degree of
equivalence between two such participants who list rather large type-B
uncertainties, ui(cal) and uj(cal) associated with the calibration of their standards at
the PTB.
• When calculating var(xi-xj), the variance of the pair wise degree of equivalence for
any pair of these participants, the effect of the correlation may be treated as follows:
var( xi  x j )  var( xi )  var( x j )  2cov( xi , x j )
 var( xi )  var( x j )  2ui (cal)u j (cal)
where the covariance is again written in terms of a correlation coefficient =1,
and the product of the standard deviations associated with the correlated
terms.
TJW EMCP 01
19
Example: CCEM-K6a (ac/dc difference)…continued
Accounting for such correlations in the analysis of CCEM-K6a was
controversial; some participants thought it excessively lowers pair-wise degrees
of equivalence. To resolve this issue the CCEM agreed to forego listing the
pairwise degrees of equivalence.
TJW EMCP 01
20
Conclusions and recommendations:
•
We are on a “learning curve” in the statistical analysis of key comparisons;
this slows down agreement of results for Appendix B but we’ll get better!
•
It is important to consider correlations, particularly in EUROMET and other
RMO comparisons.
•
In general, one cannot generate a table of uncertainties in pair wise
degrees of equivalence var(xi - xj) from the column of uncertainties with
respect to the KCRV, var(xi - xKCRV).
•
Care should be used in applying statistical expressions, particularly the
“standard deviation of the mean”.
•
In situations where uncertainty analysis seems to be intractable, consider
the possibility of making simplifying assumptions, provided, of course, that
they are stated in the report; the CCEM is flexible
TJW EMCP 01
21