Transcript P13_MEASUREMENT-UNCERTAINTIES-FOR-ALI

Measurement
Uncertainties and
Inconsistencies
Dr. Richard Young
Optronic Laboratories, Inc.
Optronic Laboratories, Inc.
Introduction
The concept of accuracy is generally
understood.
“…an accuracy of 1%.”
 What does this mean?
• 99% inaccurate?
Optronic Laboratories, Inc.
Introduction
The confusion between the concept
and the numbers has lead national
laboratories to abandon the term
accuracy.
 Except in qualitative terms e.g. high
accuracy.
The term now used is uncertainty.
 “…an uncertainty of 1%.”
Optronic Laboratories, Inc.
Introduction
Sometimes…
 Users do not know the uncertainty of
their results.
 They interpret any variations as
inconsistencies.
Optronic Laboratories, Inc.
Uncertainty vs. Inconsistency
Laboratories give different values,
but the difference is within their
combined uncertainties…
 Pure chance.
Laboratories give different values,
and the difference is outside their
combined uncertainties…
 Inconsistency.
Optronic Laboratories, Inc.
What is uncertainty?
“…an uncertainty of 1%.”
 But is 1% the maximum, average or
typical variation users can expect?
Uncertainty is a statistical quantity
based on the average and standard
deviation of data.
Optronic Laboratories, Inc.
Statistics
“There are three types of lies:
lies, damned lies and statistics.”
-attributed to Benjamin Disraeli
“The difference between statistics and
experience is time.”
-Richard Young
Statistics uses past experience to predict
likely future events.
Optronic Laboratories, Inc.
Statistics
We toss a coin:
 It is equally likely to be heads or tails.
We toss two coins at the same time:
 There are 4 possible outcomes:
•
•
•
•
Head + Head
Head + Tail
Tail + Head
Tail + Tail
These 2 are the same
and hence twice as
likely to happen as the
others.
Optronic Laboratories, Inc.
Statistics
300
250
Number of Occurrences
 Now let us throw
10 coins.
 There are 1024
possibilities (210).
 What if we threw
them 1024 times,
and counted each
time a certain
number of heads
resulted…
200
150
100
50
0
0
1
2
3
4
5
6
7
8
9
10
Number of Heads
Optronic Laboratories, Inc.
Statistics
 We get probability.
300
250
Number of Occurrences
Although the
outcome of each
toss is random…
...not every result
is equally likely.
If we divide the
number of
occurrences by the
total number of
throws…
200
150
100
50
0
0
1
2
3
4
5
6
7
8
9
10
Number of Heads
Optronic Laboratories, Inc.
Statistics
 0 = never happens
 1 = always
happens
0.3
0.25
Probability of Occurrence
Here is the same
plot, but shown as
probability.
Probability is just a
number that
describes the
likelihood between:
0.2
0.15
0.1
0.05
0
0
1
2
3
4
5
6
7
8
9
10
Number of Heads
Optronic Laboratories, Inc.
Statistics
 Shown in red
It uses just 2
values:
 The average
 The standard
deviation
0.3
0.25
Probability of Occurrence
Gauss described a
formula that
predicted the
shape of any
distribution of
random events.
0.2
0.15
0.1
0.05
0
0
1
2
3
4
5
6
7
8
9
10
Number of Heads
Optronic Laboratories, Inc.
Statistics
 Now throw 100 coins…
0.09
0.08
Probability of Occurence
0.07
0.06
0.05
The
Gaussian
curve fits
exactly.
0.04
0.03
0.02
We have an averageAnd a standard
= 50
deviation = 5
And the familiar
bell-shaped
distribution.
0.01
0
0
10
20
30
40
50
Number of Heads
60
70
80
90
100
Optronic Laboratories, Inc.
Confidence
 Now throw 100 coins…
0.09
Since the total
probability must =1,
the standard
deviation marks off
certain
probabilities.
0.08
Probability of Occurence
0.07
0.06
0.05
0.04
0.03
0.02
0.01
0
0
10
20
30
40
50
Number of Heads
60
70
80
90
100
Optronic Laboratories, Inc.
Confidence
 Now throw 100 coins…
0.09
Since the total
probability must =1,
the standard
deviation marks off
certain
probabilities.
0.08
Probability of Occurence
0.07
0.06
0.05
0.04
0.03
About 67% of
all results lie
within  1
standard
deviation.
“I am 67% confident that a
new throw will give
between 45 and 55 heads.”
0.02
0.01
0
0
10
20
30
40
50
Number of Heads
60
70
80
90
100
Optronic Laboratories, Inc.
Confidence
 Now throw 100 coins…
0.09
Since the total
probability must =1,
the standard
deviation marks off
certain
probabilities.
0.08
Probability of Occurence
0.07
0.06
0.05
0.04
0.03
About 95% of
all results lie
within  2
standard
deviations.
“I am 95% confident that a
new throw will give
between 40 and 60 heads.”
0.02
0.01
0
0
10
20
30
40
50
Number of Heads
60
70
80
90
100
Optronic Laboratories, Inc.
Real Data
Real data, such as the result of a
measurement, is also characterized
by an average and standard
deviation.
To determine these values, we must
make measurements.
Optronic Laboratories, Inc.
Real Data
 NVIS radiance measurements are unusual.
 The signal levels at longer wavelengths can
be very low – close to the dark level of the
system.
 The signal levels at longer wavelengths
dominate the NVIS radiance result.
 The uncertainty in results close to the dark
level can be dominated by PMT noise.
 Therefore: Variations in NVIS results can
be dominated by PMT noise.
Optronic Laboratories, Inc.
Real Data
The net signal from the PMT is used
to calculate the spectral radiance.
Dark current, which is subtracted
from each current reading during a
scan, contains PMT noise.
Scans at low signals contain PMT
noise.
Optronic Laboratories, Inc.
Real Data
PMT noise present in each of these
current readings does not have the
same effect on results:
 A high or low dark reading will raise
or lower ALL points.
 Current readings during scans contain
highs and lows that cancel out to
some degree.
Optronic Laboratories, Inc.
Real Data
2.3E-12
2.2E-12
Dark Current [A]
2.1E-12
2E-12
1.9E-12
1.8E-12
Excel:
Excel:“=“=average()”
stdev()” 1E-13
2E-12
1.7E-12
0
20
40
60
80
100
120
140
160
180
200
Measurement #
Optronic Laboratories, Inc.
Real Data
6E-13
Dark = min
4E-13
Net signal [A]
2E-13
0
-2E-13
-4E-13
-6E-13
0
20
40
60
80
100
120
140
160
180
200
Measurement #
Optronic Laboratories, Inc.
Real Data
6E-13
Dark = min
Dark = max
4E-13
Net signal [A]
2E-13
0
-2E-13
-4E-13
-6E-13
0
20
40
60
80
100
120
140
160
180
200
Measurement #
Optronic Laboratories, Inc.
Real Data
Dark = min
Dark = max
Dark = average
6E-13
4E-13
Net signal [A]
2E-13
0
-2E-13
-4E-13
-6E-13
0
20
40
60
80
100
120
140
160
180
200
Measurement #
Optronic Laboratories, Inc.
Calculations
 We can describe the effects of noise on
class A NVIS radiance mathematically:
 ss is the standard deviation of the noise
 C(l) is the calibration factors
 GA(l) is the relative response of class A
NVIS Signal averaging
s NVISa = 1 + 930
1
Dark subtraction
ò G A (l ) × dl
930
* ò s s × C (l ) × G A (l ) × dl
450
450
Optronic Laboratories, Inc.
Calculations
A similar equation, but using NVIS
class B response instead of class A,
can give the standard deviation in
NVISb radiance.
The standard deviations should be
scaled to the luminance to give the
expected variations in scaled NVIS
radiance.
Optronic Laboratories, Inc.
Calculations
Noise can be reduced by multiple
measurements.
If we generalize the equation to
include multiple dark readings (ND)
and scans (S):
s NVISa =
930
æ
ö
ç N D + ò G A (l ) × dl ÷
ç
÷ 930
450
è
ø * s × C (l ) × G (l ) × dl
A
930
ò s
Brain overload
S × N D × ò G A (l ) × dl
450
450
Optronic Laboratories, Inc.
Spreadsheet
Moving on to the benefits…
Introducing
The Spreadsheet
Optronic Laboratories, Inc.