Transcript 6_1 - Department of Mathematics and Computer Science

6
Point Estimation
Copyright © Cengage Learning. All rights reserved.
6.1
Some General Concepts
of Point Estimation
Copyright © Cengage Learning. All rights reserved.
Some General Concepts of Point Estimation
Statistical inference is almost always directed toward
drawing some type of conclusion about one or more
parameters (population characteristics).
To do so requires that an investigator obtain sample data
from each of the populations under study.
Conclusions can then be based on the computed values of
various sample quantities.
For example, let  (a parameter) denote the true average
breaking strength of wire connections used in bonding
semiconductor wafers.
3
Some General Concepts of Point Estimation
A random sample of n = 10 connections might be made,
and the breaking strength of each one determined,
resulting in observed strengths x1, x2, . . . , x10.
The sample mean breaking strength x could then be used
to draw a conclusion about the value of .
Similarly, if  2 is the variance of the breaking strength
distribution (population variance, another parameter), the
value of the sample variance s2 can be used to infer
something about  2.
4
Some General Concepts of Point Estimation
When discussing general concepts and methods of
inference, it is convenient to have a generic symbol for the
parameter of interest.
We will use the Greek letter  for this purpose. The
objective of point estimation is to select a single number,
based on sample data, that represents a sensible value for
.
As an example, the parameter of interest might be , the
true average lifetime of batteries of a certain type.
5
Some General Concepts of Point Estimation
A random sample of n = 3 batteries might yield observed
lifetimes (hours) x1 = 5.0, x2 = 6.4, x3 = 5.9.
The computed value of the sample mean lifetime is
x = 5.77, and it is reasonable to regard 5.77 as a very
plausible value of  — our “best guess” for the value of 
based on the available sample information.
Suppose we want to estimate a parameter of a single
population (e.g.,  or  ) based on a random sample of size
n.
6
Some General Concepts of Point Estimation
We know that before data is available, the sample
observations must be considered random variables (rv’s)
X1, X2, . . . , Xn.
It follows that any function of the Xi ’s—that is, any
statistic—such as the sample mean X or sample standard
deviation S is also a random variable.
The same is true if available data consists of more than
one sample. For example, we can represent tensile
strengths of m type 1 specimens and n type 2 specimens
by X1, . . . , Xm and Y1, . . . , Yn, respectively.
7
Some General Concepts of Point Estimation
The difference between the two sample mean strengths is
X – Y, the natural statistic for making inferences about
1 – 2, the difference between the population mean
strengths.
Definition
8
Some General Concepts of Point Estimation
In the foregoing battery example, the estimator used to
obtain the point estimate of  was X, and the point estimate
of  was 5.77.
If the three observed lifetimes had instead been x1 = 5.6,
x2 = 4.5, and x3 = 6.1, use of the estimator X would have
resulted in the estimate x = (5.6 + 4.5 + 6.1)/3 = 5.40.
The symbol (“theta hat”) is customarily used to denote
both the estimator of  and the point estimate resulting from
a given sample.
9
Some General Concepts of Point Estimation
Thus = X is read as “the point estimator of  is the
sample mean X .” The statement “the point estimate of  is
5.77” can be written concisely as = 5.77 .
Notice that in writing = 72.5, there is no indication of how
this point estimate was obtained (what statistic was used).
It is recommended that both the estimator and the resulting
estimate be reported.
10
Example 6.2
Reconsider the accompanying 20 observations on
dielectric breakdown voltage for pieces of epoxy resin.
24.46
27.98
25.61
28.04
26.25
28.28
26.42
28.49
26.66 27.15 27.31 27.54 27.74
28.50 28.87 29.11 29.13 29.50
27.94
30.88
The pattern in the normal probability plot given there is
quite straight, so we now assume that the distribution of
breakdown voltage is normal with mean value .
Because normal distributions are symmetric,  is also the
median lifetime of the distribution.
11
Example 6.2
cont’d
The given observations are then assumed to be the result
of a random sample X1, X2, . . . , X20 from this normal
distribution.
Consider the following estimators and resulting estimates
for  :
a. Estimator = X, estimate = x = xi /n = 555.86/20 = 27.793
b. Estimator = , estimate =
= (27.94 + 27.98)/2 = 27.960
c. Estimator = [min(Xi) + max(Xi)]/2 = the average of the
two extreme lifetimes,
estimate = [min(xi) + max(xi)]/2 = (24.46 + 30.88)/2
= 27.670
12
Example 6.2
cont’d
d. Estimator = Xtr(10), the 10% trimmed mean (discard the
smallest and largest 10% of the sample and then
average),
estimator = xtr(10)
=
= 27.838
Each one of the estimators (a)–(d) uses a different
measure of the center of the sample to estimate . Which
of the estimates is closest to the true value?
13
Example 6.2
cont’d
We cannot answer this without knowing the true value.
A question that can be answered is, “Which estimator,
when used on other samples of Xi’s, will tend to produce
estimates closest to the true value?”
We will shortly consider this type of question.
14
Some General Concepts of Point Estimation
In the best of all possible worlds, we could find an estimator
for which =  always. However, is a function of the
sample Xi ’s, so it is a random variable.
For some samples, will yield a value larger than  ,
whereas for other samples will underestimate  . If we
write
=  + error of estimation
then an accurate estimator would be one resulting in small
estimation errors, so that estimated values will be near the
true value.
15
Some General Concepts of Point Estimation
A sensible way to quantify the idea of being close to  is
to consider the squared error (
)2. For some samples,
will be quite close to  and the resulting squared error will
be near 0.
Other samples may give values of far from ,
corresponding to very large squared errors.
An omnibus measure of accuracy is the expected or mean
square error MSE = E[(
)2]. If a first estimator has
smaller MSE than does a second, it is natural to say that
the first estimator is the better one.
16
Some General Concepts of Point Estimation
However, MSE will generally depend on the value of .
What often happens is that one estimator will have a
smaller MSE for some values of  and a larger MSE for
other values.
Finding an estimator with the smallest MSE is typically not
possible. One way out of this dilemma is to restrict attention
just to estimators that have some specified desirable
roperty and then find the best estimator in this restricted
group.
A popular property of this sort in the statistical community is
unbiasedness.
17
Unbiased Estimators
18
Unbiased Estimators
Suppose we have two measuring instruments; one
instrument has been accurately calibrated, but the other
systematically gives readings smaller than the true value
being measured.
When each instrument is used repeatedly on the same
object, because of measurement error, the observed
measurements will not be identical.
However, the measurements produced by the first
instrument will be distributed about the true value in such a
way that on average this instrument measures what it
purports to measure, so it is called an unbiased instrument.
19
Unbiased Estimators
The second instrument yields observations that have a
systematic error component or bias.
Definition
That is, is unbiased if its probability (i.e., sampling)
distribution is always “centered” at the true value of the
parameter.
20
Unbiased Estimators
Suppose is an unbiased estimator; then if  = 100, the
sampling distribution is centered at 100; if  = 27.5, then
the sampling distribution is centered at 27.5, and so on.
Figure 6.2 pictures the distributions of several biased and
unbiased estimators. Note that “centered” here means that
the expected value, not the median, of the distribution of
is equal to .
The pdf’s of a biased estimator
and an unbiased estimator
Figure 6.2
for a parameter 
21
Unbiased Estimators
It may seem as though it is necessary to know the value of
 (in which case estimation is unnecessary) to see whether
is unbiased.
This is not usually the case, though, because
unbiasedness is a general property of the estimator’s
sampling distribution—where it is centered—which is
typically not dependent on any particular parameter value.
In Example 6.1 the sample proportion X/n can be used as
an estimator of p, where X, the number of sample
successes, has a binomial distribution with parameters n
and p.
22
Unbiased Estimators
Thus
E( ) = E
=
E(X) =
(np) = p
Proposition
No matter what the true value of p is, the distribution of the
estimator will be centered at the true value.
23
Example 6.4
Suppose that X, the reaction time to a certain stimulus, has
a uniform distribution on the interval from 0 to an unknown
upper limit  (so the density function of X is rectangular in
shape with height 1/ for 0  x  ).
It is desired to estimate  on the basis of a random sample
X1, X2, . . . , Xn of reaction times.
Since  is the largest possible time in the entire population
of reaction times, consider as a first estimator the largest
sample reaction time: = max (X1 ,..., Xn).
24
Example 6.4
cont’d
If n = 5 and x1 = 4.2, x2 = 1.7, x3 = 2.4, x4 = 3.9, and
x5 = 1.3, the point estimate of  is
= max(4.2, 1.7, 2.4, 3.9, 1.3) = 4.2.
Unbiasedness implies that some samples will yield
estimates that exceed  and other samples will yield
estimates smaller than  —otherwise  could not possibly
be the center (balance point) of ’s distribution.
However, our proposed estimator will never overestimate 
(the largest sample value cannot exceed the largest
population value) and will underestimate  unless the
largest sample value equals .
25
Example 6.4
cont’d
This intuitive argument shows that is a biased estimator.
More precisely, it can be shown that
The bias of is given by n /(n + 1) – = – /(n + 1), which
approaches 0 as n gets large.
It is easy to modify to obtain an unbiased estimator of .
Consider the estimator
 max(X1 ,…, Xn)
26
Example 6.4
cont’d
Using this estimator on the data gives the estimate
(6/5)(4.2) = 5.04. The fact that (n + 1)/n > 1 implies that
will overestimate  for some samples and underestimate it
for others. The mean value of this estimator is
 max(X1 ,…, Xn)
If is used repeatedly on different samples to estimate  ,
some estimates will be too large and others will be too
small, but in the long run there will be no systematic
tendency to underestimate or overestimate .
27
Unbiased Estimators
According to this principle, the unbiased estimator in
Example 6.4 should be preferred to the biased estimator
Consider now the problem of estimating  2.
.
28
Unbiased Estimators
Proposition
The estimator that uses divisor n can be expressed as
(n – 1)S2/n, so
29
Unbiased Estimators
This estimator is therefore not unbiased. The bias is
(n – 1) 2/n –  2 = – 2 /n.
Because the bias is negative, the estimator with divisor n
tends to underestimate  2, and this is why the divisor n – 1
is preferred by many statisticians (though when n is large,
the bias is small and there is little difference between the
two).
Unfortunately, the fact that S2 is unbiased for estimating  2
does not imply that S is unbiased for estimating .
30
Unbiased Estimators
Taking the square root messes up the property of
unbiasedness (the expected value of the square root is not
the square root of the expected value).
Fortunately, the bias of S is small unless n is quite small.
There are other good reasons to use S as an estimator,
especially when the population distribution is normal.
31
Unbiased Estimators
In Example 6.2, we proposed several different estimators
for the mean  of a normal distribution.
If there were a unique unbiased estimator for  , the
estimation problem would be resolved by using that
estimator. Unfortunately, this is not the case.
Proposition
32
Unbiased Estimators
The fact that X is unbiased is just a restatement of one of
our rules of expected value: E(X ) =  for every possible
value of  (for discrete as well as continuous distributions).
The unbiasedness of the other estimators is more difficult
to verify.
The next example introduces another situation in which
there are several unbiased estimators for a particular
parameter.
33
Example 6.5
Under certain circumstances organic contaminants adhere
readily to wafer surfaces and cause deterioration in
semiconductor manufacturing devices.
The paper “Ceramic Chemical Filter for Removal of
Organic Contaminants” (J. of the Institute of Environmental
Sciences and Technology, 2003: 59–65) discussed a
recently developed alternative to conventional charcoal
filters for removing organic airborne molecular
contamination in cleanroom applications.
34
Example 6.5
cont’d
One aspect of the investigation of filter performance
involved studying how contaminant concentration in air
related to concentration on a wafer surface after prolonged
exposure.
Consider the following representative data on x = DBP
concentration in air and y = DBP concentration on a wafer
surface after 4-hour exposure (both in g/m3, where DBP
dibutyl phthalate).
Obs. i: 1
x: .8
y: .6
2
1.3
1.1
3
1.5
4.5
4
3.0
3.5
5
11.6
14.4
6
26.6
29.1
35
Example 6.5
cont’d
The authors comment that “DBP adhesion on the wafer
surface was roughly proportional to the DBP concentration
in air.”
Figure 6.3 shows a plot of y versus x-i.e., of the (x, y) pairs.
Plot of the DBP data from Example 6.5
Figure 6.3
36
Example 6.5
cont’d
If y were exactly proportional to x, we would have y = x for
some value , which says that the (x, y) points in the plot
would lie exactly on a straight line with slope  passing
through (0, 0).
But this is only approximately the case. So we now assume
that for any fixed x, wafer DBP is a random variable Y
having mean value x.
That is, we postulate that the mean value of Y is related to
x by a line passing through (0, 0) but that the observed
value of Y will typically deviate from this line (this is referred
to in the statistical literature as “regression through the
origin”).
37
Example 6.5
cont’d
We now wish to estimate the slope parameter . Consider
the following three estimators:
The resulting estimates based on the given data are
1.3497, 1.1875, and 1.1222, respectively. So the estimate
definitely depends on which estimator is used.
If one of these three estimators were unbiased and the
other two were biased, there would be a good case for
using the unbiased one.
38
Example 6.5
cont’d
But all three are unbiased; the argument relies on the fact
that each one is a linear function of the Yi’s (we are
assuming here that the xi’s are fixed, not random):
39
Estimators with Minimum
Variance
40
Estimators with Minimum Variance
Suppose
and
are two estimators of  that are both
unbiased. Then, although the distribution of each estimator
is centered at the true value of , the spreads of the
distributions about the true value may be different.
41
Estimators with Minimum Variance
Figure 6.4(a) shows distributions of two unbiased
estimators. Use of the estimator with the more
concentrated distribution is more likely than the other one
to result in an estimate close to 𝜃.
Figure 6.4(b) displays estimates from the two estimators
based on 10 different samples.
42
Estimators with Minimum Variance
The MVUE is, in a certain sense, the most likely among all
unbiased estimators to produce an estimate close to the
true  .
43
Estimators with Minimum Variance
In Example 6.5, suppose each Yi is normally distributed
with mean xi and variance  2 (the assumption of constant
variance).
Then it can be shown that the third estimator
= xiYi /  not only has smaller variance than either of
the other two unbiased estimators, but in fact is the
MVUE—it has smaller variance than any other unbiased
estimator of .
44
Example 6.6
We argued in Example 6.4 that when X1, . . . , Xn is a
random sample from a uniform distribution on [0,  ], the
estimator
 max (X1 , c, Xn)
is unbiased for  (we previously denoted this estimator by
). This is not the only unbiased estimator of  .
The expected value of a uniformly distributed rv is just the
midpoint of the interval of positive density, so E(Xi) =  /2.
This implies that E(Xi) =  /2, from which E(2X) =  . That is,
the estimator = 2X is unbiased for .
45
Example 6.6
cont’d
If X is uniformly distributed on the interval from A to B, then
V(X) =  2 = (B – A)2/12. Thus, in our situation,
V(Xi) =  2/12, V(X) =  2/n =  2 /(12n), and
V( ) = V(2X) = 4V(X) =  2/(3n).
It can be shown that
V( ) =  2/[n(n + 2)]. The estimator has smaller variance
than does if 3n < n(n + 2)—that is, if 0 < n2 – n = n(n – 1).
As long as n > 1, V( ) < V( ), so
is a better estimator
than . More advanced methods can be used to show that
is the MVUE of  —every other unbiased estimator of 
has variance that exceeds  2/[n(n + 2)].
46
Estimators with Minimum Variance
One of the triumphs of mathematical statistics has been the
development of methodology for identifying the MVUE in a
wide variety of situations.
The most important result of this type for our purposes
concerns estimating the mean  of normal distribution.
Theorem
47
Estimators with Minimum Variance
In some situations, it is possible to obtain an estimator with
small bias that would be preferred to the best unbiased
estimator. This is illustrated in Figure 6.5.
A biased estimator that is preferable to the MVUE
Figure 6.5
However, MVUEs are often easier to obtain than the type of
biased estimator whose distribution is pictured.
48
Some Complications
49
Some Complications
The last theorem does not say that in estimating a
population mean , the estimator X should be used
irrespective of the distribution being sampled.
50
Example 6.7
Suppose we wish to estimate the thermal conductivity  of a
certain material. Using standard measurement techniques,
we will obtain a random sample X1, . . . , Xn of n thermal
conductivity measurements.
Let’s assume that the population distribution is a member of
one of the following three families:
(6.1)
(6.2)
51
Example 6.7
cont’d
(6.3)
The pdf (6.1) is the normal distribution, (6.2) is called the
Cauchy distribution, and (6.3) is a uniform distribution.
All three distributions are symmetric about . The Cauchy
distribution is bell-shaped but with much heavier tails (more
probability farther out) than the normal curve.
52
Example 6.7
cont’d
The uniform distribution has no tails. The four estimators for
m considered  earlier are X , , Xe (the average of the two
extreme observations), and Xtr(10), a trimmed mean.
The very important moral here is that the best estimator for 
depends crucially on which distribution is being sampled.
In particular,
1. If the random sample comes from a normal distribution,
then X is the best of the four estimators, since it has
minimum variance among all unbiased estimators.
53
Example 6.7
cont’d
2. If the random sample comes from a Cauchy distribution,
then X and Xe are terrible estimators for , whereas is
quite good (the MVUE is not known); X is bad because it
is very sensitive to outlying observations, and the heavy
tails of the Cauchy distribution make a few such
observations likely to appear in any sample.
3. If the underlying distribution is uniform, the best estimator
is Xe; this estimator is greatly influenced by outlying
observations, but the lack of tails makes such
observations impossible.
54
Example 6.7
cont’d
4. The trimmed mean is best in none of these three
situations but works reasonably well in all three.
That is, Xtr(10), does not suffer too much in comparison
with the best procedure in any of the three situations.
55
Reporting a Point Estimate: The
Standard Error
56
Reporting a Point Estimate: The Standard Error
Besides reporting the value of a point estimate, some
indication of its precision should be given. The usual
measure of precision is the standard error of the estimator
used.
Definition
57
Example 6.9
Example 6.2… continued
Assuming that breakdown voltage is normally distributed,
is the best estimator of . If the value of  is known to
be 1.5, the standard error of X is
.
If, as is usually the case, the value of  is unknown, the
estimate = s = 1.462 is substituted into
to obtain the
estimated standard error
.
58
Reporting a Point Estimate: The Standard Error
When the point estimator has approximately a normal
distribution, which will often be the case when n is large,
then we can be reasonably confident that the true value of
 lies within approximately 2 standard errors (standard
deviations) of .
Thus if a sample of n = 36 component lifetimes gives
= x = 28.50 and s = 3.60, then
= .60, so within 2
estimated standard errors, translates to the interval
28.50  (2)(.60) = (27.30, 29.70).
59
Reporting a Point Estimate: The Standard Error
If is not necessarily approximately normal but is
unbiased, then it can be shown that the estimate will
deviate from  by as much as 4 standard errors at most 6%
of the time.
We would then expect the true value to lie within 4
standard errors of (and this is a very conservative
statement, since it applies to any unbiased ).
Summarizing, the standard error tells us roughly within
what distance of we can expect the true value of  to lie.
60
Reporting a Point Estimate: The Standard Error
The form of the estimator may be sufficiently complicated
so that standard statistical theory cannot be applied to
obtain an expression for .
This is true, for example, in the case  =  , = S; the
standard deviation of the statistic S, S, cannot in general
be determined. In recent years, a new computer-intensive
method called the bootstrap has been introduced to
address this problem.
Suppose that the population pdf is f(x; ), a member of a
particular parametric family, and that data x1, x2, . . . , xn
gives = 21.7.
61
Reporting a Point Estimate: The Standard Error
We now use the computer to obtain “bootstrap samples”
from the pdf f(x; 21.7), and for each sample we calculate a
“bootstrap estimate” :
First bootstrap sample:
; estimate =
Second bootstrap sample:
; estimate =



Bth bootstrap sample:
; estimate =
B =100 or 200 is often used. Now let
sample mean of the bootstrap estimates.
, the
62
Reporting a Point Estimate: The Standard Error
The bootstrap estimate of ’s standard error is now just
the sample standard deviation of the ’s:
(In the bootstrap literature, B is often used in place of B – 1;
for typical values of B, there is usually little difference
between the resulting estimates.)
63
Example 6.11
A theoretical model suggests that X, the time to breakdown
of an insulating fluid between electrodes at a particular
voltage, has f(x; ) = e–x, an exponential distribution.
A random sample of n = 10 breakdown times (min) gives
the following data:
41.53 18.73 2.99 30.34 12.33 117.52 73.02 223.63 4.00 26.78
Since E(X) = 1/, E(X) = 1/, so a reasonable estimate of 
is = 1/x = 1/55.087 = .018153. We then used a statistical
computer package to obtain B = 100 bootstrap samples,
each of size 10, from f(x; .018153).
64
Example 6.11
cont’d
The first such sample was 41.00, 109.70, 16.78, 6.31, 6.76,
5.62, 60.96, 78.81, 192.25, 27.61, from which
= 545.8
and
= 1/ 54.58 = .01832.
The average of the 100 bootstrap estimates is = .02153,
and the sample standard deviation of these 100 estimates
is = .0091, the bootstrap estimate of ’s standard error.
A histogram of the 100 ’s was somewhat positively
skewed, suggesting that the sampling distribution of also
has this property.
65