Transcript Medical Image Analysis - National University of Kaohsiung

Statistics
Sampling Distributions and Point Estimation of
Parameters
Contents, figures, and exercises come from the textbook: Applied Statistics and Probability
for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.
Point Estimation

Statistic
 A function of observations, X 1 , X 2 ,…, X n
 Also a random variable
 Sample mean X
 Sample variance S 2
 Its probability distribution is called a
sampling distribution
Contents, figures, and exercises come from the textbook: Applied Statistics and Probability
for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.

Point estimator of

◦ A statistic ˆ  h( X1, X 2 ,..., X n )

Point estimate of

◦ A particular numerical value ˆ
Contents, figures, and exercises come from the textbook: Applied Statistics and Probability
for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.

Mean

◦ The estimate ˆ  x

Variance
2
◦ The estimate ˆ 2  s 2

Proportion
p
◦ The estimate pˆ  x / n
◦ x is the number of items that belong to the
class of interest

Difference in means,
1   2
◦ The estimate ˆ1  ˆ 2  xˆ1  xˆ2

Difference in two proportions
◦ The estimate pˆ1  pˆ 2
p1  p2
Sampling Distributions and the
Central Limit Theorem

Random sample
◦ The random variables X 1 , X 2 ,…, X n are a
random sample of size n if (a) the X i‘s are
independent random variables, and (b) every X i
has the same probability distribution

If X 1 , X 2 ,…, X n are normally and
independently distributed with mean
and variance  2

◦ X  X 1  X 2    X n has a normal distribution
n
◦ with mean
X 
    
n

◦ variance
 
2
X
 2  2  2
n
2

2
n
Contents, figures, and exercises come from the textbook: Applied Statistics and Probability
for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.

Central Limit Theorem
◦ If X 1 , X 2 ,…, X n is a random sample of size n
taken from a population (either finite or
infinite) with mean  and finite variance  2 ,
and if X is the sample mean, the limiting form
of the distribution of
Z
X 
/ n
◦ as n  , is the standard normal distribution.

Works when
◦ n  30 , regardless of the shape of the
population
◦ n  30 , if not severely nonnormal

Two independent populations with means
2
2



and 2 , and variances 1 and 2
Z
1
X 1  X 2  ( 1   2 )
 12 / n1   22 / n2
◦ is approximately standard normal, or
◦ is exactly standard normal if the two
populations are normal
Contents, figures, and exercises come from the textbook: Applied Statistics and Probability
for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.

Example 7-1 Resistors
◦ An electronics company manufactures
resistors that have a mean resistance of 100
ohms and a standard deviation of 10 ohms.
The distribution of resistance is normal. Find
the probability that a random sample of n  25
resistors will have an average resistance less
than 95 ohms

Example 7-2 Central Limit Theorem
◦ Suppose that
distribution
X
has a continuous uniform
1 / 2 4  x  6
f ( x)  
 0 otherwise
◦ Find the distribution of the sample mean of a
random sample of size n  40

Example 7-3 Aircraft Engine Life
◦ The effective life of a component used in a jet-turbine
aircraft engine is a random variable with mean 5000
hours and standard deviation 40 hours. The
distribution of effective life is fairly close to a normal
distribution. The engine manufacturer introduces an
improvement into the manufacturing process for this
component that increases the mean life to 5050
hours and decreases the standard deviation to 30
hours. Suppose that a random sample of n1  16
components is selected from the “old” process and a
random sample of n2  25 components is selected
from the “improved” process. What is the probability
that the difference in the two sample means X 2  X1
is at least 25 hours? Assume that the old and
improved processes can be regarded as independent
populations.

Exercise 7-10
◦ Suppose that the random variable X has the
continuous uniform distribution
1 0  x  1
f ( x)  
0 otherwise
◦ Suppose that a random sample of n  12 observations
is selected from this distribution. What is the
approximate probability distribution of X  6 ? Find
the mean and variance of this quantity.
Contents, figures, and exercises come from the textbook: Applied Statistics and Probability
for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.
General Concepts of Point
Estimation


ˆ
Bias of the estimator 
ˆ 
E()
ˆ is an unbiased estimator if

ˆ 
E()
 Minimum variance unbiased estimator (MVUE)

◦ For all unbiased estimator of  , the one with the
smallest variance
X is the MVUE for 
◦ If X 1 , X 2 ,…, X n are from a normal distribution with
mean  and variance  2

Standard error of an estimator
ˆ

ˆ)
 ˆ  V (

Estimated standard error
◦

If
ˆ)
ˆ ˆ or sˆ or se(
X
is normal with mean
2 /n

X 

n
and
ˆ X 

and variance
S
n
Contents, figures, and exercises come from the textbook: Applied Statistics and Probability
for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.

Mean squared error of an estimate
ˆ

ˆ )  E[(
ˆ   )2 ]
MSE (
ˆ  E (
ˆ )  E (
ˆ )   )2 ]
 E[(
ˆ  E (
ˆ ))2 ]  E[(E (
ˆ )   ) 2 ]  2 E[
ˆ  E (
ˆ )] ( E (
ˆ )  )
 E[(
ˆ  E (
ˆ ))2 ]  ( E (
ˆ )   )2
 E[(
ˆ )  (bias) 2
 V (

Relative efficiency of
ˆ

2
to
ˆ

1
ˆ )
MSE (
1
ˆ )
MSE (
2
Contents, figures, and exercises come from the textbook: Applied Statistics and Probability
for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.

Example 7-4 Sample Mean and Variance
Are Unbiased
◦ Suppose that X is a random variable with mean 
and variance  2 . Let X 1 , X 2 ,…., X n be a random
sample of size n from the population represented by
X . Show that the sample mean X and sample variance
S 2 are unbiased estimators of  and  2 , respectively.

Example 7-5 Thermal Conductivity
◦ Ten measurements of thermal conductivity were
obtained:
◦ 41.60, 41.48, 42.34, 41.95, 41.86
◦ 42.18, 41.72, 42.26, 41.81, 42.04
s
0.284
◦ Show that x  41.924 and ˆ x 

 0.0898
n
10

Exercise 7-31
◦ X 1 and S12 are the sample mean and sample variance
from a population with mean 1 and variance  12 .
Similarly, X 2 and S22 are the sample mean and sample
variance from a second independent population with
mean 2 and variance  22 . The sample sizes are n1
and n2 , respectively.
◦ (a) Show that X1  X 2 is an unbiased estimator of
1   2
◦
◦ (b) Find the standard error of X1  X 2 . How could
you estimate the standard error?
◦ (c) Suppose that both populations have the same
2
2
2





variance; that is, 1
. Show that
2
(n1  1)S12  (n2  1)S22
S 
n1  n2  2
2

◦ Is an unbiased estimator of
.
2
p
Methods of Point Estimation

Moments
◦ Let X 1 , X 2 ,…, X n be a random sample from
the probability distribution f (x) , where f (x)
can be a discrete probability mass function or
a continuous probability density function. The
k th population moment (or distribution
k
E
(
X
) , k = 1, 2,…. The
moment) is
1 n
corresponding k th sample moment is i 1 X ik
n
k = 1, 2, ….
Contents, figures, and exercises come from the textbook: Applied Statistics and Probability
for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.

Moment estimators
◦ Let X 1 , X 2 ,…, X n be a random sample from
either a probability mass function or a
probability density function with m unknown
parameters  1 , 2 ,…,  m . The moment
estimators ˆ 1 , ˆ 2 ,…, ˆ m are found by
equating the first m population moments to
the first m sample moments and solving the
resulting equations for the unknown
parameters.
Contents, figures, and exercises come from the textbook: Applied Statistics and Probability
for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.

Maximum likelihood estimator
◦ Suppose that X is a random variable with
probability distribution f ( x; ) , where  is a
single unknown parameter. Let x1 , x2 ,…, xn
be the observed values in a random sample of
size n . Then the likelihood function of the
sample is
L( )  f ( x1; )  f ( x2 ; )  f ( xn ; )
◦ Note that the likelihood function is now a
function of only the unknown parameter  .
The maximum likelihood estimator (MLE) of 
is the value of  that maximizes the likelihood
function L( ) .

Properties of a Maximum Likelihood
Estimator
◦ Under very general and not restrictive
conditions, when the sample size n is large
and if ˆ is the maximum likelihood
estimator of the parameter  ,
◦ (1) ˆ is an approximately unbiased estimator
ˆ ) ]
for  [E(
◦ (2) the variance of ˆ is neatly as small as the
variance that could be obtained with any
other estimator, and
◦ (3) ˆ has an approximate normal distribution.
Contents, figures, and exercises come from the textbook: Applied Statistics and Probability
for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.

Invariance property
◦ Let ˆ 1 , ˆ 2 ,…., ˆ k be the maximum
likelihood estimators of the parameters  1 , 2
, …, k . Then the maximum likelihood
estimator of any function h(1,2 ,...,k ) of
these parameters is the same function
ˆ ,
ˆ ,...,
ˆ )
h(
1
2
k
◦ of the estimators ˆ 1 , ˆ 2 ,…, ˆ k .
Contents, figures, and exercises come from the textbook: Applied Statistics and Probability
for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.

Bayesian estimation of parameters
◦ Sample X 1 , X 2 ,…, X n
◦ Joint probability distribution
f ( x1 , x2 ,...,xn , )  f ( x1 , x2 ,...,xn |  ) f ( )
◦ Prior distribution for 
f ( )
◦ Posterior distribution for 
f ( | x1 , x2 ,..., xn ) 
f ( x1 , x2 ,..., xn , )
f ( x1 , x2 ,..., xn )
◦ Marginal distribution
  f ( x1 , x2 ,..., xn , )

f ( x1 , x2 ,..., xn )    
f ( x1 , x2 ,..., xn , )d


 
θ discrete
θ continuous

Example 7-6 Exponential Distribution
Moment Estimator
◦ Suppose that X 1 , X 2 ,…, X n is a random sample from
an exponential distribution with parameter  . For
the exponential, E (X )  .
◦ Then E ( X )  X results in 1 /   X . ˆ  1 / X

Example 7-7 Normal Distribution
Moment Estimators
◦ Suppose that X 1 , X 2 ,…, X n is a random sample from a
normal distribution with parameters  and  2 . For
the normal distribution, E (X )   and
◦ E( X 2 )   2   2 . Equating E (X ) to X and E ( X 2 )
n
to 1n i 1 X i2 gives
n
2
2
2
1




X
◦
i
n i 1
  X and
◦ Solve these equations.

Example 7-8 Gamma Distribution
Moment Estimators
◦ Suppose that X 1 , X 2 ,…, X n is a random sample from a
gamma distribution with parameters r and  , For
the gamma distribution, E ( X )  r /  and
◦ E( X 2 )  r (r  1) / 2 . Solve
n
2
2
1
r
(
r

1
)
/


X
◦
and
r /  X
i
n i 1
Contents, figures, and exercises come from the textbook: Applied Statistics and Probability
for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.

Example 7-9 Bernoulli Distribution MLE
◦ Let X be a Bernoulli random variable. The probability
mass function is
 p x (1  p)1 x
x  0,1
f ( x; p)  
0
otherwise

◦ where p is the parameter to be estimated. The
likelihood function of a random sample of size n is
n
n
L( p )   p (1  p )
xi
1 xi
 xi
 p i1 (1  p )
n
n
 xi
i 1
i 1
◦ Find pˆ that maximizes L( p ) .
Contents, figures, and exercises come from the textbook: Applied Statistics and Probability
for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.

Example 7-10 Normal Distribution MLE
◦ Let X be normally distributed with unknown  and
known variance  2 . The likelihood function of a
random sample of size n , say X 1, X 2 ,…, X n , is
n
1
1
 ( xi   ) 2 /( 2 2 )
L(  )  
e

e
2 n/2
(2 )
2
i 1 
◦ Find ˆ .

1
2 2
n
 ( xi   ) 2
i 1
Example 7-11 Exponential Distribution
MLE
◦ Let X be exponentially distributed with parameter 
. The likelihood function of a random sample of size n
, say X 1 , X 2 ,…, X n , is
n
L( )   exi  n e
i 1
◦ Find ˆ .
n
  xi
i 1

Example 7-12 Normal Distribution MLEs
for  and  2
◦ Let X be normally distributed with mean  and
variance  2 , where both  and  2 are unknown.
The likelihood function of a random sample of size n
is
1
n
( x  )2
2 i
2
2
1
1
2 i 1
L(  ,  2 )  
e ( xi   ) /(2 ) 
e
(2 2 ) n / 2
2
i 1 
n
◦ Find ˆ and ˆ 2 .

Example 7-13
◦ From Example 7-12, to obtain the maximum
likelihood estimator of the function h( ,  2 )   2  
◦ Substitute the estimators ˆ and ˆ 2 into the function
1/ 2
n
, which yields


ˆ  ˆ 2   1n  ( X i  X ) 2 
 i 1


Example 7-14 Uniform Distribution MLE
◦ Let X be uniformly distributed on the interval 0 to a
. Since the density function is f ( x)  1 / a for 0  x  a
and zeros otherwise, the likelihood function of a
random sample of size n is
n
1 1
L( a )    n
a
i 1 a
◦ for 0  x1  a, 0  x2  a ,…, 0  xn  a
◦ Find aˆ .
Contents, figures, and exercises come from the textbook: Applied Statistics and Probability
for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.

Example 7-15 Gamma Distribution MLE
◦ Let X 1 ,X 2 ,…, X n be a random sample from the
gamma distribution. The log of likelihood function is
 n r xir 1e   xi
ln L(r ,  )  ln 
(r )
 i 1



n
n
i 1
i 1
 nr ln( )  (r  1) ln(xi )  n ln[(r )]    xi
◦ Find that
rˆ
ˆ

x
n ln(ˆ) 
' (rˆ)
ln(xi )  n

(rˆ)
i 1
n
Contents, figures, and exercises come from the textbook: Applied Statistics and Probability
for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.

Example 7-16 Bayes Estimator for the
Mean of a Normal Distribution
◦ Let X 1, X 2,…, X n be a random sample from the
normal distribution with mean  and variance  2 ,
where  is unknown and  2 is known. Assume
that the prior distribution for  is normal with mean 0
2
and variance  0 ; that is,
1
f ( ) 
e (    ) /(2 )
2  0
2
2
0
0
◦ The joint probability distribution of the sample is
f ( x1 , x2 ,..., xn |  ) 
1
(2 )
2 n/2
 (1/ 2 )
2
e
n
 ( xi   ) 2
i 1
Contents, figures, and exercises come from the textbook: Applied Statistics and Probability
for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.
◦ Show that
   02 x  ( 2 / n ) 0
(1/ 2 )( 2  2 )   
 0  / n    02  2 / n
1
f ( | x1 , x2 ,..., xn )  e
1




2
◦ Then the Bayes estimate of  is
 02 x  ( 2 / n)0
 02   2 / n
Contents, figures, and exercises come from the textbook: Applied Statistics and Probability
for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.

Exercise 7-42
◦ Let X 1, X 2 ,…, X n be uniformly distributed on the
interval 0 to a . Recall that the maximum likelihood
estimator of a is aˆ  max(X i ) .
◦ (a) Argue intuitively why aˆ cannot be an unbiased
estimator for a .
◦ (b) Suppose that E (aˆ )  na /(n  1) . Is it reasonable
that aˆ consistently underestimates a ? Show that
the bias in the estimator approaches zero as n gets
large.
◦ (c) Propose an unbiased estimator for a .
◦ (d) Let Y  max(X i ) . Use the fact that Y  y if and
only if each X i  y to derive the cumulative
distribution function of Y . Then show that the
probability density function of Y is
nyn1
f ( y)  a n
0
0 ya
otherwiae
◦ Use this result to show that the maximum likelihood
estimator for a is biased.
◦ (e) We have two unbiased estimators for a : the
moment estimator aˆ1  2 X and
◦ aˆ2  [(n  1) / n] max(X i ) , where max(X i ) is the
largest observation in a random sample of size n . It
can be shown that V (aˆ1 )  a 2 /(3n) and that
◦ V (aˆ2 )  a 2 /[n(n  2)] . Show that if n  1 , aˆ2 is a
better estimator than aˆ1 . In what sense is it a better
estimator of a ?
Contents, figures, and exercises come from the textbook: Applied Statistics and Probability
for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.

Exercise 7-50
◦ The time between failures of a machine has an
exponential distribution with parameter  . Suppose
that the prior distribution for  is exponential with
mean 100 hours. Two machines are observed, and the
average time between failures is x  1125 hours.
◦ (a) Find the Bayes estimate for  .
◦ (b) What proportion of the machine do you think will
fail before 1000 hours?
Contents, figures, and exercises come from the textbook: Applied Statistics and Probability
for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.
Contents, figures, and exercises come from the textbook: Applied Statistics and Probability
for Engineers, 5th Edition, by Douglas C. Montgomery, John Wiley & Sons, Inc., 2011.