Transcript LIMITING DISTRIBUTIONS - Middle East Technical University

ORDER STATISTICS AND
LIMITING DISTRIBUTIONS
1
ORDER STATISTICS
• Let X1, X2,…,Xn be a r.s. of size n from a
distribution of continuous type having pdf
f(x), a<x<b. Let X(1) be the smallest of Xi,
X(2) be the second smallest of Xi,…, and
X(n) be the largest of Xi.
a  X 1  X  2 
 X  n  b
• X(i) is the i-th order statistic.
X 1  min  X 1 , X 2 ,
, X n
X  n   max  X 1 , X 2 ,
, X n
2
ORDER STATISTICS
• It is often useful to consider ordered random
sample.
• Example: suppose a r.s. of five light bulbs is
tested and the failure times are observed as
(5,11,4,100,17). These will actually be observed
in the order of (4,5,11,17,100). Interest might be
on the kth smallest ordered observation, e.g. stop
the experiment after kth failure. We might also be
interested in joint distributions of two or more
order statistics or functions of them (e.g.
range=max – min)
3
ORDER STATISTICS
• If X1, X2,…,Xn is a r.s. of size n from a population
with continuous pdf f(x), then the joint pdf of the
order statistics X(1), X(2),…,X(n) is
g  x1 , x 2 ,
, x n    n! f  x1  f  x 2 
f  x n  
for x(1)  ...  x(n)
Order statistics are not independent.
The joint pdf of ordered sample is not same as the joint
pdf of unordered sample.
Future reference: For discrete distributions, we need to take ties
into account (two X’s being equal). See, Casella and Berger,
1990, pg 231.
4
Example
• Suppose that X1, X2, X3 is a r.s. from a
population with pdf
f(x)=2x for 0<x<1
Find the joint pdf of order statistics and the
marginal pdf of the smallest order statistic.
5
ORDER STATISTICS
• The Maximum Order Statistic: X(n)

GX  n   y   P X  n   y


g X  n  y   GX  n  y 
y
6
ORDER STATISTICS
• The Minimum Order Statistic: X(1)

GX 1  y   P X 1  y


g X 1  y   G X 1  y 
y
7
ORDER STATISTICS
• k-th Order Statistic
y1 y2 … yk-1 yk yk+1 …
P(X<yk)
y
yn
P(X>yk)
# of possible orderings
n!/{(k1)!1!(n  k)!}
fX(yk)
g X k  y
 
k 1
nk
n!
 FX  y  f X  y  1  FX  y  , a  y  b

 k  1! n  k !
8
Example
• Same example but now using the previous
formulas (without taking the integrals):
Suppose that X1, X2, X3 is a r.s. from a
population with pdf
f(x)=2x for 0<x<1
Find the marginal pdf of the smallest order
statistic.
9
Example
• X~Uniform(0,1). A r.s. of size n is taken.
Find the p.d.f. of kth order statistic.
• Solution: Let Yk be the kth order statistic.
n!
g Yk ( y) 
y k 1 (1  y) n  k 1
( k  1)!( n  k )!

( n  1)
y k 1 (1  y) n  k
(k )( n  k  1)
for 0  y  1
Yk ~ Bet a( k , n  k  1)
10
ORDER STATISTICS
• Joint p.d.f. of k-th and j-th Order Statistic (for k<j)
k-1 items
y1 y2 … yk-1
P(X<yk)
j-k-1 items
1 item
1 item
# of possible orderings
n!/{(k1)!1!(j-k-1)!1!(n  j)!}
y
yk yk+1
…
yj-1 yj yj+1
P(yk<X<yj)
fX(yk)
g X (k ) ,X ( j) ( y k , y j ) 
n-j items
yn
P(X>yj)
fX(yj)
n!
[F( y k )]k 1f ( yk )[F( y j )  F( y k )]j k 1f ( y j )[1  F( y j )]n  j
(k  1)!( j  k  1)!(n  j)!
11
LIMITING DISTRIBUTION
12
Motivation
• The p.d.f. of a r.v. often depends on the
sample size (i.e., n)
• If X1, X2,…, Xn is a sequence of rvs and
Yn=u(X1, X2,…, Xn) is a function of them,
sometimes it is possible to find the exact
distribution of Yn (what we have been doing lately)
• However, sometimes it is only possible to
obtain approximate results when n is large
 limiting distributions.
13
CONVERGENCE IN DISTRIBUTION
• Consider that X1, X2,…, Xn is a sequence of
rvs and Yn=u(X1, X2,…, Xn) be a function of
rvs with cdfs Fn(y) so that for each n=1, 2,…
F  y   P Y  y  ,
n
n
lim F  y   F  y  for all y
n
n
where F(y) is continuous. Then, the sequence
Yn is said to converge in distribution to Y.
d
Y Y
n
14
CONVERGENCE IN DISTRIBUTION
• Theorem: If lim F  y   F  y  for every
n
n
point y at which F(y) is continuous, then Yn
is said to have a limiting distribution with
cdf F(y). The term “Asymptotic distribution”
is sometimes used instead of “limiting
distribution”
• Definition of convergence in distribution
requires only that limiting function agrees
with cdf at its points of continuity.
15
Example
Let X1,…, Xn be a random sample from
Unif(0,1).
Find the limiting distribution of the max
order statistic, if it exists.
16
Example
Let {Xn} be a sequence of rvs with pmf
1, if x  2  1

f  x  P  X  x  
n
0, o.w.
n
Find the limiting distribution of Xn , if it exists.
17
Example
Let Yn be the nth order statistic of a random
sample X1, …, Xn from Unif(0,θ).
Find the limiting distribution of Zn=n(θ -Yn),
if it exists.
18
Example
Let X1,…, Xn be a random sample from
Exp(θ).
Find the limiting distribution of the min
order statistic, if it exists.
19
Example
• Suppose that X1, X2, …, Xn are iid from
Exp(1).
Find the limiting distribution of the max
order statistic, if it exists.
20
LIMITING MOMENT
GENERATING FUNCTIONS
• Let rv Yn have an mgf Mn(t) that exists for
all n. If
lim M n  t   M  t  ,
n
then Yn has a limiting distribution which is
defined by M(t).
21
Example
X ~ Bin  n, p  with mgf M  t    pe  1  p 
t
n
X
Let =np.
lim M
n
 t   lim  pe  1  p 
t
X
n
n

   e  1 

 lim 1 
 M t 
 e
n


t
n
n
et 1
Y
The mgf of Poisson()
The limiting distribution of Binomial rv is the
22
Poisson distribution.
CONVERGENCE IN PROBABILITY
(STOCHASTIC CONVERGENCE)
• A rv Yn convergence in probability to a rv Y if
lim P  Yn  Y     1
n
for every >0.
23
CHEBYSHEV’S INEQUALITY
• Let X be an rv with E(X)= and V(X)=2.

P  X       1  2 ,  0

2
• The Chebyshev’s Inequality can be
used to prove stochastic convergence in
many cases.
24
CONVERGENCE IN PROBABILITY
(STOCHASTIC CONVERGENCE)
• The Chebyshev’s Inequality proves the
convergence in probability if the following
three conditions are satisfied.
n  .
1. E(Yn)=n where lim
n
2. V Yn      for all n.
2
n
3. lim n2  0.
n
25
Example
Let X be an rv with E(X)= and V(X)=2<.
For a r.s.
of
size
n,
is
the
sample
mean.
X
n
p
Is X n   ?
26
WEAK LAW OF LARGE
NUMBERS (WLLN)
• Let X1, X2,…,Xn be iid rvs withn E(Xi)= and
V(Xi)=2<. Define X n  1/ n X i . Then, for
i 1
every >0,
X n converges in probability to .
WLLN states that the sample mean is a good
estimate of the population mean. For large n, the
sample mean and population mean are close to
each other with probability 1. But more to come on
the properties of estimators.
27
STRONG LAW OF LARGE
NUMBERS
• Let X1, X2,…,Xn be iid rvs withn E(Xi)= and
V(Xi)=2<. Define X n  1/ n X i . Then, for
i 1
every >0,


P lim X n      1
n
that is, X n converges almost surely to .
28
Relation between convergences
Almost sure convergence is the strongest.
a.s
p
d
Yn  Y  Yn  Y  Yn  Y
(reverse is generally not true)
29
THE CENTRAL LIMIT THEOREM
• Let X1, X2,…,Xn be a sequence of iid rvs
with E(Xi)= and V(Xi)=2<∞. Define
n
X n  1/ n X i . Then,
i 1
Z
n  Xn  

or
n
Z
X
i 1
i
 n
n
d
 N  0,1
d
 N  0,1 .
Proof can be found in many books, e.g. Bain and Engelhardt, 1992, page 239
30
Example
• Let X1, X2,…,X
n be iid rvs from Unif(0,1)
n
and Yn   Xi
i 1
• Find approximate distribution of Yn.
• Find approximate values of
–
40
P( X i  12)
i 1
– The 90th percentile of Yn.
31
SLUTKY’S THEOREM
• If XnX in distribution and Yna, a
constant, in probability, then
a) YnXnaX in distribution.
b) Xn+YnX+a in distribution.
32
SOME THEOREMS ON LIMITING
DISTRIBUTIONS
• If Xn c>0 in probability, then for any
function g(x) continuous at c, g(Xn) g(c) in
p
prob. e.g.
X n  c.
• If Xnc in probability and Ynd in probability,
then
• aXn+bYn ac+bd in probability.
• XnYn cd in probability
• 1/Xn 1/c in probability for all c0.
33
Example
1. X~Gamma(, 1). Show that
n  Xn  
Xn
d
 N  0,1.
34
Example
2. Let X1,…,Xn be r.v.s. each with
Gamma(n+1,β). Show that Xn/n converges
in probability to a constant c, and find c.
35
Example
3. Let Xn be “displaced exponential” with
p.d.f.
e ( x / n) if x   / n
f n ( x)  
o.w.
0
Find the limiting distribution.
36
Example
4. Let the sample space S={1,2,3,4}. Define the
following r.v.s:
Xn(1)=Xn(2)=1,Xn(3)=Xn(4)=0 for n=1,2,… and
X(1)=X(2)=0, X(3)=X(4)=1
a) Does Xn converge in probability to X as n→∞?
b) Does Xn converge in distribution to X as n→∞?
37