Transcript Lecture03-Probability Review

Al-Imam Mohammad bin Saud University
CS433: Modeling and Simulation
Lecture 03: Probability Review
Dr. Anis Koubâa
05 March 2010
Goals of today
Review the fundamental concepts of
probability
 Understand the difference between discrete
and continuous random variable

2
Topics


Fundamental Laws of Probability
Discrete Random Variable


Continuous Random Variable



Probability Mass Function (PMF)
Probability Density Function (PDF)
Cumulative Distribution Function (CDF)
Mean and Variance
3
Probability Review
Fundamentals
 We

measure the probability for Random Events
How likely an event would occur
 The
set of all possible events is called Sample Space
 In each experiment, an event may occur with a certain
probability (Probability Measure)
 Example:



Tossing a dice with 6 faces
The sample space is {1, 2, 3, 4, 5, 6}
Getting the Event « 2 » in on experiment has a probability 1/6
5
Probability



The probability of every set of possible events is between 0 and
1, inclusive.
The probability of the whole set of outcomes is 1.
 Sum of all probability is equal to one
 Example for a dice: P(1)+P(2)+P(3)+ P(4)+P(5)+P(6)=1
If A and B are two events with no common outcomes, then the
probability of their union is the sum of their probabilities.
 Event E1={1},
 Event E2 ={6}
 P(E1 U E2)=P(E1)+P(E2)
6
Complementary Event

Complementary Event of A is not(A)

P(A)=1-P(not A)

The probability that event A will not happen is 1-P(A).

Example

Event E1={1}

Probability to get a value different from {1} is 1-P(E1).
7
Joint Probability

Event Union (U = OR)

Joint Probability (A ∩ B)

Event Intersection (∩= AND)
The probability of two events in conjunction. It is the probability of both events together.
p A  B   p A   p B   p A  B 

Independent Events

Two events A and B are independent if
p A  B   p A   p B 
8
Example on Independence
E1: Drawing Ball 1
E2: Drawing Ball 2
E3: Drawing Ball 3
1
2
3
P(E1): 1/3
P(E2):1/3
P(E3): 1/3
p A  B   p A   p B 
Case 1: Drawing with replacement of the ball
The second draw is independent of the first draw
1 1 1
p  E 1  E 2      p  E 1  p  E 2 
3 3 9
Case 2: Drawing without replacement of the ball
The second draw is dependent on the first draw
p  E 1 E 2 
1 1 1
   p  E 1  p  E 2 
3 2 6
Quiz: Show that in Case 2, we have
p  E 1  E 2 
1
2
9
Conditional Probability

Conditional Probability p(A|B) is the probability of some
event A, given the occurrence of some other event B.
p A | B  

p A  B 
p B 
p B | A  
P A 
If A and B are independent,
then p  A | B   p  A  and p  B | A   p  B 

If A and B are independent, the conditional probability of A, given B is
simply the individual probability of A alone; same for B given A.

p(A) is the prior probability;
p(A|B) is called a posterior probability.

Once you know B is true, the universe you care about shrinks to B.

p B  A 
10
Example on Independence
E1: Drawing Ball 1
E2: Drawing Ball 2
E3: Drawing Ball 3
1
2
3
P(E1): 1/3
P(E2):1/3
P(E3): 1/3
p A | B  
p A  B 
p B 
Case 1: Drawing with replacement of the ball
The second draw is independent of the first draw
1
p  E 1| E 2  
3
1 1
p  E 1 E 2 3  3 1
p  E 1| E 2  


1
p  E 2
3
3
Case 2: Drawing without replacement of the ball
The second draw is dependent on the first draw
1
p  E 1| E 2  
2
1 1
p  E 1  E 2 3  2 1
p  E 1| E 2  


1
p  E 2
2
3
11
Baye’s Rule

We know that
p A  B   p B  A 

Using Conditional Probability definition, we have
p A | B   P B   p B | A   P A 

The Bayes rule is:
p B | A   p A 
p A | B  
p B 
12
Law of Total Probability


In probability theory, the law of total probability means
the prior probability of A, P(A), is equal to
the expected value of the posterior probability of A.
That is, for any random variable N,
p  A   E  p  A | N  

where p(A|N) is the conditional probability of A given N.
13
Law of Total Probability


Law of alternatives: The term law of total probability is
sometimes taken to mean the law of alternatives, which is
a special case of the law of total probability applying to
discrete random variables.
if { Bn : n = 1, 2, 3, ... } is a finite partition of a probability
space and each set Bn is measurable, then for any event A we
have
p A  
p A  B 

n
n
or, alternatively, (using Rule of Conditional Probability)
p A  
 p A | B
n
  p B 
n
14
Example: Law of Total Probability
Sample Space
Partitions
Event
S  {1, 2,3, 4,5,6,7}
B 1  {1,5}
B 2  {2,3, 6}
B 3  {4,7}
A  {3}
Law of Total Probability
p  A   p {3}  p  A  B1   p  A  B 2   p  A  B 3 
 0  p {3}  {2,3,6}  0  p {3}
15
Five Minutes Break
You are free to discuss with your classmates about the
previous slides, or to refresh a bit, or to ask questions.
Administrative issues

Groups Formation (mandatory to do it this week)
A random variable
► Random
variable is a measurable function from a
sample space of events  to the measurable
space S of values of possible values of the variable
X : S
Ei
xi
Event
Value
Each value/event has a probability of occurrence
17
A random variable
► Random
Variable is also know as stochastic
variable.
►
A random variable is not a variable. It is a function. It maps
from the sample space to the real numbers.
► random
variable is defined as a quantity whose values
are random and to which a probability distribution
is assigned.
18
A random variable: Examples.
► The
number of packets that arrives to the destination
► The
waiting time of a customer in a queue
► The
number of cars that enters the parking each hour
► The
number of students that succeed in the exam
19
Random Variable Types
►
Discrete Random Variable:
► possible
values are discrete (countable sample space,
Integer values)
X :   {1, 2,3, 4,...}
Ei
► Continuous
► possible
values)
xi
Random Variable:
values are continuous (uncountable space, Real
X :   1.4,32.3
Ei
xi
20
Discrete Random Variable

The probability distribution for discrete random
variable is called Probability Mass Function (PMF).
p x i   P X  x i

Properties of PMF
0  p x i   1



 p x
and
i
 1
i
Cumulative Distribution Function (CDF)
p  X  x   x
Mean value
X  E  X  
n
x
i
i x
p (x i )
 p x i

i 1
21
Discrete Random Variable

Mean (Expected) value
X  E  X  
n
x
i
 p x i

i 1

Variance
V  X   X2  E  X  E  X
V X  
n

 x i  x

2
 p x i

2
 
E X
2
 E X


2
General Equation

i 1
 n
V X    x i

 i 1

2
  n
 p x i   
x i  p x i
 
  i 1


 

2
For Discrete RV
22
Discrete Random Variable: Example

Random Variable: Grades of the students
Student ID 1
2
3
4
5
6
7
8
9
10
Grade
2
3
1
2
3
1
3
2
2
3
Probability Mass Function
2
p 1  P  X  1 
 0.2
0  p 1  1
10
4
p  2  P  X  2 
 0.4
10
4
p  3  P  X  3 
 0.4
10
PDF
0  p  2  1
0  p  3  1
Grade
23
Discrete Random Variable: Example

Random Variable: Grades of the students
Student ID 1
2
3
4
5
6
7
8
9
10
Grade
2
3
1
2
3
1
3
2
2
3
Probability Mass Function Property
 p x
i
CDF
  p 1  p  2  p 3  1
i
Cumulative Distribution Function
p  X  x   x
i
x
p (x i )
p  X  2  x
i
2
p (x i )  p 1  p  2   0.2  0.4  0.6
p  X  3   x
i
2
p (x i )  p 1  p  2   p  3  1
Grade
24
Discrete Random Variable: Example

Random Variable: Grades of the students
Student ID 1
2
3
4
5
6
7
8
9
10
Grade
2
3
1
2
3
1
3
2
2
3
The Mean Value
x  p x
Grade
i
i
  1 0.2  2  0.4  3  0.4  2.2
i
i
10
 2.2
25
Continuous Random Variable


The probability distribution for continuous random
variable is called Probability Density Function (PDF).
f x 
The probability of a given value is always 0



The sample space is infinite
p x  x i   0
For continuous random variable, we compute p a  x  b 
Properties of PDF
1. f ( x)  0 , for all x in R X
2.
 f ( x)dx  1
RX
3. f ( x)  0, if x is not in RX
26
Continuous Random Variable

Cumulative Distribution Function (CDF)
p  X  x   x
i x
p (x i )
p  X  x    f t  dt  0
x


Mean/Expected value
X  E  X  
p a  X  b    f  x  dx
b

 x  f  x  dx
a


Variance
V X  

 x  

x

2
 f  x  dx
and
 
V  X    x 2  f  x  dx

 


  x2


27
Discrete versus Continuous Random Variables
Discrete Random Variable
Continuous Random Variable
Finite Sample Space
e.g. {0, 1, 2, 3}
Probability Mass Function (PMF)
p x i   P X  x i 
1. p (x i )  0, for all i
2.  i 1 p (x i )  1

Infinite Sample Space
e.g. [0,1], [2.1, 5.3]
Probability Density Function (PDF)
f x 
1. f ( x)  0 , for all x in R X
2.
 f ( x)dx  1
RX
3. f ( x)  0, if x is not in RX
Cumulative Distribution Function (CDF) p  X  x 
p  X  x   x
i
x
p (x i )
p  X  x    f t  dt  0
x

p a  X  b    f  x  dx
b
a
28
Continuous Random Variables: Example
Example: modeling the waiting time in a queue

People waiting for service in bank queue

Time is a continuous random variable


Random Time is typically modeled as exponential distribution
 is the mean value
Exponential Distribution
Exp (µ)
1
 x

exp
 ,


f (x )   
 
0,

x 0
otherwise
29
Continuous Random Variables: Example

We assume that with average waiting time of one customer
is 2 minutes
PDF: f (time)
 1  x / 2
e , x0
f ( x)   2
0,
otherwise
time
30
Continuous Random Variables: Example

Probability that the customer waits exactly 3 minutes is:
1 3  x /2
P (x  3)  P (3  x  3)  3 e dx  0
2

Probability that the customer waits between 2 and 3 minutes is:
1 3  x /2
P (2  x  3)   e dx  0.145
2 2
P(2  X  3)  F (3)  F (2)  (1  e (3 / 2) )  (1  e 1 )  0.145

CDF
The Probability that the customer waits less than 2 minutes
P (0  X  2)  F (2)  F (0)  F (2)  1  e 1  0.632
CDF
31
Continuous Random Variables: Example
Expected Value and Variance

The mean of life of the previous device is:
E (X ) 

1

2 0
xe
x /2
dx 
xe

x /2 



  e  x / 2dx  2
0
0
To compute variance of X, we first compute E(X2):


1
x / 2
2 x / 2
2
E ( X )   x e dx   x e
  e  x / 2 dx  8
0
2 0
0
2


Hence, the variance and standard deviation of the device’s life
are:
2
V (X )  8  2  4
  V (X )  2
32
Two Minutes Break
You are free to discuss with your classmates about the
previous slides, or to refresh a bit, or to ask questions.
Administrative issues

Groups Formation (mandatory to do it this week)
Variance

The standard deviation is defined as the square
root of the variance, i.e.:
X 
2
X
 V X
 s
34
Coefficient of Variation

The Coefficient of Variance of the random variable X is
defined as:
CV
V X
 X

X  
E  X  X
35
End of lecture
There are some additional slides afterwards that may help to
explain some concepts
Discrete Probability Distribution

The probability distribution of a discrete random
variable is a list of probabilities associated with each of
its possible values.

It is also sometimes called the probability function or
the probability mass function (PMF) for discrete
random variable.
37
Probability Mass Function (PMF)

Formally

the probability distribution or probability mass function
(PMF) of a discrete random variable X is a function that
gives the probability p(xi) that the random variable equals xi,
for each value xi:
p x i   P X  x i


It satisfies the following conditions:
0  p x i   1
 p x
i
 1
i
38
Continuous Random Variable


A continuous random variable is one which takes an
infinite number of possible values.
Continuous random variables are usually measurements.

Examples include height, weight, the amount of sugar in an orange,
the time required to run a mile.
39
Probability Density Function (PDF)




For the case of continuous variables, we do not want to
ask what the probability of "1/6" is, because the answer is
always 0...
Rather, we ask what is the probability that the value is in
the interval (a,b).
So for continuous variables, we care about the derivative of
the distribution function at a point (that's the derivative of an
integral). This is called a probability density function
(PDF).
The probability that a random variable has a value in a set A is
the integral of the p.d.f. over that set A.
40
Probability Density Function (PDF)


The Probability Density Function (PDF) of a continuous
random variable is a function that can be integrated to
obtain the probability that the random variable takes a value in
a given interval.
More formally, the probability density function, f(x), of a
continuous random variable X is the derivative of the
cumulative distribution function F(x):
d
f x  
F x 
dx

Since F(x)=P(X≤x), it follows that:
b

F b   F  a   P  a  X  b   f  x   dx
a
41
Cumulative Distribution Function (CDF)


The Cumulative Distribution Function (CDF) is a
function giving the probability that the random variable X
is less than or equal to x, for every value x.
Formally

the cumulative distribution function F(x) is defined to be:
    x  ,
F x   P X  x 
42
Cumulative Distribution Function (CDF)

For a discrete random variable, the cumulative distribution
function is found by summing up the probabilities as in the
example below.
    x  ,
F x   P X  x  
 P (X
 xi ) 
x i x

 p (x
i
)
x i x
For a continuous random variable, the cumulative
distribution function is the integral of its probability density
function f(x).
b

F  a   F b   P  a  X  b   f  x   dx
a
43
Cumulative Distribution Function (CDF)
► Example
► Discrete
case: Suppose a random variable X has the
following probability mass function p(xi):
xi
0 1 2
3 4 5
p(xi) 1/32 5/32 10/32 10/32 5/32 1/32
► The
cumulative distribution function F(x) is then:
xi
0 1 2 3 4 5
F(xi) 1/32 6/32 16/32 26/32 31/32 32/32
44
Mean or Expected Value
Expectation of discrete random variable X
X  E  X  
n
x
i
 p x i

i 1
Expectation of continuous random variable X
X  E  X  


x  f  x  dx

45
Example: Mean and variance

When a die is thrown, each of the possible faces 1, 2, 3, 4, 5, 6
(the xi's) has a probability of 1/6 (the p(xi)'s) of showing. The
expected value of the face showing is therefore:
µ = E(X) = (1 x 1/6) + (2 x 1/6) + (3 x 1/6) + (4 x
1/6) + (5 x 1/6) + (6 x 1/6) = 3.5

Notice that, in this case, E(X) is 3.5, which is not a possible
value of X.
46
Variance


The variance is a measure of the 'spread' of a
distribution about its average value.
Variance is symbolized by V(X) or Var(X) or σ2.


The mean is a way to describe the location of a
distribution,
the variance is a way to capture its scale or degree of
being spread out. The unit of variance is the square of the
unit of the original variable.
47
Variance

The Variance of the random variable X is defined as:
V X



2
 X
 E X  E X

2
   E X 
E X
2
2
where E(X) is the expected value of the random variable X.
The standard deviation is defined as the square
root of the variance, i.e.:
 X  X2  V  X
 s
48