Transcript SEEDSM12_4final

Scientific Methods 1
‘Scientific evaluation, experimental design
& statistical methods’
COMP80131
Lecture 4: Statistical Methods-Probability
Barry & Goran
www.cs.man.ac.uk/~barry/mydocs/myCOMP80131
30 Nov 2012
COMP80131-SEEDSM12-4
1
Probability
There are two useful definitions of probability:
1. Bayesian probability:
A person’s belief in the truth of a statement S,
quantified on a scale from 0 (definitely not true)
to 1 (definitely true).
2. Experimental (or frequentist) probability:
Limit of M / N as N tends to infinity, where
M = number of times that a statement S is found to
be true if it is tested N times .
30 Nov 2012
COMP80131-SEEDSM12-4
2
Different language
• By either defn, probability P(S) is a number in range 0 to 1.
• Multiply by 100 to express as a percentage.
• Or express as odds:
e.g. ‘4 to 1 against’ means 1/5 = 0.2 = 20%.
• What do odds of ‘4 to 1 on’ mean?
• What does ‘50-50’ mean ?
30 Nov 2012
COMP80131-SEEDSM12-4
3
Calculating probability
• The 2 defns of probability are related but subtly different.
• By examining a coin, we could give ourselves good reason
for believing that tossing it just once will give an even
chance of getting heads, i.e. that
the Bayesian defn of P(S) = 0.5 where S = ‘get heads’.
• If coin is then tossed N = 100 times we would expect about
M = 50 occurrences of heads meaning that M/N  0.5.
• Increasing N to 1000 & then to 1000000 would be expected
to produce closer & closer approximations to P(S) = 0.5.
• If this does not happen, our ‘a-priori’ belief may be wrong.
• The coin may be ‘weighted’.
30 Nov 2012
COMP80131-SEEDSM12-4
4
Random process
•
•
•
•
Tossing a coin is a random process.
It generates a ‘random variable’ Heads or Tails.
Random because the outcome cannot be predicted exactly.
If 1= heads & 0 = tails we have a random binary number.
•
•
•
•
Throwing a dice generates a random integer in range 1-6.
Roulette wheel generates random integers in range 0-36.
Exams produces random numbers in range 0-100
These are random processes producing discrete variables.
• Some random processes produce continuous variables.
e.g. measuring people’s heights.
30 Nov 2012
COMP80131-SEEDSM12-4
5
Simulating random process
• MATLAB has functions that generate pseudo-random
numbers.
• ‘rand’ produces a pseudo-random number ‘uniformly
distributed’ in the range 0 to 1.
• May be considered ‘continuous’ since floating pt is very
accurate.
• Calling ‘rand’ repeatedly produces numbers evenly
distributed across the range 0 to 1.
• ‘Pseudo-random’ because if we know the algorithm used,
we can predict the numbers.
• So we pretend we do not know the algorithm.
• ‘rand’ may be considered to simulate some random process
that generates truly random numbers, uniformly distributed.
30 Nov 2012
COMP80131-SEEDSM12-4
6
Simulating coin tossing in MATLAB
for n=1:20
R = rand;
if R > 0.5, Heads(n)=1 else Heads(n) = 0;
end;
end; % of n loop
Heads
10110001110101011101 - 12 heads & 8 tails
• I changed 20 to 10,000 & got 5066 heads: P(Heads)  0.5066
• I ran it again & got 4918 heads : P(Heads)  0.4918
30 Nov 2012
COMP80131-SEEDSM12-4
7
Using an unfair coin
for n=1:20
R = rand;
if R > 0.4, Heads(n)=1 else Heads(n) = 0;
end;
end; % of n loop
Heads
00101001110101010101 - 10 heads & 10 tails
• I changed 20 to 10,000 & got 6012 heads: P(Heads)  0.6012
• When I ran it again, I got 5979 heads : P(Heads)  0.5979
30 Nov 2012
COMP80131-SEEDSM12-4
8
Estimating probability experimentally
• Cannot measure probability with 100% accuracy.
• All measurements are estimates
• They may be slightly or totally wrong.
• According to experimental defn, we must perform an expt an
infinite number of times to measure a probability.
• This is clearly impossible.
• In practice, we have to perform the experiment a finite
number of times
• (Cannot spend all our lives tossing coins)
• Accept resulting measurement as estimate of true probability.
30 Nov 2012
COMP80131-SEEDSM12-4
9
Bayesian Definition
• According to Bayesian defn of probability, a person’s
belief in the truth of a statement may be affected by one
or more assumption (hypotheses).
• “I assume it is a fair coin”
• Different people may have different beliefs.
• Can only estimate probability using information we have
available,
• Can modify this estimate later if we get new information.
30 Nov 2012
COMP80131-SEEDSM12-4
10
Conditional probability
• P(S  S1) = probability of ‘statement S’ being true given
that we know that another statement, S1, is definitely true.
• If S  ‘get heads’ we may at first believe that P(S) = 0.5.
• But what if someone tells us that the statement S1:
‘coin is weighted with heavier metal on one side’,
is true?
• Change our measurement of probability to P(S  S1).
• P(S) is then referred to as the ‘prior’ probability
• P(S  S1) is the ‘conditional’ or ‘posterior’ probability.
30 Nov 2012
COMP80131-SEEDSM12-4
11
Bayes’s Theorem
• Expresses probability of some fact ‘A’ being true when we
know that some other fact ‘B’ is true:
P( A B) 
P( B A)  P( A)
P( B)
• E.g. let A = ‘coin is fair’ & B = ‘get 12 heads out of 20’
• P(A) is ‘prior’ as it does not take into account any
information about B.
• Similarly P(B) is ‘prior’.
• P(A|B) and P(B|A) are ‘conditional’ or ‘posterior’ prob.
• Can write P(B) = P(B | A)P(A) + P(B | not A)P(not A)
30 Nov 2012
COMP80131-SEEDSM12-4
12
What is prob of getting 12 heads out of 20?
clear all;
% WITH FAIR COIN
H=zeros(21,1);
for rep=1:1000
for n=1:20
R = rand; % Unif random number between 0 & 1
if R > 0.5, Heads(n)=0; else Heads(n)=1; end;
end; % of n loop
Count = sum(Heads);
H(1+Count) = H(1+Count)+1;
end; % of rep loop
figure(1); stem(0:20,H);
30 Nov 2012
COMP80131-SEEDSM12-4
13
Histogram for 1000 trials
FAIR COIN
Frequency out of 1000 trials
200
180
160
140
120
100
80
60
40
20
0
0
30 Nov 2012
2
4
6
8
10
12
14
16
18
Number of Heads obtainable with 20 coin-tosses
COMP80131-SEEDSM12-4
20
14
Estimate of prob distribution based on 1000 trials
Estimate of probability distribution
FAIR COIN
0.2
0.18
0.16
0.14
0.12
0.1
0.08
0.06
0.04
0.02
0
0
30 Nov 2012
2
4
6
8
10
12
14
16
18
20
Number of Heads obtainable with 20 coin-tosses
COMP80131-SEEDSM12-4
15
Probability distribution (discrete)
• When there is a finite number of possible outcomes,
• For each possible outcome, gives its probability of occurring.
Probability
0.2
0.1
Outcome
1
30 Nov 2012
2
3
4
COMP80131-SEEDSM12-4
16
Probability density function(pdf)
• For an infinite number of possible outcomes,
• Real numbers, continuous over range -∞ to 
• Prob outcome lies betw P & Q:
pdf

Q
P
pdf ( x)dx
• Area under the curve.
• NB prob (P) = prob(Q) = 0
P
30 Nov 2012
Q
Possible outcome x
COMP80131-SEEDSM12-4
17
Continuous random processes
• Characterised by probability density functions (pdf)
pdf(x)
Uniform pdf: Prob of the random variable x
lying between a and b is:
1
b
x
ab
pdf(x)
 pdf ( x)dx  b  a
a
1
Gaussian (Normal) pdf with mean m & std dev .
1
pdf ( x) 
e
 2
1  x m  2
 

2  
b
Pr ob   pdf ( x)dx
a
m-
30 Nov 2012
m
m+
68%
ab
x
COMP80131-SEEDSM12-4
95.5% for m  2
99.7% for m  3
18
Continuous distributions
• More about these later.
• Back to discrete now.
30 Nov 2012
COMP80131-SEEDSM12-4
19
Probability estimate (fair coin)
• Estimated discrete probabilities:
for 0:9 heads
0 0 0 0 0.008 0.011 0.024 0.087 0.119 0.160
for 10:20 heads
0.194 0.157 0.115 0.076 0.03 0.012 0.003 0.003 0.001 0 0
• Estimated probability of getting 12 heads out of 20 with a
fair coin is 0.115.
30 Nov 2012
COMP80131-SEEDSM12-4
20
What is prob of getting 12 heads out of 20?
clear all;
%WITH 60-40 WEIGHTED COIN
H=zeros(21,1);
for rep=1:1000
for n=1:20
R = rand; % Unif random number betw 0 & 1
if R > 0.4, Heads(n)=1; else Heads(n)=0; end;
end; % of n loop
Count = sum(Heads);
H(1+Count) = H(1+Count)+1;
end; % of rep loop
figure(1); stem(0:20,H);
30 Nov 2012
COMP80131-SEEDSM12-4
21
HISTOGRAM for ‘60-40’ weighted coin
200
Frequency out of 1000 trials
180
160
140
120
100
80
60
40
20
0
0
30 Nov 2012
2
4
6
8
10
12
14
16
18
Number of Heads obtainable with 20 coin-tosses
COMP80131-SEEDSM12-4
20
22
Estimate of prob distribution based on 1000 trials
Prob distribution estimate for ‘60-40’
weighted coin
0.2
0.18
0.16
0.14
0.12
0.1
0.08
0.06
0.04
0.02
0
0
30 Nov 2012
2
4
6
8
10
12
14
16
18
Number of Heads obtainable with 20 coin-tosses
COMP80131-SEEDSM12-4
20
23
Cumulative prob distrib (discrete CDF)
• Prob of ordered output being less
than or equal to a given outcome x.
CDF
1
0.1
1
30 Nov 2012
2
3
4
Outcome x
COMP80131-SEEDSM12-4
24
Estimate Cumulative Prob Distrib
• Prob of getting between 0 and n Heads
• Easily derived from a Histogram or Prob Distribution.
CDF(1)= H(1)/1000;
for n=2:21,
CDF(n)=CDF(n-1) + H(n)/1000;
end;
figure(3); stem(0:20,CDF);
30 Nov 2012
COMP80131-SEEDSM12-4
25
Estimate of cumulative prob dist based on 1000 trials
Estimate of Cumulative Prob Dist
1
FAIR COIN
0.9
0.8
0.7
Usually an S
shaped
function
0.6
0.5
0.4
0.3
0.2
0.1
0
0
30 Nov 2012
2
4
6
8
10 12 14 16 18 20
Number of Heads obtainable with 20 coin-tosses
COMP80131-SEEDSM12-4
26
4 coin-tosses: how many possible outcomes?
0000
0001
0010
0011
0100
0101
0110
0111
1111
1001
1010
1011
1100
1101
1110
1111
30 Nov 2012
How many with 0 heads? 1
How many with 1 heads? 4 = 4C1
How many with 2 heads? 6 = 4C2 = 43/ (2!)
How many with 3 heads? 4 = 4C3
How many with 4 heads? 1
Combinations:
nCr
= no of ways of choosing r from n
= n(n-1) …(n-r+1) / (r!)
COMP80131-SEEDSM12-4
27
Binomial Prob Distribution
• Distributions have up to now been estimated.
• For random processes with just 2 outputs, we can derive
a true distribution:
• If p=prob(Heads), prob of getting Heads exactly r times
in n independent coin-tosses is:
r (1-p)(n-r)
C
p
n r
• For a fair coin. p=0.5,  this becomes nCr /2n
• For a fair dice, prob of throwing 3 sixes in five throws is:
[54/(3 2 1)] (1/6)3  (5/6)2
30 Nov 2012
COMP80131-SEEDSM12-4
28
Implementing formula (fair coin)
•
•
•
•
•
•
•
•
30 Nov 2012
p = 0.5; % for fair coin tossing
n=20;
for r=0:n
nCr = prod(n:-1:(n-r+1))/prod(1:r);
P(1+r) = nCr * (p^r) * (1-p)^(n-r);
end;
figure(4); stem(0:20,P);
axis([0 20 0 0.2]); grid on;
COMP80131-SEEDSM12-4
29
True probability of getting that no of heads
True prob distribution (n=20)
0.2
0.18
Fair coin
0.16
0.14
0.12
0.1
0.08
0.06
0.04
0.02
0
0
30 Nov 2012
2
4
6
8
10 12 14 16 18 20
No of heads obtainable with n coin-tosses
COMP80131-SEEDSM12-4
30
True probability from formula
• For 0-9 heads:
0 0 0.0002 0.0011 0.0046 0.015 0.037 0.074 0.12 0.16
• For 10-20 heads:
0.176 0.16 0.12 0.074 0.037 0.015 0.0046 0.0011 0.0002
00
• True prob of getting 12 heads with a fair coin is 0.12.
– True prob of getting ≥12 heads with fair coin is 0.252.
• Changing p to 0.4, we find that true probability of getting
12 heads out or 20 with a ‘60-40’ weighted coin is: 0.18
– True prob of getting ≥12 heads with weightd coin is 0.61.
30 Nov 2012
COMP80131-SEEDSM12-4
31
Back to Bayes’s Theorem
• There are 2 coins a fair one & a ‘60-40’ weighted one.
• Choose a coin at random & toss it 20 times.
• What is the probability of having a weighted coin when I
get 12 heads out of 20?
• A  ‘coin is weighted 60-40’ & B  ‘get 12 heads from 20’
• We know that P(B Fair coin) is 0.12 & P(B A) is 0.18.
• P(B) = P(B | not A)  P(not A) + P(B | A) P(A)
= 0.12  0.5 + 0.18 0.5 = 0.15
• P(A B) = P(B A)  P(A) / P(B)
= 0.18  0.5 /0.15 = 0.6
• If B  ‘get ≥12 heads from 20’,
P(A/B)=0.60.5/(0.6 0.5+0.252 0.5) = .3/.42 = 0.71
30 Nov 2012
COMP80131-SEEDSM12-4
32
Further illustration of Bayes Theorem
• At a college there are:
10 students from France 5 girls & 5 boys
15 from UK 5 girls & 10 boys
20 from Canada 5 girls & 15 boys
30 Nov 2012
COMP80131-SEEDSM12-4
33
Calculation
• If we choose a student at random, the a-priori probability
that this student is French is P(French) = 10/45 = 2/9  0.22
• If we notice that this student is a boy, how does this change
the probability that the student is French?
• Use Bayes’ Theorem as follows:
P( French Boy ) 
P( Boy French)  P( French)
P( Boy )
= 0.5  (10/45) / (30/45) = 1/6  0.167
• The fact that we notice that the chosen student is a boy
gives us additional information that changes the probability
that the student chosen at random will be French.
30 Nov 2012
COMP80131-SEEDSM12-4
34
Check the calculation
• We can check the previous result by common sense,
• Notice that out of 30 boys, 5 are from France.
• Therefore, P(FB) = 5/30 = 1/6.
30 Nov 2012
COMP80131-SEEDSM12-4
35
Usefulness of Bayes Theorem
• It allows us to take additional information into account
when calculating probabilities.
• Without the additional information, we have a ‘prior’
probability
• With it we have a ‘conditional’ or ‘posterior’ probability.
30 Nov 2012
COMP80131-SEEDSM12-4
36
Bayes Theorem in medicine
• A patent goes to a doctor with a bad cough & a fever.
The doctor needs to decide whether he has ‘swine flu’.
• Let statement S = ‘has bad cough and fever’ and
statement F = ‘has swine flu’.
• The doctor consults his medical books and finds that
about 40% of patients with swine-flu have these same
symptoms.
• Assuming that, currently, about 1% of the population is
suffering from swine-flu and that currently about 5%
have bad cough and fever (due to many possible causes
including swine-flu), we can apply Bayes theorem to
estimate the probability of this particular patient having
swine-flu.
30 Nov 2012
COMP80131-SEEDSM12-4
37
Another problem to solve
• A doctor in another country knows form his text-books
that for 40% of patients with swine-flu,
• The statement S, ‘has bad cough and fever’ is true.
• He sees many patients and comes to believe that the
probability that a patient with ‘bad cough and fever’
actually has swine-flu is about 0.1 or 10%.
• If there were reason to believe that, currently, about 1%
of the population have a bad cough and fever, what
percentage of the population is likely to be suffering from
swine-flu?
30 Nov 2012
COMP80131-SEEDSM12-4
38
Concept of a ‘null-hypothesis’
• A null-hypothesis is an assumption that is made and
then tested by a set of experiments designed to reveal
that it is likely to be false, if it is false.
• Testing is done by considering how probable the results
are, assuming the null hypothesis is true.
• If the results appear very improbable the researcher may
conclude that the null-hypothesis is likely to be false.
• This is usually the outcome the researcher hopes for
when he or she is trying to prove that a new technique is
likely to have some value.
30 Nov 2012
COMP80131-SEEDSM12-4
39
An example
• Assume we wish to find out if a proposed technique
designed to benefit users of a system is likely to have
any value.
• Divide the users into two groups and offer the proposed
technique to one group and something different to the
other group.
• The null-hypothesis would be that the proposed
technique offers no measurable advantage over the
other techniques.
30 Nov 2012
COMP80131-SEEDSM12-4
40
The testing
• Carried out by looking for differences between the sets
of results obtained for each of the two groups.
• Careful experimental design to eliminate differences not
caused by the techniques being compared.
• Must take a large number of users in each group &
randomize the way the users are assigned to groups.
• Once other differences have been eliminated as far as
possible, remaining difference will hopefully be indicative
of effectiveness of techniques being investigated.
• Vital question is whether they are likely to be due to the
advantages of the new technique, or the inevitable
random variations that arise from the other factors.
• Are the differences statistically significant?
• Can employ statistical significance test to find out.
30 Nov 2012
COMP80131-SEEDSM12-4
41
Failure of the experiment
• If results are not found to look improbable under the nullhypothesis,
i.e. if differences between the two groups are not
statistically significant,
then no conclusion can be made.
• Null-hypothesis could be true, or it could still be false.
• It would be a mistake to conclude that the ‘null-hypothesis’
has been proved likely to be true in this circumstance.
• It is quite possible that the results of the experiment give
insufficient evidence to make any conclusions at all.
30 Nov 2012
COMP80131-SEEDSM12-4
42
P-Value
• Probability of obtaining a test result at least as extreme as
the one that was actually observed, assuming that the null
hypothesis is true.
• Reject null-hypothesis if p-value is less than some value α
(significance level) which is often 0.05 or 0.01.
• When null-hypothesis is rejected, result is statistically
significant.
30 Nov 2012
COMP80131-SEEDSM12-4
43
Checking whether a coin is fair
• Suppose we obtain heads 14 times out of 20 flips.
• The p-value for this test result would be the probability of
a fair coin landing on heads at least 14 times out of 20 flips.
• This is:
(20C14 + 20C15+20C16+20C17+20C18+20C19+20C20) / 220 = 0.058
• This is probability that a fair coin would give a result as
extreme or more extreme than 14 heads out of 20.
30 Nov 2012
COMP80131-SEEDSM12-4
44
Significance test
• Reject null-hypothesis if p-value  α .
• If α= 0.05, rejection of null-hypothesis is at the 5%
(significance) level.
• Probability of wrongly rejecting null-hypothesis
(Type 1 error) will be equal to α.
• This is considered ‘sufficiently low’.
• In this ‘coin testing’ case, p-value > 0.05,
• Therefore observation is consistent with null-hypothesis
and we cannot reject it.
• Cannot conclude that coin is likely to be unfair.
• But we have NOT proved that coin is likely to be fair.
• 14 heads out of 20 flips can be ascribed to chance alone.
• It falls within the range of what could happen 95% of the
time with a fair coin.
30 Nov 2012
COMP80131-SEEDSM12-4
45