Transcript H 0 - LICH

SUMMARY
Hypothesis testing
Self-engagement assesment
𝜇 = 7.8
𝜎 = 0.76
Null hypothesis
song
Null hypothesis: I assume
that populations without
and with song are same.
At the beginning of our
calculations, we
assume the null
hypothesis is true.
no song
Hypothesis testing song
• population 𝜇 = 7.8, 𝜎 = 0.76
• sample 𝑛 = 30, 𝑥 = 8.2
𝑍=
Because of such a low probability,
we interpret 8.2 as a significant
increase over 7.8 caused by
undeniable pedagogical qualities
of the 'Hypothesis testing song'.
8.2 − 7.8
= 2.85
0.76
30
corresponding probability is 0.0022
7.8 8.2
Four steps of hypothesis testing
1. Formulate the null and the alternative (this includes
one- or two-directional test) hypothesis.
2. Select the significance level α – a criterion upon which
we decide that the claim being tested is true or not.
--- COLLECT DATA --3. Compute the p-value. The p-value is the probability that
the data would be at least as extreme as those
observed, if the null hypothesis were true.
4. Compare the p-value to the α-level. If p ≤ α, the
observed effect is statistically significant, the null is
rejected, and the alternative hypothesis is valid.
One-tailed and two-tailed
one-tailed (directional) test
two-tailed (non-directional) test
Z-critical value,
what is it?
NEW STUFF
Decision errors
• Hypothesis testing is prone to misinterpretations.
• It's possible that students selected for the musical lesson
were already more engaged.
• And we wrongly attributed high engagement score to the
song.
• Of course, it's unlikely to just simply select a sample with
the mean engagement of 8.2. The probability of doing so
is 0.0022, pretty low. Thus we concluded it is unlikely.
• But it's still possible to have randomly obtained a sample
with such a mean mean.
Four possible things can happen
Decision
State of
the world
Reject H0
Retain H0
H0 true
1
3
H0 false
2
4
In which cases we made a wrong decision?
Four possible things can happen
Decision
Reject H0
State of
the world
H0 true
H0 false
Retain H0
1
4
In which cases we made a wrong decision?
Four possible things can happen
Decision
Reject H0
State of
the world
H0 true
H0 false
Retain H0
Type I error
Type II error
Type I error
• When there really is no difference between the
populations, random sampling can lead to a difference
large enough to be statistically significant.
• You reject the null, but you shouldn't.
• False positive – the person doesn't have the disease, but
the test says it does
Type II error
• When there really is a difference between the populations,
random sampling can lead to a difference small enough to
be not statistically significant.
• You do not reject the null, but you should.
• False negative - the person has the disease but the test
doesn't pick it up
• Type I and II errors are theoretical concepts. When you
analyze your data, you don't know if the populations are
identical. You only know data in your particular samples.
You will never know whether you made one of these
errors.
The trade-off
• If you set α level to a very low value, you will make few
Type I/Type II errors.
• But by reducing α level you also increase the chance of
Type II error.
Clinical trial for a novel drug
• Drug that should treat a disease for which there exists no
•
•
•
•
•
•
therapy
If the result is statistically significant, drug will me
marketed.
If the result is not statistically significant, work on the drug
will cease.
Type I error: treat future patients with ineffective drug
Type II error: cancel the development of a functional drug
for a condition that is currently not treatable.
Which error is worse?
I would say Type II error. To reduce its risk, it makes
sense to set α = 0.10 or even higher.
Harvey Motulsky, Intuitive Biostatistics
Clinical trial for a me-too drug
• Drug that should treat a disease for which there already
•
•
•
•
•
exists another therapy
Again, if the result is statistically significant, drug will me
marketed.
Again, if the result is not statistically significant, work on
the drug will cease.
Type I error: treat future patients with ineffective drug
Type II error: cancel the development of a functional drug
for a condition that can be treated adequately with
existing drugs.
Thinking scientifically (not commercially) I would minimize
the risk of Type I error (set α to a very low value).
Harvey Motulsky, Intuitive Biostatistics
Engagement example, n = 30
H0 : 𝜇 = 𝜇𝑆𝑜𝑛𝑔
HA : 𝜇 ≠ 𝜇𝑆𝑜𝑛𝑔
𝜇 = 7.8
𝜎 = 0.76
𝑛 = 30
𝑥 = 8.06
𝜇𝑆𝑜𝑛𝑔 = 7.91
Z = 1.87
Z = 0.79
𝛼 = 0.05
two-tailed test
𝜇=0
www.udacity.com – Statistics
Engagement example, n = 30
Which of these four quadrants represent the result
of our hypothesis test?
Decision
Reject H0
State of
the world
Retain H0
H0 true
H0 false
www.udacity.com – Statistics
Engagement example, n = 30
Which of these four quadrants represent the result
of our hypothesis test?
Decision
Reject H0
State of
the world
H0 true
H0 false
Retain H0
X
Engagement example, n = 50
H0 : 𝜇 = 𝜇𝑆𝑜𝑛𝑔
HA :𝜇 ≠ 𝜇𝑆𝑜𝑛𝑔
𝜇 = 7.8
𝜎 = 0.76
𝒏 = 𝟓𝟎
𝑥 = 8.06
𝜇𝑆𝑜𝑛𝑔 = 7.91
Z = 2.42
Z = 1.02
𝛼 = 0.05
two-tailed test
𝜇=0
www.udacity.com – Statistics
Engagement example, n = 50
Which of these four quadrants represent the result
of our hypothesis test?
Decision
Reject H0
State of
the world
Retain H0
H0 true
H0 false
www.udacity.com – Statistics
Engagement example, n = 50
Which of these four quadrants represent the result
of our hypothesis test?
Decision
Reject H0
State of
the world
H0 true
Retain H0
X
H0 false
www.udacity.com – Statistics
population of students that did
not attend the musical lesson
parameters are known
𝜇0
𝜎0
population of students that did
attend the musical lesson
unknown
sample
𝜇
𝜎
statistic
is known
𝑥
Test statistic
test statistic
𝑥 − 𝜇0
𝑍=𝜎
0
𝑛
Z-test
We use Z-test if we know the population
mean 𝜇0 and the population s.d. 𝜎0 .
New situation
• An average engagement score in the population of 100
students is 7.5.
• A sample of 50 students was exposed to the musical
lesson. Their engagement score became 7.72 with the
s.d. of 0.6.
• DECISION: Does a musical performance lead to the
change in the students' engagement? Answer YES/NO.
• Setup a hypothesis test, please.
Hypothesis test
• H0: 𝜇0 = 𝜇
• H1: 𝜇0 ≠ 𝜇
• In this case doing two-sided test is the only way to test the null.
You compare the sample mean of 7.72 with the population mean of
7.5. It seems that sample mean is larger than the population mean
(7.72 > 7.5), but the sample s.d. is 0.6. You can't setup the onetailed test as you can't guess the correct direction of the
relationship. Actually, you could very easily miss the correct
direction.
• 𝛼 = 0.05
Formulate the test statistic
𝑥 − 𝜇0
𝑍=𝜎
0
𝑛
population of students that did
not attend the musical lesson
𝜇0 known
𝜎0 unknown
but this is unknown!
• Instead of 𝜎0 we only know the sample s.d.
• We can use it as the point estimate of population
s.d.
• However, this will estimate s.d. for the population
exposed to the musical lesson, 𝜎0 in the above
formula is for "unperturbed" population.
• In this case, it is common to make an assumption
that both populations have the same standard
deviation.
population of students that did
attend the musical lesson
unknown
sample
𝜇
𝜎
𝑥
𝑠
t-statistic
𝑥 − 𝜇0
𝑡= 𝑠
𝑛
one sample t-test
jednovýběrový t-test
Choose a correct alternative in the following statements:
1. The larger/smaller the value of 𝑥, the strongest the
evidence that 𝜇 > 𝜇0 .
2. The larger/smaller the value of 𝑥, the strongest the
evidence that 𝜇 < 𝜇0 .
3. The further the value 𝑥 from 𝜇0 in either direction, the
stronger/weaker evidence that 𝜇 ≠ 𝜇0 .
t-distribution
One-sample t-test
𝑥 − 𝜇0
𝑡= 𝑠
𝑛
𝐻0 : 𝜇 = 𝜇0
𝐻𝐴 : 𝜇 < 𝜇0
𝜇 > 𝜇0
𝜇 ≠ 𝜇0
𝛼 level
Quiz
𝑥 − 𝜇0
𝑡= 𝑠
𝑛
• What will increase the t-statistic? Check all that apply.
1. A larger difference between 𝑥 and 𝜇0 .
2. Larger 𝑠.
3. Larger 𝑛.
4. Larger standard error.
Z-test vs. t-test
• Use Z-test if
• you know the standard deviation of the population.
• If you know the sample 𝑠 AND you have large sample size
(traditionally over 30). In addition, you assume that the population
standard deviation is the same as the sample standard deviation.
• Use t-test if
• you don't know the population standard deviation (you know only
sample standard deviation 𝑠) and have a relatively small sample
size.
• Tip: If you know only the sample standard deviation,
always use t-test.
• For two sided test and 𝛼 = 0.05, what are the critical
values at Z- and t-distributions?
Typical example of one-sample t-test
• You have to prepare 20 tubes with 30% solution od NaCl.
When you're finished, you measure the strength of 20
solutions. The mean strength is 31.5%, with the s.d. of
1.15%.
• Decide if you have 30% solution or not?
• 𝜇0 = 30%
• 𝐻0 : 𝜇 = 30%, 𝐻1 : 𝜇 ≠ 30%
• You use t-test in such a situation.
• You could use Z-test if you have a large sample (e.g., you
prepared 100 tubes), but generally it is always correct to
use t-test.
Dependent t-test for paired samples
• Two samples are dependent when the same subject
takes the test twice.
• paired t-test (párový t-test)
• This is a two-sample test, as we work with two samples.
• Examples of such situations:
• Each subject is assigned to two different conditions (e.g., use
QWERTZ keyboard and AZERTY keyboard and compare the error
rate).
• Pre-test … post-test.
• Growth over time.
Example
• 25 students attended a normal lesson. Their mean
engagement is 𝑥𝑁 = 5.08.
• The same 25 students then heard the „Hypotheses testing
song“. Their mean engagement score is 𝑥𝑆 = 7.80.
student 1
student 2
⋮
student n
𝒙𝒊 𝒚𝒊 𝑫𝒊
𝑥1 𝑦1 𝐷1
𝑥2 𝑦2 𝐷2
⋮ ⋮ ⋮
𝑥𝑛 𝑦𝑛 𝐷𝑛
song
no song
𝑥𝑖 − 𝑦𝑖
Do the hypothesis test
• Now we follow the same procedure as for the one-sample
t-test, except that we use values of differences 𝐷.
• What will be the null? 𝑛 = 25, 𝑥𝑁 = 5.08, 𝑥𝑆 = 7.8
• 𝐻0 ∶ 𝜇𝑁 = 𝜇𝑆
• But this is equivalent to stating 𝐻0 ∶ 𝜇𝑁 − 𝜇𝑆 = 0
• And the alternative?
• 𝐻0 ∶ 𝜇𝑁 ≠ 𝜇𝑆
• What is our point estimate for 𝑥𝑁 − 𝑥𝑆 ?
• 𝑥𝑁 − 𝑥𝑆 = 5.08 − 7.8 = −2.72
Do the hypothesis test
• What else do we need to calculate a t-statistic?
• Wee need the standard deviation 𝑠 of mean differences.
• We have a paired samples table, so we know each value,
and we can easily calculate 𝑠 (do not forget, you're dividing
by 𝑛 − 1!).
• Let's say it is 𝑠 = 3.69.
• The t-statistic 𝑡 =
𝑥𝑁 −𝑥𝑆
𝑠
𝑛
−2.72
= 3.69
= −3.68
25
• Do we reject the null or do we fail to reject the null at the 𝛼 =
0.05?
• Critical values for 𝑑. 𝑓. = 𝑛 − 1 = 24 for two-tailed 𝛼 = 0.05 are ±2.064.
• We reject the null.
Dependent samples
• e.g., give one person two different conditions to see how
he/she reacts. Maybe one control and one treatment or
two types of treatments.
• Advantages
• we can use fewer subjects
• cost-effective
• less time-consuming
• Disadvantages
• carry-over effects
• order may influence results
Independent samples
• Disadvantages of dependent samples become
advantages of dependent samples and vice versa.
• We need more subjects, it's generally more time consuming and
more expensive.
• No carry-over effects (each subject only gets one treatment).
• Everything else is same
• 𝐻0 ∶ 𝑥1 − 𝑥2 = 0, 𝐻1 ∶ 𝑥1 ≠ 𝑥2
• 𝑡=
𝑥1 −𝑥2
SE
• Reject 𝐻0 if 𝑝 < 𝛼, fail to reject 𝐻0 if 𝑝 > 𝛼.
Independent samples
• However, the standard error changes because it is based on
two sample sizes and two standard deviations.
• If we subtract normally distributed data from another normally
distributed data, we get a new data set
𝑁 𝜇1 , 𝜎1 − 𝑁 𝜇2 , 𝜎2 = 𝑁 𝜇1 − 𝜇2 , 𝜎12 + 𝜎22
• Similarly, for the sample:
𝑠. 𝑑. =
𝑠12 + 𝑠22
This is true only if two
samples are independent!
• standard error
𝑠. 𝑑.
=
𝑛
𝑠12 + 𝑠22
=
𝑛
𝑠12 + 𝑠22
=
𝑛
𝑠12 𝑠22
+
𝑛
𝑛
Independent samples
• However, the standard error changes because it is based on
two sample sizes and two standard deviations.
• If we subtract normally distributed data from another normally
distributed data, we get a new data set
𝑁 𝜇1 , 𝜎1 − 𝑁 𝜇2 , 𝜎2 = 𝑁 𝜇1 − 𝜇2 , 𝜎12 + 𝜎22
• Similarly, for the sample:
𝑠. 𝑑. =
𝑠12 + 𝑠22
• standard error
𝑠. 𝑑.
=
𝑛
𝑠12 + 𝑠22
=
𝑛
𝑠12 + 𝑠22
=
𝑛
𝑠12
𝑠22
+
𝑛1
𝑛2
An example
• Again, the musical lesson.
• Let's teach nN = 10 students without the musical
performance, and expose different n𝑆 = 20 students to the
song.
• What will be the null and the alternative?
• 𝐻0 : 𝜇𝑁 = 𝜇𝑆 , 𝐻𝐴 : 𝜇𝑁 ≠ 𝜇𝑆
• Which direction will we use?
• two-tailed
An example
• 𝑛𝑁 = 10, 𝑛𝑆 = 20
• 𝑥𝑁 = 5.08, 𝑠𝑁 = 2.65
• 𝑥𝑆 = 7.80, 𝑠𝑆 = 2.18
• Standard error
𝑆𝐸 =
𝑠𝑁2 𝑠𝑆2
+
=
𝑛𝑁 𝑛𝑆
2.652 2.182
+
= 0.97
10
20
• Calculate t-statistic
𝑥𝑁 − 𝑥𝑆 5.08 − 7.80
𝑡=
=
= −2.80
𝑆𝐸
0.97
• How will you proceed further?
• calculate d.f., define 𝛼, find the critical t-value, compare the t-
statistic with the t-critical, decide about the null
An example
• 𝑑. 𝑓. = 10 + 20 − 2 = 28
• t-critical value for 𝛼 = 0.05 is ±2.048
• Reject or fail to reject the null?
• Reject the null.
Summary of t-tests
• one-sample test (jednovýběrový test)
• you test H0 : 𝜇 = 𝜇0
• two-sample test (dvouvýběrový test)
• you test H0 : 𝜇1 − 𝜇2 = 0
• dependent samples
• paired t-test (párový test)
• independent samples
• equal variances 𝜎1 ~𝜎2
• unequal variances 𝜎1 ≠ 𝜎2
two-sample tests
F-test of equality of variances
• How to know if our variances are equal or not?
• var.test() in R, 𝐻0 : 𝜎1 = 𝜎2
• Test statistic is a ratio of two variances. It has an F-
distribution. Each numerator and denominator has certain
number of d.f.
source: Wikipedia
t-test in R
• t.test()
• Let's have a look into R manual:
http://stat.ethz.ch/R-manual/R-patched/library/stats/html/t.test.html
• See my website for link to pdf explaining various t-test in
R (with examples).
Assumptions
1. Unpaired t-tests are highly sensitive to the violation of
the independence assumption.
2. Populations samples come from should be
approximately normal.
• This is less important for large sample sizes.
• What to do if these assumptions are not fullfilled
1. Use paired t-test
2. Let's see further
Check for normality – histogram
Check for normality – QQ-plot
qqnorm(rivers)
qqline(rivers)
Check for normality – tests
• The graphical methods for checking data normality still
leave much to your own interpretation. If you show any of
these plots to ten different statisticians, you can get ten
different answers.
• H0: Data follow a normal distribution.
• Shapiro-Wilk test
• shapiro.test(rivers):
Shapiro-Wilk normality test
data: rivers
W = 0.6666, p-value < 2.2e-16
Nonparametric statistics
• Small samples from considerably non-normal
distributions.
• non-parametric tests
• No assumption about the shape of the distribution.
• No assumption about the parameters of the distribution (thus they
are called non-parametric).
• Simple to do, however their theory is extremely
complicated. Of course, we won't cover it at all.
• However, they are less accurate than their parametric
counterparts.
• So if your data fullfill the assumptions about normality, use
paramatric tests (t-test, F-test).
Nonparametric tests
• If the normality assumption of the t-test is violated, and
the sample sizes are too small, then its nonparametric
alternative should be used.
• The nonparametric alternative of t-test is Wilcoxon test.
• wilcox.test()
• http://stat.ethz.ch/R-manual/R-patched/library/stats/html/wilcox.test.html