Transcript 10.3 Statistical Significance

Significance Toolbox
1)
2)
Identify the population of interest (What is the topic of discussion?)
and parameter (mean, standard deviation, probability) you want to
draw conclusions about. State the null and alternative
hypotheses.
Choose the appropriate inference procedure (type of test) and
verify conditions (what kind of information is given about
population/sample. Is there an SRS? If not, we may be able to perform the
test because of the finite number of observations – central limit theorem but
generalizations may not be necessarily true especially if the distribution is
severely nonnormal).
3)
If the conditions are met, carry out the inference procedure
(find the mean, deviation, and P-value).
4)
Interpret your results
(Is the information statistically significant?).
One-sample z statistic


H0: µ = µ0
The test uses
z
x  0


x  0
x
n



Ha: µ > µ0 is P(Z ≥ z)
Ha: µ < µ0 is P(Z ≤ z)
Ha: µ ≠ µ0 is 2P(Z ≥ |z|)
View graphs on page
573.
Fixed Significant Level for Z tests
for Population Mean
The outcome of a test is significant at level
alpha if P-value ≤  .
 Once we have computed the z test statistic,
reject H0 at significant level  against a
one sided alternative when Ha: µ>µ0 if
z ≥ z* and Ha: µ < µ0 if z ≤ - z*
Reject H0 at significant level alpha against a
two-sided alternative Ha: µ≠µ0 if |z| ≥ z*

10.3
Making Sense of Statistical Significance
Choosing a level, 


Standard: the level of significance gives a
clear statement of the degree of evidence
provided by the sample against the null
hypothesis.
Best practice: Decide on a significance level
prior to testing. If the result satisfies the
level, reject the null. If the result fails the
level, find the null acceptable (fail to reject).

If we have a fixed significance level,
we
should ask how much evidence is required to
reject H0.
 If H0 represents an assumption people have
believed for years, strong evidence (small  )
is needed.
 If rejecting H0 for Ha means making
expensive changeover (products), strong
evidence must show sales will soar.

Significant vs Insignificant



There is no sharp border between significant
and insignificant only increasingly strong
evidence as the P-value decreases.
When a null hypothesis can be rejected (5%
or 1% level), there is good evidence that an
effect is present.
To keep statistical significance in its place,
pay close attention to the actual data and the
P-value.
Statistical inference is not always valid



Surveys and experiments that are designed
badly will produce invalid results.
Outliers in the data and testing a hypothesis
on the same data that suggested the
hypothesis invalidates the test.
Since tests of significance and confidence
intervals are based on the laws of probability,
randomization in sampling or experimentation
ensures these laws apply.
Assignment

Exercises 10.44, 10.57, 10.58, 10.62,
10.64
10.4 Inference as Decision

Reminders:



Tests of significances assess the strength of
evidence ______ (for/against) the null hypothesis.
Measurement: P-value which is the probability
computed under the assumption that null
hypothesis is ______ (true/false).
The alternative hypothesis helps us to see what
outcomes count ______ (for/against) the null
hypothesis.
Strength  Decision

A significance level chosen in advance points
to the outcome of the test as a decision.



If the result is significant, we reject the null
hypothesis in favor of the alternate.
If the result is not significant, we fail to reject the
null (null hypothesis is acceptable).
Making the decision to either fail to reject
(acceptable) or reject results should be left to
the user but at times the final decision is
stated during the interpretation.
Acceptance Sampling


A decision or action must be made as
an end result of inference.
Failing to reject (Acceptable) or
rejecting the end product.
Type I and II Errors
In tests of significance
-
H0 - the null hypothesis
Ha - the alternative hypothesis
However, when dealing with Type I and Type II
errors, these hypotheses will represent accepting
one decision and rejecting the other.
-
Now …
-
H0 should be considered the initial hypothesis
Ha the secondary hypothesis
Type I Error
We have been calculating this type of
error all along.
If we reject H0 (acceptable Ha) when in
fact H0 is true.
Type II Error
If we find that the H0 is acceptable
(reject Ha) when in fact Ha is true.
Quick Comparison
Decision based on sample
Truth about the Population
H0 True
Ha True
Reject H0
Type 1 Error
Correct
Decision
Fail to reject H0
Correct
Decision
Type 2 Error
(acceptable)
Cancer Scenario
Ho: “We suspect
that you have
cancer”
Reject the null: “You
don’t have cancer!”
Fail to reject the null:
“You have cancer!”
Ho is True
Ha is True
Diagnosis: Cancer
Diagnosis: No Cancer
Type I Error
Correct Decision
Diagnosis: Cancer
Diagnosis: No Cancer
Correct Decision
Type II Error
Significance and Type I Error


The significance level of any fixed level
test is the probability of a Type I error. 
is the probability that the test will reject
the null hypothesis H0 when H0 is in fact
true.
Example 10.68, Page 598
Example 10.68
You have an SRS of size n = 9 from a
normal distribution with  = 1. You wish
to test
H0: µ = 0
Ha: µ > 0.
You decide to reject H0 if x > 0 and to
accept H0 otherwise.
Power




A significance test measures the ability to detect an
alternative hypothesis.
The power against a specific alternative is the
probability that the test will reject H0 when the
alternative is true.
Calculate the power of a specific alternative:
subtract the probability of the Type II error for the
alternative from 1.
Class example 10.68. Accept that the mean, H0, will
be less than or equal to 0 18.4% of the time;
however, the mean should be greater than 0 81.6%
of the time (100% - 18.4%).
Power


continued
Power works best for fixed significance
levels.
Larger sample sizes will increase the
power for a fixed significant level.
Increase the Power




If the strength of evidence required for
rejection is too low, increase the
significance level.
Consider an alternative farther away
from µ0.
Increase the sample size.
Decrease the standard deviation, .
Assignment
(Work due on Monday, 3/28)
Exercises 10.67, 10.69, 10.71 and 10.81
Scenarios
OJ Simpson – Guilty man goes free.
Ho: OJ is innocent
Ha: OJ is guilty
Found not guilty – Type 2
Guilty man goes free.
Movie: A Time to Kill –
Ho: Father is innocent
Ha: Father is guilty
Found to be innocent –Type 2
To Kill a Mockingbird – Send an innocent man to jail
Ho: Tom Robinson is innocent
Ha: Tom Robinson is guilty
Found guilty – Type 1
The Green Mile