Transcript Chapter 10.3

Chapter 10.3-10.4
Making Sense of
Statistical Significance
& Inference as Decision
Choosing a Level of Significance


“Making a decision” … the choice of alpha depends on:
Plausibility of H0:


How entrenched or long-standing is the current belief. If it is
strongly believed, then strong evidence (small ) will be
needed. Subjectivity involved.
Consequences of rejecting H0 :

Expensive changeover as a result of rejecting H0? Subjectivity!

No sharp border – only increasingly strong evidence

P-Value of 0.049 vs. 0.051 at the ,0.05 alpha-level? No
real practical difference.
Statistical vs. Practical Significance








Even when we reject the Null Hypothesis – and
claim – “There is an effect present”
But how big or small is the “effect”?
Is a slight improvement a “big enough deal”?
Statistical significance is not the same thing as
practical significance.
Pay attention to the P-Value!
Look out for outliers
Blind application of Significance Tests is not good
A Confidence Interval can also show the size of the
effect
When is it not valid for all data?
Badly designed experiments and surveys often
produce invalid results.
 Randomization is paramount!
 Is the data from a normal population
distribution? Is the sample big enough to insure
a normal sampling distribution? (allows you to be

able to generalize/infer about the population)
Is the population greater than ten times the
sample? (affects sample st. dev.)
 Individuals in the sample are independent.

HAWTHORNE EFFECT




Does background music cause an increase in
productivity?
After discussing the study with workers - a
significant increase in productivity occurred
Problems: No control … and the idea of being
studied
Any change would have produced similar effects
Beware the Multiple Analyses
If you test long enough … you will
eventually find significance by random
chance.
 Do not go on a “witch-hunt” … looking for
variables that already stand out … then
perform the Test of Significance on that.
 Exploratory searching is OK … but then
design a study.

ACCEPTANCE SAMPLING

A decision MUST be made at the end of an
inference study:
 Fail
to Reject the lot (“accept?”)
 Reject the lot
H0: the batch of potato chips meets standards
 Ha: the potato chips do not meet standards
 We hope our decision is correct, but …we
could accept a bad batch, or we could reject a
good one. (both are mistakes/errors)

TYPE I AND TYPE II ERRORS

If we reject H0 (accept Ha) when in fact H0
is true, this is a Type I error. (α - alpha)

If we reject Ha (accept H0) when in fact Ha
is true, this is a Type II error. (β - beta)
EXAMPLE 10.21
ARE THE POTATO CHIPS TOO SALTY?





Mean salt content is supposed to be 2.0mg
The content varies normally with  = .1 mg
n = 50 chips are taken by inspector and tests
each chip
The entire batch is rejected if the mean salt
content of the 50 chips is significantly different
from 2mg at the 5% level
Hypotheses? z* values? Draw a picture with
acceptance and rejection regions shaded.
EXAMPLE 10.21
ARE THE POTATO CHIPS TOO SALTY?
What if we actually have a batch where
the true mean is μ = 2.05mg?
 There is a good chance that we will reject
this batch, but what if we don’t! What if we
accept the H0 and fail to reject the “out of
spec … bad” batch?
 This would be an example of a Type II
error …accepting μ = 2 when in reality μ =
2.05

EXAMPLE 10.21
ARE THE POTATO CHIPS TOO SALTY?


Finding the probability of a Type II error
Step 1 … find the interval if acceptance for sample
means, assuming the μ = μ0 = 2.
 2
1.96(0.1)
… (1.9723, 2.0277)
50

Now find the probability that this interval/region would
contain a sample mean about μa = 2.05

Standardize each endpoint of the interval relative to μa
= 2.05 and find the area of the alternative distribution
that overlaps the H0 distribution acceptance interval.
EXAMPLE 10.21
ARE THE POTATO CHIPS TOO SALTY?
So …  = 0.0571 … a Type II Error … we are likely to (in error)
accept almost 6% of batches too salty at the 2.05mg level
And …  = 0.05 … a Type I Error … we are likely to (in error)
reject 5% of salty batches at the perfect 2mg level
SIGNIFICANCE AND TYPE I ERROR
 The
significance level alpha of
any fixed number is the
probability of a Type I error.
That is, the probability  that the
test will reject H0 when H0 is
nevertheless true.
POWER
probability that a fixed level 
significance test will reject H0 when a
particular Ha is in fact true is called the
power of the test against the alternative.
 The
 The
power of a test is 1 minus the
Probability of a Type II error for that
alternative …
 Power
=1 - 
INCREASING POWER

Increase alpha () …  and  “work at
odds” of each other

Consider an alternative (Ha) farther away

Increase sample size (n)

Decrease sigma ()