Transcript Slide 1

Significance, Importance, and Undetected Differences
Real Importance versus Statistical Significance
A statistically significant relationship or difference does not necessarily mean an important one.
•Whether results are statistically significant or not, it is helpful to examine a confidence interval so
that you can determine the magnitude of the effect.
•From width of the confidence interval, also learn how much uncertainty there was in sample
results.
Example : Is Aspirin Worth the Effort?
Relationship between taking aspirin and incidence of heart attack. Null (no relationship) vs
alternative (yes relationship), chi-squared (test) statistic over 25 with p-value < 0.00001.
The Magnitude of the Effect
The test statistic and p-value do not provide information about the magnitude of the effect.
Representing the Size of the Effect
Rates of heart attack: 9.4 per 1000 for aspirin group and
17.1 per 1000 for placebo group, difference < 8 people per 1000, about 1 less heart attack for every
125 who took aspirin.
Relative risk: Aspirin group had half as many heart attacks; so could cut risk almost in half.
Estimated relative risk as 0.53, with a 95% confidence interval extending from 0.42 to 0.67.
Significance, Importance, and Undetected Differences
Role of Sample Size in Width of Confidence Intervals
Precision = variation/square root of the sample size
• In a study looking at a continuous variable such as age, height or blood pressure, that variation is
expressed in terms of standard deviation:
a large sample standard deviation of say, age, means that the individuals in your study are all sorts
of ages from young to old (e.g., a study of voters);
a small sample standard deviation means that most of those in your study are a similar age (e.g., a
study of professional football players).
•So if you want to make your results more precise, you need to increase your sample size: put more
patients on a clinical trial for example
•But precision is related to the square root of sample size, so if you want to double your precision
you need to quadruple your sample size.
Significance, Importance, and Undetected Differences
Role of Sample Size in Statistical Significance
There is almost always a slight relationship between two variables, or a difference between two
groups, and if you collect enough data, you will find it.
If the sample size is too small, an important relationship or difference can go undetected.
In that case, we would say that the power of the test is too low.
Significance, Importance, and Undetected Differences
Type I, Type II errors and Power of a Test
Significance, Importance, and Undetected Differences
The Power of a Test - Calculating Sample Size
The power of a test is the probability of making the correct decision when the alternative hypothesis
is true.
Medical Testing Example: Detecting that a person has the disease when they truly have it, a true
positive.
•The typical question that scientists ask isn't, "OK, I've got 100 people here. What are my chances of
a positive result?" but, "I want to test a hypothesis. How many people will I need?"
•This sort of question is particularly important in medical research.
•In a trial of a new drug, for example, you want to have a good chance of a statistically significant
result if the drug is effective because it would be great to have another way to help sick people.
•But you can't have too large a sample size-drugs often have side effects, so you don't want to
give a new drug to lots of people if it doesn't work.
Significance, Importance, and Undetected Differences
The Power of a Test - Calculating Sample Size
A typical question asked of statisticians in medical research might be something like:
• About 20% of patients recover from a typical cold within 36 hours. We think that our new cold
treatment might increase this to 30%.
• How big a sample do we need to avoid making a type II error – failing to reject the null hypothesis
when it is false – detecting a true difference.
•As a general rule, we use the usual alpha of 5% and a power of 80%, that is, if the drug is effective,
we want a 80% chance of showing that it does indeed work.
How many patients do we need?
Significance, Importance, and Undetected Differences
If the Outcome
is:
5%
10%
15%
20%
25%
30%
35%
And we want to detect a Difference
of:
Sample Size
Required*
1%
6,391
2%
1,725
3%
821
1.5%
5,236
3%
1,388
5%
534
3%
1,882
6%
502
9%
236
2%
5,109
5%
857
10%
229
25%
3,818
5%
980
12.5%
168
3%
2,951
9%
342
15%
127
3.5%
2,337
9%
362
17.5%
98
META-ANALYSIS
SOME BENEFITS OF META-ANALYSIS
1. Detecting small or moderate relationships
2. Obtaining a more precise estimate of a relationship
3. Determining future research
4. Finding patterns across studies
CRITICISMS OF META-ANALYSIS
1. Simpson’s Paradox
2. Confounding variables
3. Subtle differences in treatments of the same name
4. The file drawer problem
5. Biased or flawed original studies
6. Statistical significance versus practical importance