Transcript Lecture 5

CRIM 483
Statistical Significance
Hypothesis Testing
• The null hypothesis assumes that a
relationship occurs simply by chance
• The research hypotheses represents the
more extreme outcome
• The significance level or probability of an
outcome determines whether the null or
the research hypothesis is a more
attractive explanation
Statistical Significance
• In a comparison of adolescent attitudes toward maternal
employment a significant difference is found between
adolescents whose mothers worked and adolescents
whose mothers did not work
• Significant means that any difference between the
attitudes of the two groups is due to some systematic
influence and not due to chance
– What is the systematic influence…having a mother who worked
vs. a mother who didn’t work
– Assumes that you have accounted for all other variables that
may affect this relationship (i.e., the relationship is non-spurious)
Understanding Significance
• Since the world is not perfect, we cannot be 100% sure that we are
not wrong when we conclude that an outcome is significant
– There will ALWAYS be a certain amount of error that cannot be
controlled
• Therefore, researchers use significance levels as a safety net
– Significance levels=the risk associated with not being 100%
confident that what you observe in a study is due to what is
begin tested
– p < .05 means that there is a 1 chance in 20 that any differences
found were not due to the hypothesized reason but due to some
other, unknown reason
• The probability of observing such an outcome is less than .05
(rare); therefore, there must be an influencing factor
Understanding Significance,
Cont’d.
• A significance level is the risk a research is
willing to take that he/she is wrong
• p < .01 = 1% risk of being wrong
• p < .05 = 5% risk of being wrong—conventional
standard
• p < .10 = 10% risk of being wrong
• Significance levels are associated with a critical value
(e.g., a z-score that represents the point at which 5% of
scores rest above the value)
• Any value above the critical value is considered
significant because any such value will be associated
with a probability that is less than .05
Interpreting Significance
• When you conclude that an outcome is
significant, you are basically saying: “This
could not have occurred by chance
alone—something else is going on.”
– The null (assumes chance) is not the most
attractive explanation
– The research hypothesis (assumes
difference/inequality) is a more attractive
explanation for the outcome
Example
• You want to test the difference in achievement between children who
went to preschool and children who didn’t.
– What is the null?
– What is the research hypothesis?
• You must do your best to account for all relevant factors, have the
best sample, utilize the best data collection methods, etc.
• If you conclude that the difference in achievement is due to
preschool attendance, you still have to accept a level of risk that you
are wrong
– If you use p<.05, then you are implying that the degree of risk that you
willing to take that you will reject a null hypothesis when it is actually
true is 5%.
• If, in reality, there is no difference, you have made a Type I error by
rejecting the null when it was actually true
Different Types of Errors
• Type I: Reject null when it is actually true
– Significance level is associated with this error
– p<.05 = There is a 5% chance that you will reject the null when it
is actually true, concluding there is a difference when there really
is none.
• Type II: Accept null when it is actually false
– Also known as a false negative
– More conservative approach
• Important to remember: We never really know the true
nature of the null hypothesis because the population is
never directly tested
– Impractical
– Use inferential statistics to infer information about the population
from a sample
Controlling Error
• Type I Error
– Use the significance level to control the likelihood that
you will make this error
• Type II Error
– More difficult to control
– Related to sample size: As the sample size increases,
the likelihood of making a Type II Error decreases
– Why? Because the larger the sample, the more
closely it matches the characteristics of a population
• The more a sample reflects the characteristics of a
population, the less likely you will accept a false null
Significance v. Meaningfulness
• Small differences can often be significant
– Group 1 (Classroom Teaching) Reading Test
Average Score=75.6
– Group 2 (Computer Learning) Reading Test
Average Score=75.7
– Difference=.1 and sig. at .05 level
• What does it mean?
Determining Meaningfulness
• Study has a strong conceptual base and
lends meaning to the significance of the
outcome
• Relative impact of making a change
reflected in the significant difference
(would an increase in .5 point warrant an
overhaul of the system)
• Null results are just as important as finding
significant results
Using Inferential Statistics
• Select representative sample
• Collect appropriate data from or on the sample
• Use appropriate statistical test to determine if
difference (outcome) is significant
• Draw a conclusion about the relationship and
infer knowledge about the population from what
is found in the sample
Using Significance Testing
•
•
•
•
•
State the null hypothesis & research hypothesis
Set the level of risk (or level of significance)
Select the appropriate test statistic
Compute the test statistic value
Determine the value needed to reject the null hypothesis using the
appropriate table of critical values for the particular statistic (i.e.,
identify the critical value for the level of risk selected)
• Compare the obtained value to the critical value
• If the obtained value is > the critical value, the null hypothesis
cannot be accepted
• If the obtained value doe not exceed the critical value, the null
hypothesis is the most attractive explanation
Selecting the Appropriate
Test Statistic