Transcript Chap 4 An Overview of E

Slides to accompany Weathington,
Cunningham & Pittenger (2010),
Chapter 4: An Overview of Empirical
Methods
1
Objectives
• Internal, statistical conclusion, and external
validity
• Empirical methods
• Intact groups and quasi-experimental designs
• Surveys
• Correlational studies
• Single-N methods
• Meta-analysis
2
Internal Validity
• Shown by the degree to which a study
rules out alt. explanations for IV  DV
• Requires ruling out alternative
explanations
• Threats include sources of confounding
variables
– 4 general categories
3
Threats to Internal Validity
• Unintended sequence of events
– Carryover effects: drug at Time 1 hurts
performance at Time 2 (but the drug is not
what we wanted to test)
– Maturation: Changes in answers between 6
and 10 year olds may be due to normal
learning rather than a reading intervention
– Intervening events: being burglarized may
change your response to a social psychology
experiment involving eye witnesses
4
Threats to Internal Validity
• Nonequivalent groups
– Confounds interpretation of cause and
effect between IV and DV
– Can be caused by:
• Non-random sampling
• Mortality/attrition
• Subject characteristics (variables)
5
Threats to Internal Validity
• Measurement errors
– Non-valid test
– Low reliability of measurement
– Ceiling and floor effects
– Regression to the mean
• Ambiguity of cause and effect
– Which came first, X or Y?
6
Statistical Conclusion Validity
• Were the proper statistical or analytical
methods used when studying the data?
• “Proper” = best allowing the researcher to:
– Demonstrate relationship between IV and DV
– Identify the strength of this relationship
7
Threats to Statistical Conclusion Validity
• Low statistical power: increases risk of
missing an effect that really exists
• Violating assumptions of tests: no
statistical tests are perfect in all research
situations; you need to know your “tools”
• Unreliability in measurement and
setting: inconsistencies in the measurement
process make it impossible for you to draw
valid inferences from the statistics
8
External Validity
• Do our findings/results generalize beyond
our sample?
–More likely if representative sample
• Can we generalize our findings to the
population?
• Can we generalize our conclusions from
one population to another?
9
Internal vs. External Validity
Data
Interpreting the
data for cause
and effect
INTERNAL
VALIDITY
Generality of
findings
Generality of
conclusions
Population
EXTERNAL
VALIDITY
10
Threats to External Validity
• NOT always just the “lab setting”
• Participant recruitment
–How + who you select to study matters
–Need to be as representative as possible
• May require replication, extension
studies
11
Threats to External Validity
• Situation effects
– Where you do the study matters
– Control for what you can and consider
replicating in different settings
• History effects
– Be aware that phenomena may change over
time
12
True Experiment
• Best method for testing cause and effect
• “Easiest” control for internal validity threats
• Not always a practical/ethical option
• You know it is a true experiment if:
1. The IV can be controlled/manipulated
2. Random assignment to conditions occurs
3. Control conditions can be created
13
True Experiment
Sampling frame
Random assignment
Group 1
n = 10
Treatment for
Group 1
Results for
Group 1
Group 2
n = 10
Treatment for
Group 2
Results for
Group 2
Group 3
n = 10
Treatment for
Group 3
Results for
Group 2
Nonrandom differences among the groups in terms of
the measured DV leads us to conclude that the
manipulations of the IV may have caused those
differences
Assuming random
assignment into
groups, differences
among the groups at
this stage are due to
random effects
Separate conditions
controlled by the
researcher (different
levels of IV)
Differences among
groups due to
random effects +
effect of treatment
(level of IV)
14
Intact Groups Design
• No random assignment possible
• Multiple samples (by subject variables), from
multiple populations
• Cannot establish cause and effect
– Unknown 3rd variable and temporal order
• Can compare differences across samples
Independent
variable
Dependent
variable
“Third”
variable
Independent
variable
Dependent
variable
15
Intact Groups Design
Population
1
Population
2
Population
3
Group 1
n = 10
Group 1
n = 10
Group 1
n = 10
Results for
Group 1
Results for
Group 2
Results for
Group 2
Groups formed by
randomly selecting
members of each
population into one
of the three
treatment groups
Differences among
groups due to
random effects +
effect of population
(group membership)
16
Quasi-Experimental Design
• No random assignment; grouping by some
other factor
• An IV is manipulated
• One group is treated as a “control”, while the
other is exposed to the manipulated IV
• Still problem with unknown 3rd variable and
temporal order
17
Quasi-experimental Design
Group 1
Baseline
Measurement
Treatment
Measurement
Group 2
Baseline
Measurement
No treatment
Measurement
18
How does the true experiment differ
from the intact groups and quasiexperiment design?
19
Surveys
• For estimating population parameters
• Good for large-scale data collection
–Quick and inexpensive
• “Bad” because of respondent error
–Honesty and personal bias
20
Correlational Study
• Usually to estimate population parameters
• Often data from surveys
• Good for initial understanding and
“prediction” of complex behaviors
• Bad at supporting cause and effect
– Unknown 3rd variable
– Temporal order issues
21
Single-N Methods
• Sometimes better to focus in-depth on one or
a few participants
– Single-participant experiment
– Case study
• Good if IV and situational variables are wellcontrolled
• Bad for generalizability (potentially) and also
because of participant bias/error
22
Meta-analysis
• Analysis of multiple outcomes from multiple
studies
• Good because takes advantage of more
representative sampling of participants and
measures/methods
• Bad because depends on which studies are
entered
– Principle of GI, GO
23
What is Next?
• **instructor to provide details
24