Transcript Linking Data Collection to Causality

Linking Data Collection
to Causality
Collecting for a Causality
When we collect data, we have varying purposes.
Sometimes we just want to describe a population.
Other times we want to determine whether variables are
causally related.
When measuring phenomena, the timing and whom we
contact should match our objectives: Describe or
Explain.
How does timing and whom we contact (design) affect
ability to make causal statements?
Collecting for a Causality
X
Y
Independent
Variable
Dependent
Variable
Y
X
z
Z
One thing causes another when there is:
a) Association—when X and Y change in tandem
b) Time Order—for X to cause Y, value of X must occur prior to
value of Y
c) Nonspuriousness—relationship between X and Y is
not coincidental or caused by changes in a third variable (z)
Collecting for a Causality
Cross-Sectional Design
Collecting data at one point in time, using same
instruments for everyone—observing or asking
questions only during a single limited time-frame.
 Great for descriptive work.
 Effect on Causality: 1. Can establish association,
2. Time-order is hard to establish

Answers on variables such as sex and race can be assumed
to have pre-dated answers on other variables
Answers to many variables, however, do not clearly precede
answers to others
We often rely on respondents’ memories to establish time
order and this can be erroneous (why?)
Collecting for a Causality
Longitudinal Designs



Collecting data, using same instrument for
everyone, at more than one point in time—
observing or asking questions across time,
typically at discrete points.
Works for description over time.
Effect on Causality: Depends on design,
trend or fixed-sample design?
Collecting for a Causality
Longitudinal Designs

Repeated Cross-Sectional Designs or Trend Studies
New sample used to collect data at each new time point.

Political Polls, General Social Survey.
Descriptive: Can see change over time.
Effect on Causality:
1. Can establish association at distinct times. Cannot
establish association at the individual level over time points.
May establish macro level association over time points.
2. Like cross-sectional design at each time point. Cannot
establish time order at the individual level over time points.
May establish macro level time order over time points.
Collecting for a Causality
Longitudinal Designs

Fixed-Sample Panel Design or Panel Study
Same sample used to collect data at each new time point.
Descriptive: Can see change over time.
Effect on Causality:
1. Can establish association at distinct times, at the individual level
over time points, and at the macro level over time points.
2. Can establish time order at the individual level over time points
and at the macro level over time points. An independent variable’s
value at a previous time can be linked to a dependent variable’s
value at a subsequent time.
Very time-consuming, expensive.
Collecting for a Causality
Non-Spuriousness
Cross-sectional and longitudinal designs cannot
establish that associations are not spurious.
Breadth of data collection—having collected
enough of the right variables—allows one to
take into account other extraneous variables.
Can you establish nonspuriousness with your
papers’ analyses?
Collecting for a Causality
Experiments


Treating groups differently, but collecting the same
information from them.
True experiments have:
At least two comparison groups (experimental and control)
Random assignment of subjects to comparison groups.
Variation (or manipulation) in an independent variable before
assessment of outcome on the dependent variable
Independent Variable Dependent Variable
Sample
Random
Assignment
Experimental
Group
Vary a
condition, X
Control
Group
Do nothing, X
Measure Y
Compare scores
Measure Y
Collecting for a Causality
Experiments
Devised to assess causality by controlling everything possible while
allowing for a change in just one variable to see how it would affect
variables of interest in subjects.
Control is created by randomly placing persons in two or more
groups and treating them the same except…
Time-order is established by manipulating an independent variable
between groups—changing just one thing for one group but not the
other.
Association is determined by observing change in the dependent
variable after allowing only the independent variable to vary.
Non-spuriousness is determined by not allowing anything else to
vary between groups. If nothing else is changing, there is no
extraneous variable influencing those of interest.

Random assignment (NOT RANDOM SAMPLING) of persons to
comparison groups eliminates possibility of systematic variation
between groups.
Collecting for a Causality
Experiments



Sometimes, pretests are used prior to manipulation of
the independent variable.
This does not establish causality as much as it
provides a baseline allowing one to determine just
how much the dependent variable changes
and can demonstrate similarity of comparison groups
prior to manipulation.
Pre-Measure Y
Independent Variable Dependent Variable
Sample
Random
Assignment
Experimental
Group
Vary a
condition, X
Control
Group
Do nothing, X
Measure Y
Compare scores
Measure Y
Collecting for a Causality
Experiments



Sometimes, matching of subjects influences
assignment. This is so that one can guarantee
similarity along certain dimensions across comparison
groups.
If using matching alone, the design is “quasi –
experimental,” ‘quasi’ meaning “something that
appears to be something it is not”
Matching can be used with random assignment
Independent Variable Dependent Variable
Matching
Sample
&
Random
Assignment
Experimental
Group
Vary a
condition, X
Control
Group
Do nothing, X
Measure Y
Compare scores
Measure Y
Collecting for a Causality
Experiments are good when one can control and
manipulate.


Experiments are much more common in the natural
sciences
Sociologists rarely use experiments, generalizeability
for complex social phenomena is limited:
Ethical concerns lead us to observe rather than control and
manipulate (we just can’t control the way we’d have to)
Control is artificial, setting up nonrepresentative contexts
Observation changes the observed, especially among
humans
Variables of interest are more complex than can be
represented in a controlled setting
Subjects forming the sample are typically recruited, leading
to nonrepresentative samples
Collecting for a Causality
Quasi-Experiments





Quasi-experiments attempt to adapt good things about experiments to
situations where controlled experiments are impossible.
Helpful if it is impossible to randomly assign people to groups that
determine their experiences—like when studying real-world situations or
interventions
They are common in evaluation research—determining whether an
intervention is effective.
Missing typically is Random Assignment to groups.
Technically, groups should be determined prior to manipulation of the
independent variable or “intervention.”
Independent Variable Dependent Variable
Sample
Random
Assignment
Experimental
Group
Vary a
condition, X
Control
Group
Do nothing, X
Measure Y
Compare scores
Measure Y
Collecting for a Causality
Quasi-Experiments

Nonequivalent control group designs: A ;T-O ; N-S 
1. Individual Matching
Persons are assigned to different groups in “pairs” so that experimental
and control groups will be similar.
2. Aggregate Matching
Another group of persons that resembles the experimental group is
selected to act as the control group.
matching
Independent Variable Dependent Variable
Sample
Random
Assignment
Experimental
Group
Vary a
condition, X
Control
Group
Do nothing, X
Measure Y
Compare scores
Measure Y
Collecting for a Causality
Quasi-Experiments

Before-and-After designs: A
;T-O ; N-S 
1. A group acts as it’s own control. A pretest measure (the control) is
compared with a posttest measure.
The control group becomes the experimental group and is then
compared with itself.
Helpful when a control group is almost impossible to create or find,
such as when an entire organization changes procedures.
Sample
Random
Assignment
Independent Variable Dependent Variable
continue
Experimental
Vary a
Measure Y
Group
condition, X
Compare scores
Start here
Control
Measure Y
Do
nothing,
X
Group
Collecting for a Causality
Quasi-Experiments

Before-and-After designs
2. Comparing multiple groups that experience the same independent
variable manipulation improves confidence in conclusions about
causality. Repeated measurement prior to and after change in the
independent variable provides even more evidence for causality and
permits analysis of how long effects last.
Sample
Random
Assignment
Independent Variable Dependent Variable
continue
Experimental
Vary a
Measure Y
Group
condition, X
Compare scores
Start here
Control
Measure Y
Do
nothing,
X
Group
Collecting for a Causality
Nonexperiments
These lack some key element of experiments such as lacking
random assignment to groups, lacking matching prior to
manipulation of the independent variable or lacking comparison
groups.

Ex Post Facto Control Group Design —A ;T-O ; N-S 
Experimental
Group
The groups cannot be determined in advance, so there is the
possibility of extraneous factors determining group membership.
This is often necessary when studying events that have occurred or
practices that are already in place.
Vary a
Measure Y
condition, X
Compare
Find another
scores on Y
similar group.
Control
Measure Y
Do
nothing,
X
Group
Collecting for a Causality
Factorial Surveys —A ;T-O ; N-S 




Random
Sample
A research “bright spot” where researchers attempt to combine
generalizability of a random sample with random assignment to groups.
Randomly selected participants randomly get treatment or no treatment
in the survey, typically vignettes, and then dependent variable is
measured later.
Often survey methods are tested this way, with randomly selected
sample being randomly surveyed with different techniques such as with
interview, paper/pencil, or web-based.
The biggest issue is typically that only attitudes can be measured, not
particular behaviors.
Independent Variable Dependent Variable
Random
Assignment
Experimental
Group
Vary a
condition, X
Control
Group
Do nothing, X
Measure Y
Measure Y
Collecting for a Causality
A Note:

Regardless of the research method you
employ, you should be thinking in terms of:
Association
Time-order
Nonspuriousness
Collecting for a Causality
Some other things to consider, threats to determining causality and validity.
Make sure you study these.

Selection bias
Differential attrition

Endogenous Change

Testing
Maturation
Regression Effect

External Events

Contamination

Treatment Misidentification
Researcher demand
Self-fulfilling prophesies
Placebo effect
Hawthorne effect
Collecting for a Causality
In-class Group Assignment (worth 2 Q & A)



The Fantasy Island Preservation Society has offered
you a lot of money to do research. They believe that
watching Fantasy Island increases willingness to
pursue dreams.
Your job is to devise an experiment that is reasonably
feasible that will determine whether watching Fantasy
Island affects pursuit of dreams.
Due at the end of class!