Transcript Survey Experiments

Survey Experiments
Defined
• Uses a survey question as its measurement
device
• Manipulates the content, order, format, or
other characteristics of the survey as a
treatment
Methodological Issues
• Missing Data
• Matching
• Both can be an issue in experiments other
than surveys
Missing Data
• Some observations missing data on the DV or
IVs
• If missing at random, not a problem to drop
from the analysis
• But usually not missing at random
• Deleting non-random missing causes bias
Missing Data II
• Data can also be missing intentionally:
• Some cases not “treated”
• Possible to “guess” what would have
happened to a subject had they been in
another treatment group
– Allows within-subject comparison of two
treatments, the one they received and the one
they could have received
Solution: Imputation
•
•
•
•
Suppose Yi = a + b1Xi1 + b2Xi2 +ei
But Yi missing for some observations
Xi1 and Xi2 not missing
Regress Y on Xi1 and Xi2 for all non-missing
observations
• Use b1 and b2 to calculate predicted Ypi
Better Yet: Multiple Imputation
• Ypi is a predicted value with uncertainty
• Multiple imputation predicts multiple values
for Ypi drawn from a distribution of predicted
values
• 5 or so predicted Ypi sufficient for inference,
no need for many
• Gary King’s Amelia program available free online
Matching
• Experiments can be pre-matched to avoid
large random sample
• Match subjects on important characteristics
such as
– Sex
– Race
– Age
– Education levels
– Other traits?
Matching
• Often necessary in field experiment when
randomization more difficult to control
• propensity score is the probability of an
observation being assigned to a particular
treatment in a study given a set of known
variables.
• Propensity scores reduce selection bias by
equating groups based on these variables
A Theory of Nonseparable
Preferences in Survey Responses
Question
• Why do people change their answers to
survey questions if the order of questions
changes?
• Does changing survey responses indicate that
people do not have well-formed opinions
Theory
• Nonseparable Preferences: What a person
wants on one issue depends on what she gets
on another issue
• Separable Preferences: What a person wants
on every issue is independent of what they get
on other issues
Measuring Nonseparable Preferences
Method
• Randomize the order of pairs of survey
questions
– For some issues, aggregate responses different
across question order
• Each subject answers questions in order
– Issue 1 then Issue 2
– Issue 2 then Issue 1
Method
• Impute what subject would have answered had
they heard questions in different order
• For each question we then have
Yi (if first) – Yi (if second)
• One of these will be imputed for each person
since they cannot answer a question both first
and second in the order
• First study to analyze individual differences in
question orders, not simply aggregate differences
Conclusions
• Nonseparable preferences explain question
order effects
• Political information level does not
• Response instability not due to uninformed
respondents
Are Survey Experiments
Externally Valid?
JASON BARABAS and JENNIFER JERIT
American Political Science Review
2010
Question
• Many survey experiments expose subjects to
different information to show effect of on
responses
• In a survey experiment, subjects are a “captive
audience” that must pay attention
• Do the same information effects appear in the
real world
• Compare survey experiments with natural
experiments
Method
• Survey experiments give people to political
information about immigration and medical
care
• Pre-post survey also in field during change in
medical insurance and immigration
– Ask respondents which media sources they use
• Is the effect of information in the survey
experiment as large as in the natural
experiment?