Transcript 19.nonexp - Illinois State University Department of Psychology

Non-Experimental designs:
Surveys & Quasi-Experiments
Psych 231: Research Methods in
Psychology
Announcements

Lab attendance is critical this week because
group projects are being administered
Attendance will be taken.
 Turn in the group project rating sheet 1

Non-Experimental designs

Sometimes you just can’t perform a fully
controlled experiment
– Because of the issue of interest
– Limited resources (not enough subjects,
observations are too costly, etc).
•
•
•
•
Surveys
Quasi-Experiments
Developmental designs
Small-N designs
Surveys

What are they (review chpt 7)?
– Questionnaires and interviews that ask people to
provide information about themselves

Why conduct them?
– To compliment experimental work
• Good/common first step, can collect a lot of data about a
lot of variables
– Best way to collect some kinds of information:
• Descriptive, behavioral, and preferential
– (e.g. demographic information, recreational behavior, and
attitudes)
Surveys

Advantages
– Can generalize about an entire population based
on relatively small samples of individuals
– Large amounts of data can be collected quickly
with relatively little cost (effort, time, etc.)
• But they’re often not as “cheap” as you may think
– One can investigate internal events (for example,
attitudes)
Surveys

Disadvantages
– Correlational: causal claims shouldn’t be made
– Non-response bias
• Why doesn’t everybody respond?
• Does response rate interact with variables of interest?
– Large data sets are sometimes difficult to analyze
– Self-reports may not be truthful
• Response set - tendency to respond from a particular
perspective (e.g., how a “moral” person would answer)
Stages of survey research

Stage 1) Identify the focus of the study and
select your research method
– What are the objectives of the research?
– Is a survey method the best approach?
– What kind of survey should be used?
Surveys methods

Many different methods are used to
administer surveys
–
–
–
–
–
–
Group administration (e.g. MASS testing session)
Mail surveys
Internet surveys
Telephone surveys
Face-to-face interviews
Focus group interviews
Stages of survey research

Stage 2) Determining the research schedule
and budget
 Stage 3) Establishing an information base
– Find out what’s been done, what’s known
• E.g., Find other related surveys

Stage 4) Identify the sampling frame
– The actual population that the sample is drawn
from (as opposed to the ideal population)
• Think of it as operationalizing the conceptual level
population
Stages of survey research

Stage 5) Determining the sample size and
sampling method
– Review Probability and Non-Probability methods
Voluntary response methods

A kind of convience sampling methods
commonly used
• TV uses a lot of these
– call XXX-YYYY if you support Y
– call XXX-ZZZZ if you support Z
• Problem: You typically get only individuals with
strong opinions to respond, so the results are
often extremely biased
Importance of sample size

Sampling error - how is the sample different
from the population?
– Confidence intervals
• An estimate of where the mean or percentage in the
overall population is, based on the sample data
– “John Doe has 55% of the vote, with a margin of error ±
3%”
• Margin of error (that “± 3%” part)
– Which would you be more likely to believe
» We asked 10 people …
» We asked 1000 people …
– The larger your sample size, the smaller your margin of
error will be.
Survey Questions

Stage 6) Designing the survey instrument
– Question construction: How the questions are
written is very important
• Clearly identify the research objectives
– Do your questions really target those research objectives?
• Take care wording of the questions
– Keep it simple, don’t ask two things at once, avoid loaded
or biased questions, etc.
• How should questions be answered?
Survey Questions

Question types
– Open-ended (fill in the blank, short answer)
• Can get a lot of information, but
• Coding is time intensive and potentially ambiguous
– Close-ended (pick best answer, pick all that apply)
• Easier to code
• Response alternatives are the same for everyone
– Rating scales
• Used for “how much” judgments
– e.g., Likert scale – measures attitudes, agree/disagree
• Take care with your labels
– Range of scores, anchors
Stages of survey research cont.

Stage 7) Pre-testing the survey instrument
– Fix what doesn’t seem to be working

Stage 8) Selecting and training interviewers
– For telephone and in-person surveys
– Need to avoid interviewer bias

Stage 9) Implementing the survey
 Stage 10) Coding and entering the data
 Stage 11) Analyzing the data and preparing a
final report
Error in survey research

Sampling error

Response rate
– What proportion of the sample actually
responded to the survey
• Hidden costs here - what can you do to
increase response rates
– Non-response error (bias)
• Is there something special about the data that you’re
missing? From the people who didn’t respond

Measurement error
– Are your questions really measuring what you want them
to?
Quasi-experiments

What are they?
–

Almost “true” experiments, but with an inherent
confounding variable
General types
1) An event occurs that the experimenter doesn’t
manipulate
–
Something not under the experimenter’s control
»
(e.g., flashbulb memories for traumatic events)
2) Interested in subject variables
–
high vs. low IQ, males vs. females
3) Time is used as a variable
Quasi-experiments

Advantages
– Allows applied research when experiments not
possible
– Threats to internal validity can be assessed
(sometimes)
Quasi-experiments

Disadvantages
– Threats to internal validity may exist
– Designs are more complex than traditional
experiments
– Statistical analysis can be difficult
• Most statistical analyses assume randomness
Quasi-experiments

Program evaluation
–
Research on programs that is implemented to achieve
some positive effect on a group of individuals.
–
–
e.g., does abstinence from sex program work in schools
Steps in program evaluation
–
–
–
–
–
Needs assessment - is there a problem?
Program theory assessment - does program address the
needs?
Process evaluation - does it reach the target population? Is it
being run correctly?
Outcome evaluation - are the intended outcomes being
realized?
Efficiency assessment- was it “worth” it? The the benefits
worth the costs?
Quasi-experiments

Nonequivalent control group designs
– with pretest and posttest (most common)
(think back to the second control lecture)
Non-Random
Assignment
Dependent
Variable
Measure
Independent
Variable
Experimental
group
Dependent
Variable
Measure
participants
Measure
Control
group
Measure
– But remember that the results may be compromised
because of the nonequivalent control group (review threats
to internal validity)
Quasi-experiments

Interrupted time series designs
– Observe a single group multiple times prior to and after a
treatment
Obs Obs Obs Obs Treatment Obs Obs Obs Obs
• Look for an instantaneous, permanent change

Variations of basic time series design
– Addition of a nonequivalent no-treatment control group time
series
OOOTOOO
& OOO_OOO
– Interrupted time series with removed treatment
• If treatment effect is reversible
Next time

Go to labs this week, attendance will be taken
 Non experimental designs cont.
– Read chapters 9 & 13

Reminder, journal summary 2 (Ariely, D. &
Wertenbroch, K. (2003). Procrastination, deadlines, and
performance… is coming up (due next week in
class). Please put your GA’s name on it