Transcript Theories - the Department of Psychology at Illinois State University

Non-Experimental designs:
Surveys & Quasi-Experiments
Psych 231: Research Methods in
Psychology


Lab attendance is critical this week because
group projects are being administered
 Attendance will be taken.
Turn in the group project rating sheet 1 after
you have run your experiments (Monday is
okay)
Announcements

Sometimes you just can’t perform a fully controlled
experiment


Because of the issue of interest
Limited resources (not enough subjects, observations are too
costly, etc).
•
•
•
•

Surveys
Quasi-Experiments
Developmental designs
Small-N designs
This does NOT imply that they are bad designs

Just remember the advantages and disadvantages of each
Non-Experimental designs

What are they?


Questionnaires and interviews that ask people to
provide information about themselves
Why conduct them?

To compliment experimental work
• Good/common first step, can collect a lot of data about a
lot of variables
• Do not have to directly observe behaviors

Best way to collect some kinds of information:
• Descriptive, behavioral, and preferential
• e.g., demographic information, recreational behavior, and
attitudes
Surveys

Advantages



One can investigate internal events (for example,
attitudes & opinions)
Can generalize about an entire population based
on relatively small samples of individuals
Large amounts of data can be collected quickly
with relatively little cost (effort, time, etc.)
• But they’re often not as “cheap” as you may think
Surveys

Disadvantages


Correlational: causal claims shouldn’t be made
Non-response bias
• Why doesn’t everybody respond?
• Does response rate interact with variables of interest?


Large data sets are sometimes difficult to analyze
Self-reports may not be truthful
• Response set - tendency to respond from a particular
perspective (e.g., how a “moral” person would answer)
Surveys

Stage 1) Identify the focus of the study and
select your research method



What are the objectives of the research?
Is a survey method the best approach?
What kind of survey should be used?
•
•
•
•
•
•
Group administration
Mail surveys
Internet surveys
Telephone surveys
Face-to-face interviews
Focus group interviews
Stages of survey research


Stage 2) Determining the research schedule
and budget
Stage 3) Establishing an information base

Find out what’s been done, what’s known
• E.g., Find other related surveys

Stage 4) Identify the sampling frame

The actual population that the sample is drawn
from (as opposed to the ideal population)
• Think of it as operationalizing the conceptual level
population
Stages of survey research cont.

Stage 5) Determining the sample method
and sampling size

Review Probability and Non-Probability methods
• Voluntary response method

Importance of sample size
Stages of survey research cont.
A kind of convenience sampling methods
commonly used
Should leftover Halloween candy be given out
to students who get an A on the exam?

Call 123-4YES if you think YES
Call 123-NONO if you think NO
• Problem: Typically only individuals with strong
opinions respond, so the results are often
extremely biased
Voluntary response methods

Sampling error - how is the sample different
from the population?

Confidence intervals
• An estimate of the mean or percentage of the population,
based on the sample data
• “John Doe has 55% of the vote, with a margin of error ±
3%”
• Margin of error (that “± 3%” part)
• The larger your sample size, the smaller your margin of
error will be.
• Which would you be more likely to believe
• “We asked 10 people …”
• “We asked 1000 people …”
Importance of sample size

Sampling error - how is the sample different
from the population?

Response rate
• What proportion of the sample actually responded to
the survey?
• Hidden costs here - what can you do to increase
response rates
• Non-response error (bias)
• Is there something special about the data that you’re
missing (From the people who didn’t respond)?
Importance of sample size

Stage 6) Designing the survey instrument

Question construction: How the questions are
written is very important
• Clearly identify the research objectives
• Do your questions really target those research
objectives (think Internal and External Validity)?
• Take care wording of the questions
• Keep it simple, don’t ask two things at once, avoid
loaded or biased questions, etc.
• How should questions be answered (question type)?
Stages of survey research cont.
Poor
Was the FDC negligent
by ignoring the warnings
about Vioxx during testing
and approving it for sale?
Problem: emotionally
charged words
a) Yes
b) No
c) Unsure
Good
If the FDC knew that
Vioxx caused serious
side effects during testing,
what should it have done?
a) Ban it from ever being
sold
b) Require more testing
before approving it
c) Unsure
Good and poor questions
Poor
Good
Are you against same sex
marriage and in favor of a
constitutional amendment
to ban it?
What is your view on
same sex marriage?
Problem:
a)
Yes Biased in
more
b) than
No one direction
c) Unsure
Problem: Asks two
questions
a) I think marriage is a
matter of personal
choice
b) I’m against it but don’t
want a constitutional
amendment
c) I want a constitutional
amendment banning it
Good and poor questions

Question types

Open-ended (fill in the blank, short answer)
• Can get a lot of information, but
• Coding is time intensive and potentially ambiguous

Close-ended (pick best answer, pick all that apply)
• Easier to code
• Response alternatives are the same for everyone

Rating scales
• Used for “how much” judgments
• e.g., measures attitudes, agree/disagree
• Take care with your labels
• Range of scores, anchors
Survey Questions

Stage 7) Pre-testing the survey instrument


Stage 8) Selecting and training interviewers





Fix what doesn’t seem to be working
For telephone and in-person surveys
Need to avoid interviewer bias
Stage 9) Implementing the survey
Stage 10) Coding and entering the data
Stage 11) Analyzing the data and preparing a
final report
Stages of survey research cont.
What are they?



Almost “true” experiments, but with an inherent
confounding variable
General types
1) An event occurs that the experimenter doesn’t
manipulate
•
Something not under the experimenter’s control
•
(e.g., flashbulb memories for traumatic events)
2) Interested in subject variables
–
high vs. low IQ, males vs. females
3) Time is used as a variable
Quasi-experiments

Advantages



Allows applied research when experiments not
possible
Threats to internal validity can be assessed
(sometimes)
Disadvantages



Threats to internal validity may exist
Designs are more complex than traditional
experiments
Statistical analysis can be difficult
• Most statistical analyses assume randomness
Quasi-experiments
Program evaluation

–
Research on programs that is implemented to achieve
some positive effect on a group of individuals.
–
–
e.g., does abstinence from sex program work in schools
Steps in program evaluation
–
–
–
–
–
Needs assessment - is there a problem?
Program theory assessment - does program address the
needs?
Process evaluation - does it reach the target population? Is it
being run correctly?
Outcome evaluation - are the intended outcomes being
realized?
Efficiency assessment- was it “worth” it? The the benefits
worth the costs?
Quasi-experiments

Nonequivalent control group designs

with pretest and posttest (most common)
(think back to the second control lecture)
Independent
Non-Random Dependent
Variable
Variable
Assignment
Measure
Experimental
group
Dependent
Variable
Measure
participants
Measure
Control
group
Measure
– But remember that the results may be compromised
because of the nonequivalent control group (review threats
to internal validity)
Quasi-experiments

Interrupted time series designs

Observe a single group multiple times prior to and after a
treatment
Obs Obs Obs Obs Treatment Obs Obs Obs Obs
• Look for an instantaneous, permanent change

Variations of basic time series design

Addition of a nonequivalent no-treatment control group time
series
OOOTOOO

& OOO_OOO
Interrupted time series with removed treatment
• If treatment effect is reversible
Quasi-experiments
Go to labs this week, attendance will be
taken
 Non experimental designs cont.
 Reminder, journal summary 2 assignment
is coming up

Next time