Transcript 19.nonexp - Illinois State University Department of Psychology

Non-Experimental Designs:
Surveys
Psych 231: Research
Methods in Psychology


I will aim to have the exams graded and entered into
ReggieNet by Monday
Remember that piloting is happening in labs this week
Announcements

Sometimes you just can’t (or don’t want to) perform a
fully controlled experiment



Because of the issue of interest
Limited resources (not enough subjects, observations are too
costly, etc.).
There are other kinds of designs that may be used
• Surveys
• Correlational studies
• Quasi-Experiments
• Developmental designs
• Small-N designs

This does NOT imply that they are bad designs

Just remember the advantages and disadvantages of each
Non-Experimental designs

Sometimes you just can’t (or don’t want to) perform a
fully controlled experiment



Because of the issue of interest
Limited resources (not enough subjects, observations are too
costly, etc.).
There are other kinds of designs that may be used
• Surveys
• Correlational studies
• Quasi-Experiments
• Developmental designs
• Small-N designs

This does NOT imply that they are bad designs

Just remember the advantages and disadvantages of each
Non-Experimental designs

What are they?


Questionnaires and interviews that ask people to
provide information about themselves
Why conduct them?

Best way to collect some kinds of information:
• Descriptive, behavioral, and preferential
• e.g., demographic information, recreational behavior,
personal preferences, and attitudes

To compliment experimental work
• Good/common first step, can collect a lot of data about a
lot of variables
• Do not have to directly observe behaviors

Surveys
A good resource:
http://www.people-press.org/

Check out the Methodology Pages

Advantages

One can investigate internal events
• e.g., attitudes & opinions


Can generalize about an entire population based
on relatively small samples of individuals
Large amounts of data can be collected quickly
with relatively little cost (effort, time, etc.)
• But they’re often not as “cheap” as you may think
Surveys

Disadvantages

Correlational: causal claims shouldn’t be made
• Often used for descriptive and predictive research
questions
• Note: surveys are sometimes used as part of an experimental
design, which may allow testing of causal claims

Non-response bias
• Why doesn’t everybody respond?
• Does response rate interact with variables of interest?

Self-reports may not be truthful
• Response set - tendency to respond from a particular
perspective
• Social desirability bias (e.g., how a “moral” person would answer)

Large data sets are sometimes difficult to analyze
Surveys

Stage 1) Identify the focus of the study and
select your research method



What are the objectives of the research?
Is a survey method the best approach?
What kind of survey should be used?
•
•
•
•
•
•
Group administration
Mail surveys
Internet surveys
Telephone surveys
Face-to-face interviews
Focus group interviews
11 Stages of survey research


Stage 2) Determining the research schedule
and budget
Stage 3) Establishing an information base

Find out what’s been done, what’s known
• E.g., Find other related surveys

Stage 4) Identify the sampling frame

The actual population that the sample is drawn
from (as opposed to the ideal population)
• Think of it as operationalizing the conceptual level
population
• Be aware of potential coverage error – when the
sampling doesn’t lead to a good representativeness
11 Stages of survey research

Stage 5) Determining the sample method
and sampling size

Review Probability and Non-Probability methods
• Voluntary response method

Importance of sample size
11 Stages of survey research
A kind of convenience sampling methods
commonly used
Should leftover Halloween candy be given out
to students who get an A on the exam?

Call 123-4YES if you think YES
Call 123-NONO if you think NO
• Problem: Typically only individuals with strong
opinions respond, so the results are often
extremely biased
Daily show clip
(5:30 mins)
Voluntary response methods

Sampling error - how is the sample different
from the population?
Often focus on this part
But this part is important too

Confidence intervals
• An estimate of the mean or percentage of the population,
based on the sample data
• “John Doe has 55% of the vote, with a margin of error ± 3%”
• Margin of error (that “± 3%” part)
• The larger your sample size, the smaller your margin
of error will be.
• Which would you be more likely to believe
• “We asked 10 people …”
• “We asked 1000 people …”
Importance of sample size

Sampling error - how is the sample different
from the population?

Response rate
• What proportion of the sample actually responded to
the survey?
• Hidden costs here - what can you do to increase
response rates
• Non-response error (bias)
• Is there something special about the data that you’re
missing (From the people who didn’t respond)?
Importance of sample size

Stage 6) Designing the survey instrument

Question construction: How the questions are
written is very important
• Clearly identify the research objectives
• Do your questions really target those research
objectives (think Internal and External Validity)?
• Take care wording of the questions
• Keep it simple, don’t ask two things at once, avoid
loaded or biased questions, etc.
• How should questions be answered (question type)?
11 Stages of survey research
Poor
Was the FDC negligent by ignoring the
warnings about Vioxx during testing
and approving it for sale?
Yes
Problem: a)emotionally
b) No
charged
c) words
Unsure
Do you favor eliminating the wasteful
excess in the public school budget?
a)
b)
c)
Yes
No
Unsure
Good
If the FDC knew that Vioxx caused
serious side effects during testing,
what should it have done?
a)Ban it from ever being sold
b)Require more testing before
approving it
c)Unsure
Do you favor reducing the public school
budget?
a)Yes
b)No
c)Unsure
Good and poor questions
Poor
Should senior citizens be given more
money for recreation centers and
food assistance programs?
a) Yes
b) No
Problem:
asks
c) Unsure
two
different questions
Good
Should senior citizens be given
more money for recreation centers?
a)
b)
c)
Yes
No
Unsure
Should senior citizens be given
more money for food assistance
programs?
a)
b)
c)
Yes
No
Unsure
Good and poor questions
Poor
Are you against same sex marriage
and in favor of a constitutional
amendment to ban it?
Good
What is your view on same sex
marriage?
a)
a)
b)
c)
Yes
No
Unsure
Problem: Biased in
more than one direction
b)
c)
I think marriage is a matter of
personal choice
I’m against it but don’t want a
constitutional amendment
I want a constitutional
amendment banning it
Problem: Asks two
questions
Good and poor questions

Question types

Open-ended (fill in the blank, short answer)
• Can get a lot of information, but
• Coding is time intensive and potentially
ambiguous

Close-ended (pick best answer, pick all that apply)
• Easier to code
• Same response alternatives for everyone
• Take care with your labels
• Decide what kind of scale
• Decide number/label of response
alternatives
Survey Questions
What is the best thing
about ISU?
What is the best thing about
ISU? (choose one)

1. Location

2. Academics

3. Dorm food

4. People who sell
things between
Milner and the
Bone

Decide what kind of rating scales
• Rating:
e.g., Likert scale
PSY 231 is an important course in the major.
1
Strongly
Agree
2
Agree
3
Neutral
4
Disagree
5
Strongly
Disagree
• Semantic differential:
Rate how you feel about PSY 231 on these dimensions
Important _____:
_____:
_____: _____:
_____: Unimportant
Boring
_____:
_____: _____:
_____: Interesting
_____:
• Nonverbal scale for children:
Point to the face that shows how you feel about the toy.
Survey Questions: Close-ended

Decide number/label of response alternatives
• Use odd number (mid point and equal # of responses above
and below the mid point)
• Questions should be uni-dimensional (each concerned with
only one thing)
• Labels should be clear
Survey Questions: Close-ended

Stage 7) Pre-testing the survey instrument


Stage 8) Selecting and training interviewers





Fix what doesn’t seem to be working
For telephone and in-person surveys
Need to avoid interviewer bias
Stage 9) Implementing the survey
Stage 10) Coding and entering the data
Stage 11) Analyzing the data and preparing a
final report
11 Stages of survey research