Transcript Survey Research I: Sampling Designs

Research Methods
Survey Research
Outline: Survey Research
 I. Sampling
 II. Survey Contents
I. Sampling from a Population
“Sampling is the process of selecting units (e.g., people,
organizations) from a population of interest so that by
studying the sample we may fairly generalize our
results back to the population from which they were
chosen.”



Population - complete set of individuals having some
common characteristic
– e.g., Australians
Sampling frame – subset of the population from which
the sample is actually drawn
– e.g., White pages
Sample – the set of people included in the study (i.e.,
selected from the sampling frame)
– e.g., Every 1000th person in the white pages
Population and Sample
Population
Sample
Use statistics to
summarize features
Use parameters to
summarize features
Inference on the population from the sample
Confidence Intervals
 The problem is to use the sample to make inferences
about the population
 If we were to repeat the entire process of drawing a
sample and computing the statistic many, many
times, we would find that the statistic varies some
 A confidence interval takes advantage of those
variations and allows us to specify a range that
probably contains the true value of the parameter


Technically, for an X% confidence interval, the
parameter will fall inside the range X% of the time in
repetitions of the study
Confidence intervals usually vary between 90% and
99.9%
Define a Population
 The first step in choosing a sample is to
define the population, or the overall set of
cases that we’re interested in



If we want to predict the results of an election,
then our population is “likely voters”
If we want to understand who decides to vote,
then our population is “citizens eligible to vote”
If we want to understand who decides to use
marijuana in the United States, our population
is probably “everyone in the United States”
Types of Samples Used
 Nonprobability Sample
 Items included are chosen without
regard to their probability of occurrence
 Probability Sample
 Items in the sample are chosen on the
basis of known probabilities
Types of Sampling Methods
Samples
Non-Probability
Samples
Probability Samples
Simple
Random
Convenience
Quota
Stratified
Cluster
Systematic
Probability Sampling
 Subjects of the sample are chosen based on
known probabilities
Probability Samples
Simple
Random
Systematic
Stratified
Cluster
Simple Random Samples
 Every individual or item from the frame has an
equal chance of being selected
 Selection may be with replacement or without
replacement
 Samples obtained from table of random
numbers or computer random number
generators
Systematic Samples
 Decide on sample size: n
 Divide frame of N individuals into groups of k
individuals: k=N/n
 Randomly select one individual from the 1st
group
 Select every k-th individual thereafter
N = 64
n=8
k=8
First Group
Stratified Samples
 Population divided into two or more groups
according to some common characteristic
 Simple random sample selected from each
group
 The two or more samples are combined into
one
Stratified Sampling Example
Population
Cash holdings of
All Financial
Institutions in the
United States
Financial Institutions
Stratified Population
Stratum 1
Large Institutions
Select n1
Stratum 2
Medium Size Institutions
Select n2
Stratum 3
Small Institutions
Select n3
Cluster Samples
 Population divided into several “clusters,” each
representative of the population
 Randomly select certain clusters
 The samples are combined into one
Population
divided
into 4
clusters.
Cluster Sampling Example
Mid-Level Managers by Location for Morrison-Knudsen
Construction Company
Algeria
25
Illinois Scotland California
42
22
105
Alaska New York Florida
20
36
52
Idaho
152
Mexico Australia
76
37
Cluster Sampling Example
Mid-Level Managers by Location for Morrison-Knudsen
Construction Company
Illinois Scotland
42
22
Florida
52
All members selected from these clusters
Types of Survey Errors
 Coverage error
Excluded from
frame.
 Non response error
Follow up on
non responses.
 Sampling error
 Measurement error
Chance
differences from
sample to sample.
Bad Question!
II. Survey Contents
 Remember that your respondents will be
“Lazy Thinkers.”
Survey Contents
 When creating questions, consider the effects
of the following elements:

Type
Constraints, if any, placed on responses

Wording
Clarity of questions and response options

Order
Relative placement of questions in the instrument

Content
Topics the questions cover
Survey Contents: Type
 Open-ended
A survey question to which
the respondent replies in
his or her own word,
either by writing or by
talking
 difficult to analyze,
subjective analysis
 time consuming
 rich information
 useful for descriptive,
exploratory work
 Closed-ended or fixed-
choice
A survey question that
provides preformatted
response choices for the
respondent to circle,
check, mark, etc.
 easier to analyze
 efficient
 useful for hypothesis
testing
 important info may be
lost forever
Closed-ended Questions
 Closed-ended or fixed-choice

Response options are limited





Yes/no
True/false
Multiple choice with an “other” option
 Race: White, Black, Hispanic, Asian, Other
Likert Scales
1
2
3
4
5
Strongly Agree Neither Disagree Strongly
Agree
A nor D
Disagree
Rating Scales
Ask respondents to rate something like a person, place, thing, idea,
attitude, etc. on a numbered scale, often Likert scales
 Semantic Differential
a fifth grader is
Bad
Good
Awful ⊲
⊲ ⊲ ⊲  ⊳ ⊳ ⊳
⊳ Nice

Survey Contents: Wording
 Wording
Clarity of questions and response options



Respondents should understand your questions
Pre-testing is a very effective way to see if they do
Examples of bad questions:
 Do you hate taxes?
 The answer will tell you little because our society has many
different taxes, loves to complain about them, but thinks they
are necessary

Do you hate getting up in the morning?
Survey Contents: Wording
 Guidelines for good questions:
1.
2.
3.
4.
Be direct
Maintain simplicity
Be specific
Take the role of your respondent
The following points elaborate on these four main
themes.
Survey Contents: Wording

No complex rhetoric, syntax, or disciplinary slang or
jargon.

Do not expect them to learn new information just
answer a question. (Sometimes, you may establish
context with a short paragraph then ask a series of
short, specific questions.)

Avoid phrasing questions to seem too personal or
direct, especially when dealing with culturally sensitive
issues. For example, “Do you abuse your kids?”
Survey Contents: Wording

Avoid Double-barreled questions. They contain two
questions in one. For example, “Do you think that
students and Professors should be given discounts on
sports tickets?”

Avoid Double-negative questions. For example, “Do you
disagree that professors should not be required to help
students outside of class?”

Avoid hypothetical questions (unless you are studying
hypothetical situations). You will not gain useful
information from these questions. For example, “If men
could have children, would your husband or boyfriend
stay with you?”
Survey Contents: Wording

Avoid ambiguous questions.



“Do you teach your children to effectively function?” “Does your
boss engage you in interactive dialogue?” “Effectively function”
and “interactive dialogue” are subject to interpretation.
Words such as "usually" or "normally" mean different things to
different people. “Do your customers normally complain?” has a
variety of interpretations.
Avoid biased questions. Avoid making one response
option look more suitable than the other, using
emotionally loaded terms, or using unbalanced response
categories.


“Don’t you think that suffering terminal cancer patients should be
allowed to be released from their pain by choosing death?”
Virginia pays teachers more than similar states. Virginia should:
spend more, keep spending the same, reduce spending little,
reduce spending some, reduce spending a lot, dramatically reduce
spending
Survey Contents: Order
 Order
Relative placement of questions in the instrument

Opening questions should be simple and introduce the
topic of the survey.

Try not to mix topics. Put like things into sets of questions.

Avoid framing later questions with topics that can be linked
to them in previous questions. For example, one should
not ask about attitudes toward crack use right before
asking about attitudes toward the urban poor. This will
invoke stereotypes about the poor.
Survey Contents: Order

Sensitive questions should never be at the
beginning. Put in middle. May need opening
paragraph for them.

Consider need to transition between types of
questions. Write transitory questions or
directions.

Place your most important questions earlier—R’s
may fatigue before answering them if they are
later.
Survey Contents: Order
 Do not put in too many complex format questions (e.g.,
“skipping” or “go to next section”). R’s will get
confused/frustrated.
 Closed-ended questions are easier to answer. Put open-
ended questions later in survey. Investment of time up
to that time will likely keep them through the openended.
Survey Contents: Content
 Content
Topics the questions cover

You should make the topic of the survey clear to R’s. Do
not surprise them with questions about unrelated topics.

If including sensitive questions, provide a rationale for
asking them. Explain how honest answers will be helpful
to others.

Questions relevant to deviance should include normalizing
statements. For example, “Many people use drugs for a
variety of reasons. Have you ever used ecstasy (or x, e,
MDMA) to feel closer to other people?”
Survey Contents: Content

The more sensitive the topic, the more reassurance of
confidentiality should be stated.

Never open a survey with questions about sensitive topics,
and do not end surveys with questions that will negatively
linger in R’s minds.

Continually reassure R’s that there are no right or wrong
answers to questions—just truthful or not.

Questions that require lots of specific details or a good
memory are typically useless. If they are necessary,
employ techniques to prompt recollection.
Survey Contents
 General Advice

Always consult other surveys first



They will give you good ideas for exploring your
topic
They may provide good examples of how to ask
questions
You avoid “reinventing the wheel”
 Especially useful for validated scales
 Be sure that you have permission to use

Always pre-test your instrument
Survey Contents
 Consider these issues:

Not everyone will have an opinion on every topic.
Consider “no opinion” as an option.

People want to appear to agree. Consider negative and
positive statements.

R’s may lose track and choose salient options (such as
first in series). Keep questions simple.

Response set problem: In a series with the same
response options, R’s may just quickly check the same
response for all questions.
Survey Contents
 Consider these issues:
 Bad memory leads to:





Forward telescoping (reporting that events occurred
more recently than in reality)
Backward telescoping (reporting events further back
than in reality).
Salient events are overreported
Mundane events are underreported
“Habitual” events will fill in for lost information.
Provide aides to recall such as reference points,
landmark events, etc. Use limited time frames in
questions.
Class Exercise
 Split into groups
 Select a topic for a survey
 Develop 2 closed-ended and 2 open-ended
questions. For each question, write a ‘poor’
version and a ‘good’ version
 Administer the survey to member of another
group. Critique each other’s questions