Transcript Here - Psychology 242

1
Foundations of Psychology 242, Dr. McKirnan.
Research
Survey research
✓ Topic areas & formats

 Testing Hypotheses
 General issues in Survey research
 Sources of bias (or fraud…)
 Examples of surveys
3/10/14
Foundations of
Research
Survey research;
General uses of surveys
Survey methods have a wide range of applications,
 from single-item consumer satisfaction (“How useful did
you find this web site”)
 to full-fledged, theory-driven behavioral research.
For convenience we will consider 5 categories:
 Descriptive research
 Testing hypotheses;
 Testing the generalizability of experimental results;
 Predicting an event or outcome;
 Pragmatic / applied questions.
2
Foundations of
Research
Uses of surveys; descriptive research
3
 Descriptive research
 Epidemiology is the study of how behaviors, disease states, or
similar issues are distributed across the population.
 Epidemiology uses many methods, such as standard crime or
disease reporting.
 Even Google search data (Google Trends) can be to track, e.g.,
heroin overdose.
 Many epidemiological studies use direct survey methods, such as
phone or face-to-face survey interviews.
 Knowledge of, e.g., how to access health care…
 Feelings or moods, such as the rate and distribution of depression…
 Behavioral patterns, such as alcohol or drug use or gun ownership…
Foundations of
Research
Uses of surveys; descriptive research
4
 Descriptive research
 Epidemiology is the study of how behaviors, disease states, or
similar issues are distributed across the population.
 Epidemiology uses many methods, such as standard crime or
disease reporting.
 Even Google search data (Google Trends) can be to track, e.g.,
heroin overdose.
 Many epidemiological studies use direct survey methods, such as
phone or face-to-face survey interviews.
 Knowledge of, e.g., how to access health care…
 Feelings or moods, such as the rate and distribution of depression…
 Behavioral patterns, such as alcohol or drug use or gun ownership…
Foundations of
Research
Counting Google searches to assess social interest
5
Do Chicago Cubs’ quest for the World Series inspire national interest?
Google Search data during the MLB Division playoffs, 2016:
 Interest in the LA Dodgers is limited to California & Nevada;
 Interest in Cleveland is limited to Ohio;
?
 Interest in the
Toronto Blue Jays
is limited to
Canada;
 The Cubs truly
arouse national
attention.
Foundations of
Research
The origins of epidemiology
6
 In London of 1854 a cholera outbreak raged through
several poor neighborhoods of London.
 Sewage and other effluvia that ran through gutters
created a dreadful smell (a “miasma”) that was blamed
for the outbreak.
 The concept of infectious disease transmission through
water supplies was not well understood.
 Dr. John Snow, one of the physicians charged with stopping the
epidemic, noted a particularly fetid cesspool in front of 40 Broad St.,
proximal to a water pump used by the neighborhood.
 He decided to empirically map the cholera cases in the area.
 He proposed that water from the pump, not the miasma, was the cause
of the outbreak.
 He was was generally disbelieved, but convinced the town governors by
his evidence.
Image:
http://leabridge.org.uk/waterworks/timeline/1
852-1880.html
Only much later would tracking of disease
outbreak be labeled ‘epidemiology’.
Click for Wikipedia article.
Dr. John Snow’s Cholera map & the closing of the Broad St. pump.
Foundations of
Research
7
The origins of epidemiology
Dr. John Snow’s Cholera map & the closing of the Broad St. pump.
of cases to be near the pump
at 40 Broad St….
Click for Ted talk by Steven Johnson.
 …and to radiate out from
there.
 As he noted in his 1855 book:
"I had an interview with the Board of
Guardians of St. James's parish, on the
evening of Thursday, 7th September, and
represented the above circumstances to them.
In consequence of what I said, the handle of
the pump was removed on the following day.”
 By carefully describing the
distribution of cases and the
circumstances around the pump, Snow was able to empirically
demonstrate a likely cause.
 His hypothesis was supported by the epidemic quickly subsiding once
the pump handle was removed.
S. Johnson, The Ghost Map (2007), Riverhead Books.
 Snow’s map showed the bulk
Foundations of
Research
Uses of surveys; descriptive research
8
 Descriptive research
 Epidemiology
 Political / social description is what we often think of as surveys.
 Opinion polls about society, the government, or current events,
e.g. Gallup Polls, or systemic studies by Pew Memorial Trust.
 The Consumer Confidence Index is a highly standardized poll that
is used for basic economic decision making.
 The Census, of course, is our national information source.
Foundations of
Research
Uses of surveys; descriptive research
9
 Descriptive research
 Epidemiology
 Political / social description.
 Testing hypothesis
 Assessing blocking variables
 We often assess blocking variables to test how a given attitude or
behavior varies across important social groups.
e.g., gender, age group, ethnicity, geographic location…
 Correlational studies
 A key form of analysis is examining the association among different
variables
e.g., what are the correlates of dieting…
Foundations of
Research
Uses of surveys; descriptive research
 Descriptive research
 Epidemiology
 Political / social description.
 Testing hypothesis
 Assessing blocking variables
 Correlational studies
 Examine generalizability of experimental results
 E.g., The Consumer Reports survey on therapy we discussed in quasiexperiments
 Predict event or outcome;
 E.g., election polling
 Pragmatic / applied,
 E.g., marketing, or consumer surveys.
10
Foundations of
Research
What do surveys measure?
 Knowledge
 Information re: current events, political or consumer choices
 Awareness of Public health resources, health practices, etc.
 Attitudes and Beliefs
 Preferences or evaluations: e.g., attitudes toward gays, ethnic
groups, etc., consumer preferences.
 Beliefs about political or social events: “which party provides the
strongest security for the U.S….?”
 Feelings or moods: quality of life, depression / anxiety, marital
satisfaction, etc.
 Behavior
 Behavioral intentions; Intent to vote, financial plans, etc.
 Self-reports of previous or on-going behavior; topics range from
voting to alcohol and drug use.
11
Foundations of
Research
Surveys; populations
 Who do we want to generalize to ?
 Our sampling frame is based on our hypothesis or
empirical question.
 Sampling: breadth  internal validity tradeoff
 Key dimensions:
 Demographic
 ethnic / age / gender groups,
 “all Americans”…
 Behavioral
 “likely voters”, alcohol users, home buyers...
 Self-identification
 Republicans / Democrats,
 “students”…
12
Question Formats: Closed-ended
Foundations of
Research
items

Specific rating scale or highly structured prompts

Most reliable for concrete behaviors

An attitude can be assessed in several ways:

Direct (face valid) assessment
Psychology 242 is a wonderful course…

Does not 1 2 3 4 5 6 7 strongly
agree at all
agree
Behavioral (content valid) indictors
How many times this semester have you skipped 242 class?
How many hours per day do you spend reading 242 material?

13
0
______
2.5
______
Researchers typically use the M of several related
items to create a more reliable measure of a variable.
Foundations of
Research


14
“Closed-ended” items, cont.
Example: CES-D Depression inventory:
We may use the M score of these 9 items as our
depression index…
Moods & Feelings
Below is a list of different feelings. Circle the number that shows how many days you felt each
of these over the PAST WEEK.
I was bothered by things that usually do
not bother me.
I felt I could not shake off the blues even
with help from my friends or family.
I had trouble keeping my mind on what
I was doing.
I felt depressed.
I felt that everything I did was an effort.
My sleep was restless.
I was happy.
I enjoyed life.
I felt sad.
Rarely or
none of
the time
A Little
of the Time
A moderate
amount of
the time
Most or all of
the time
(less than 1 day)
(1 or 2 days)
(3 - 4 days)
(5 - 7 days)
0
1
2
3
0
1
2
3
2
3
2
2
2
2
2
2
3
3
3
3
3
3
0
1
Sum
of item
ratings
/19
0
0
0
0
0
0
1
1
1
1
1
These items are “reversed”
in the final score
Foundations of
Research


15
“Closed-ended” items, cont.
Or we may count the number of symptoms
We may have a cut point: e.g., moderate depression is
defined as 4+ symptoms
Moods & Feelings
Below is a list of different feelings. Circle the number that shows how many days you felt each
of these over the PAST WEEK.
I was bothered by things that usually do
not bother me.
I felt I could not shake off the blues even
with help from my friends or family.
I had trouble keeping my mind on what
I was doing.
I felt depressed.
I felt that everything I did was an effort.
My sleep was restless.
I was happy.
I enjoyed life.
I felt sad.
Rarely or
none of
the time
A Little
of the Time
A moderate
amount of
the time
Most or all of
the time
(less than 1 day)
(1 or 2 days)
(3 - 4 days)
(5 - 7 days)
0
1
2
3
0
1
2
3
0
1
2
3
1
1
1
1
1
2
2
2
2
2
2
3
3
3
3
3
3
# of symptoms:
items0 rated 2 1or 3
0
0
0
0
0
Foundations of
Research


Closed-ended items, summary
Chief virtue: clear operationalization

Specific & concrete; we know exactly what the
participant is responding to

Easy to quantify & use statistically

Can be tested for reliability
Chief liability: potential insensitivity

Brief & simply worded; potentially superficial

“Top down”; issues are imposed on the participant


Discrimination studies: no option for “has no attitude”
Attitudes / moods: not sensitive to participants’ personal
perspectives (?).
16
Foundations of
Research
Survey formats;
“Open-ended” items
General textual / qualitative response;

More sensitive to the respondent
How have you enjoyed your methods class so far?
Please list the three things that first come to mind when you
think of Psychology 242.

More difficult to interpret

Can be analyzed as qualitative data (see discussion in Descriptive data.)

Can be quantified;
frequency counts of citations or statements
 “linkages” analysis of co-occurring statements


Often presented as textual portrayal plus minor
quantitative analysis.
17
Foundations of
Research
18
Example of mixed survey formats
Example of mixed question
format from survey of women’s
sexual practices.

Closed-ended attitude scale

Open-ended description

Simple behavioral index.
Personal Safer Sex Guidelines
How strict are your personal guidelines or rules for safer sex (e.g., condom use,
“safe relationships,” etc.)?
1
Not at all
Strict
2
3
Somewhat
Strict
4
5
6
Very
Strict
7
Extremely
Strict
What are your rules for safer sex?
Have you ever refused to have sex
with someone to stay safe?
0
1
2
3
Never
once or
twice
a few
times
many
times
Foundations of
Research
SUMMARY




19
Survey topics & item types
Surveys assess:

Knowledge

Attitudes or preferences

Ongoing or intended behavior
Closed-ended formats
Psychology 242 is a wonderful course…

Highly structured, easy to analyze

Potentially insensitive
Open-ended formats
…list the three things that first come to mind…

More sensitive to the participant

Potentially ambiguous or difficult to analyze
Surveys typically…

Use multiple items

Employ several formats.
Foundations of Psychology 242, Dr. McKirnan.
Research
Survey research
 Topic areas & formats
✓
 Testing Hypotheses
 General issues in Survey research
 Sources of bias (or fraud…)
 Examples of surveys
20
Foundations of
Research
Alcohol: Knowledge 1
What is the chemical in
alcohol?
Let’s do our own practice survey!
a) ethanol
b) zymase
c) dehydrogenase
d) nicotine
21
Foundations of
Research
Alcohol: Knowledge 2
Which of these drinks
contains the least
alcohol?
A = 12 ounces of beer
B = 5 ounces of wine
C = 1 1/2 ounces of vodka
D = 8 ounces of malt liquor
E = All are equal
22
Foundations of
Research
Alcohol: Knowledge 3
How long does it take for
alcohol to affect the
brain?
a) 10 seconds
b) 60 seconds
c) 10 minutes
d) 90 minutes
23
Foundations of
Research
Alcohol attitudes, 1
Drinking helps me feel
whatever way I want to
feel.
A = agree strongly
B = agree
C = somewhat agree
D = slightly agree
E = do not agree
24
Foundations of
Research
Alcohol attitudes, 2
Alcohol makes me feel
happy.
A = agree strongly
B = agree
C = somewhat agree
D = slightly agree
E = do not agree
25
Foundations of
Research
Alcohol attitudes, 3
Drinking adds a certain
warmth to social
occasions.
A = agree strongly
B = agree
C = somewhat agree
D = slightly agree
E = do not agree
26
Foundations of
Research
Alcohol behavior, 1
How often do you have a
drink containing
alcohol?
A = never
B = monthly or less
C = 2 to 4 times a month
D = 2 to 3 times a week
E = 4 or more times a week
27
Foundations of
Research
Alcohol behavior, 2
How many drinks
containing alcohol do
you have on a typical
day when you are
drinking?
A=1
B=2
C=3
D = 4 or 5
E = 6 or more
28
Foundations of
Research
Alcohol behavior, 3
How often during the last
year have you found that
you were not able to stop
drinking once you had
started?
A = never
B = less than monthly
C = monthly
D = weekly
E = daily or almost every day
29
Foundations of
Research
Alcohol behavior, 4
How often during the last
year have you failed to do
what was normally
expected from you
because of drinking?
A = never
B = less than monthly
C = monthly
D = weekly
E = daily or almost every day
30
Foundations ofKnowledge / attitude / behavior composites
Research
Knowledge
31
Here
is our
survey.
Our data
set:
 Several items assess each
Hypothetical Construct.
 Which of these drinks contains the least alcohol?
 Typically we also collect
 How long does it take for alcohol to affect
the brain? (age, gender,
demographics
ethnicity…) and other Ψ
Attitudes and beliefs
variables.
 Drinking helps me feel whatever wayI want
to feel. items based on our
We choose
 What is the chemical in alcohol
 Alcohol makes me feel happy.
Theory about alcohol use and
problems.
 Drinking adds a certain warmth to social occasions.
 We can simply Describe our
Behavior
data in each category.
 How often do you have a drink containing alcohol?
 We might Block the data by
 How
many
drinks
containing
alcohol doimportant
you have categories
on a typicalofday
How
many
students
use
alcohol at
participants
all?when you are drinking?
Do
more the
positive
 men
How have
often aduring
last year have you
found
thatTest
you Hypotheses
were not
 Or
we can
attitude
alcoholonce
than you had started?
able totoward
stop drinking
about correlations among
women?
variables
 How often during the last year have you failed to do what was
normally expected from you because of drinking?
Foundations ofKnowledge / attitude / behavior composites
Research
Knowledge
32
Research questions:
 How well do the items within each category
 What is the chemical correlate?
in alcohol
 Which of these drinks
least alcohol?
• contains
Are they the
internally
Reliable?
 How long does it take
alcohol
to affectathe
brain?
• for
Can
we construct
Composite
Variable from
Attitudes and beliefs
each item set?
Blocking
: Gender?
 Drinking helps me 
feel
whateverVariables
way I want
to feel. Ethnicity? Age?...
What
are the Correlations among composite
 Alcohol makes mefeel
happy.
variables?
 Drinking adds a certain warmth to social occasions.
• How much are knowledge or attitudes & beliefs
associated with alcohol use?
Behavior
• Are
alcohol
use andalcohol?
problems correlated?
 How often do you have
a drink
containing
• Are these
effects
Mediated
byaother
variables?
 How many drinks containing
alcohol
do you
have on
typical
day
when you are drinking? Do depression, isolation, or stress make people
vulnerable
to alcohol
problems?
 How often during the last year
have you
founduse
thatoryou
were not
able to stop drinking once you had started?
 How often during the last year have you failed to do what was
normally expected from you because of drinking?
Foundations of
Research
Class data, n = 97
Multiple items assessed the Hypothetical Constructs of
Alcohol- related information, attitudes, use, and problems.
 Information: # of correct answers for items 1  3.
 Attitudes: items 4, 5 & 6.
 Use: drinking frequency * # drinks / occasion (items 7 & 8).
Example
 Problems: Items 9 & 10.
Information:
Mode = 2 items correct.
33
Foundations of
Research
Class survey data, Attitudes
 To test weather the 3 attitude items comprised a single
Hypothetical Construct, we examine their
intercorrelations
Example
 All the correlations and high and statistically significant,
so we can combine them into a single composite variable.
34
Foundations of
Research
Class survey data, Attitudes
 The frequency distribution for the composite attitude
variable shows an approximate normal distribution.
Example
 The Mean and Median are
equal, near the center of the
distribution
 The mode reflects a larger
block of participants with
moderately strong attitudes.
35
Foundations of
Research
Class survey data, Alcohol problems
 We perform the same operations for items 9 & 10, the alcohol
problems measures.
 The correlation between these items is very high.
Example
 We can validly combine them into one variable.
36
Foundations of
Research
Class survey data, Alcohol problems
 Most people report no problems, as shown in the Mean and
Median
 The rest of the distribution shows a strong positive skew.
Example
 This skew pulls the Mean higher than the Mode & Median.
37
Foundations of
Research
Class survey data, Correlation results
 The amount of information people have about alcohol is
unrelated to their alcohol use or problems:
 The correlations are low;
 Neither is statistically significant.
 Positive attitudes toward alcohol are strongly associated with
alcohol use.
Example
 … and moderately (statistically significantly) with alcohol problems.
 Thus, attitudes are more important to alcohol abuse than is
knowledge.
38
Foundations of
Research
39
Correlations and mediators
With more data we could use mediating analyses to
develop to test theories about alcohol use / problems

Say we find that people who are lonely or depressed (negative
affect) have more alcohol use & problems.

Why might this be?

People with negative affect may come to expect (hope?) that
alcohol will make them feel better.

The attitude may be what actually leads to alcohol use.

Here attitudes / beliefs would mediate the effect of negative
affect on alcohol use & problems.
Depression
loneliness
stress
Positive
attitudes
toward
alcohol
Alcohol
use &
Problems
Foundations of
Research
40
Correlations and mediators
Using mediating analyses to develop to test theories
about alcohol use / problems


Say we find that people who are lonely or depressed (negative
 use
Using
a mediating analysis to ask
affect) have more alcohol
& problems.
“why” is the core step in Theory
Why might this be?
Development.
 People with negative affect may come to expect (hope?) that
alcohol will make them feel better.

The attitude may be what actually leads to alcohol use.

Here attitudes / beliefs would mediate the effect of negative
affect on alcohol use & problems.
Depression
loneliness
stress
Positive
attitudes
toward
alcohol
Alcohol
use &
Problems
Foundations of
Research
41
Correlations and mediators
Survey data are important for developing or testing
new theories of behavior

Can have high external validity

Can model different relations among variables.
Problems?

The data are correlational and cross-sectional.

The causal arrows may go in different directions.
Depression
loneliness
stress
Positive
expectation
of alcohol
Alcohol
use &
Problems
Foundations of
Research
42
Correlations and mediators
Survey data are important for developing or testing
new theories of behavior

Can have high external validity

Can model different relations among variables.

The causal arrows may go in different directions.

There may be a third variable that we did not measure.
Genetics:
Positive
alcohol effects
Problems?
& negative

The data are correlational
and cross-sectional.
affectivity
Depression
loneliness
stress
Positive
expectation
of alcohol
Alcohol
use &
Problems
Foundations of
Research
cures
Cures for correlation problems?

Longitudinal studies


Examine changes over time to
better determine cause and
effect
Experimental studies

Simulate and manipulate the
predictors, e.g.;
 Induce temporary stress /
negative affect
 Assess attitudes
 Assess alcohol use
43
Foundations of
Research
44
We Use Longitudinal data
 To examine change
 To approximate “cause and effect”

Research has shown that
depression, loneliness &
stress are correlated.

So, we combine them into a
single composite variable.

This is our Predictor
Depression
Loneliness
Negative
Affect
Stress
(Measurement version of IV)


The same for alcohol use
and problems.
This is our outcome
(Measurement version of DV)
Alcohol
use
Alcohol
problems
Alcohol
abuse
Foundations of
Research


45
Longitudinal data: Path analysis
We test the hypothesis that Negative Affect
correlate with alcohol abuse & problems…

…at baseline

…and at follow-up
Data at each time frame
are cross-sectional:
 We cannot determine if
emotions “cause” the
alcohol outcome, or
visa versa.
Time 1
Time 2
(baseline)
(follow-up)
Negative
Affect
Negative
Affect
Alcohol
abuse
Alcohol
abuse
Foundations of
Research

46
Longitudinal data: Path analysis
Looking at patterns across time lets us make more sensitive
statements.
Time 1
(baseline)
If the measures are
reliable Negative Affect at
Time 1 will correlate with
NA at Time 2.
Time 2
(follow-up)
Negative
Affect
Negative
Affect
And the same for the
alcohol variable
Alcohol
abuse
Alcohol
abuse
Foundations of
Research

47
Longitudinal data: Path analysis
To test the hypothesis we look at the “cross” correlations…
Time 1
(baseline)
Negative
Affect
Alcohol
abuse
How strongly does
Affect at baseline
correlate with Alcohol
at follow-up?
Time 2
(follow-up)
Negative
Affect
A strong correlation
would tell us that Affect
does lead to more
Alcohol abuse over time.
Alcohol
abuse
Foundations of
Research

48
Longitudinal data: Path analysis
To test the hypothesis we look at the “cross” correlations…
Time 1
(baseline)
Negative
Affect
Alcohol
abuse
Similarly, how
strongly does Alcohol
at T1 correlate with
changes in Affect at
T2?
This correlation tells us
how much alcohol use
leads to negative affect
over time.
Time 2
(follow-up)
Negative
Affect
Alcohol
abuse
Foundations of
Research

49
Longitudinal data: Path analysis
Testing the hypothesis: does Negative Affect ‘cause’ Alcohol
use & problems
Time 1
(baseline)
Negative
Affect
Alcohol
abuse
If the correlation
between Affect at Time
1 and Alcohol at
follow-up is very
strong…
Time 2
(follow-up)
Negative
Affect
Alcohol
abuse
Foundations of
Research

50
Longitudinal data: Path analysis
Testing the hypothesis: does Negative Affect ‘cause’ Alcohol
use & problems
Time 1
(baseline)
Negative
Affect
Alcohol
abuse
…and the correlation
between Alcohol at T1
and Affect at T2 is
relatively weak…
Time 2
(follow-up)
Negative
Affect
Alcohol
abuse
Foundations of
Research

51
Longitudinal data: Path analysis
Testing the hypothesis

Time 1
(baseline)
Negative
Affect
Alcohol
abuse
…we can conclude
that Affect may be a
more important
“cause” of alcohol
use / problems than
the other way around.
Time 2
(follow-up)
Negative
Affect
Alcohol
abuse
Foundations of
Research

52
Longitudinal data: Path analysis
Testing the hypothesis
Time 1
(baseline)
Of course we may find
that Alcohol “causes”
Affect, not the other
way around…
Negative
Affect
Alcohol
abuse
Time 2
(follow-up)
Negative
Affect
Either way, we have a
better sense of cause
and effect than with
cross-sectional data.
Alcohol
abuse
Foundations of
Research

Surveys typically use multiple items to measure
each hypothetical construct

Summary



Summary: Testing Hypotheses
Correlations among items tell us if they are reliable in
measuring the same construct.
We use Mediating Analyses to

Test hypotheses about correlations between constructs

Build or test theory
Cross-sectional analyses are difficult to interpret

Causal direction?

3rd variable problem
Longitudinal analyses help us determine causal
direction
53
Foundations of Psychology 242, Dr. McKirnan.
Research
 Topic areas & formats
 Testing Hypotheses
 General issues in Survey research
 Sources of bias (or fraud…)
 Examples of surveys
54
Forms of survey administration
Self-report questionnaire
Foundations of
Research



Primarily closed-ended, structured questions

Limited open-ended items

Assume at least moderate reading level

Cheap & easy to administer


“Paper and pencil” or internet-based;
All data collection
increasingly
Internet: Representativeness very dubious
computer-based
Face-to-face interview

“Door step”, formal research center, or telephone

Allows in-depth qualitative questions

Many studies combine questionnaire & interview formats

(Telephone version becoming obsolete)
55
Foundations of
Research

Cost / population access

Different methods are more / less likely to reach certain
populations, e.g.:




General issues in surveys
Disfranchised / poor populations often not reached by internet or telephone
Cell phones & avoidance of telemarketers  less availability for telephone
surveys
Stigmatized populations less available for face-to-face interviews, more
available via internet.
Participant sophistication

Participants may not be able to accurately report certain topics

Attitudes toward stem cell research from readings.

What factors are most important to your choice of political candidate....


Describe the amounts and types of proteins you eat during a typical
week...
“Rationality bias”; many questions (incorrectly?) assume a rational
reason for behavior:

Why do you have unsafe sex...

What is your chief reason for using alcohol each night…
56
Foundations of
Research


Social Desirability Responding
Clear face-valid items addressing embarrassing topics yield
less valid responses

How often are you dishonest with your friends?

Have you ever cheated on an exam....?
High social desirability wording elicits inaccurate responses…

Do you support protecting our Nation’s forests for future generations?
(Does “yes” mean you an “environmentalist”?).

Do you feel there are ways your husband could be closer...?
(Does “yes” make you are unhappy in your marriage”?).

Populations differ in social desirability responding; may be a
confound in studying group differences
 Women report more suicidal thoughts, but may be more willing to
disclose, creating a possible confound…

57
Desirability can be minimized by:

Anonymous surveys

Assurances of confidentiality

Computer administration (no personal interaction)

Careful wording / pilot testing of items
Foundations of
Research

Surveys: Social
desirability responding
Do people lie on surveys?

Men routinely report more sex partners
than do women.

If the sample is unbiased by gender,
partners should balance for men &
women.

Social desirability hypothesis:

Women underestimate partners

Men overstate partners

Much of the difference due to:
 A high proportion of women who report 1 partner
 A few men who report many partners.

Possible sample bias (confound?) in who responds to such
surveys?
Click for the article
58
Foundations of
Research

General issues in surveys: Time Frames
Rare(er) events require a long time frame to assess


When was your last doctor’s visit…
These questions asses the last time you left a romantic
relationship…
 Longer term recall can be surprisingly unreliable
 Recall of last doctor visit highly unreliable when checked against
medical records

Shorter time frame yields more reliable responding
 Memory is better for more recent effects
 “Exit interviews” from medical visits far more reliable than even 2week retrospective measures.
 Current, concrete behaviors are more accurately reported than
behavioral trends.
 In general, how often do you miss a dose of your medication
Less reliable than…
 Let’s go over each of the past 7 days and tell me if you took or
missed your medication dose…
59
Foundations of
Research

General issues in surveys: Question Order
Questions trigger participant’s memory or attention,
and can bias questions that follow, e.g.:
Do you think Social Security & Medicare payments have kept up
with inflation..
then
Do you favor or oppose Democratic efforts to expand Medicare
payments...

Bias can be limited by counterbalancing questions.

Using different question orders in different versions of the
survey.
60
Foundations of
Research

Summary

61
Survey administration

Internet increasingly important as self-report method

Face-to-face interviews more common in clinical research

Time frames & question order can influence responses
Population access & sophistication

Some groups are difficult to reach

Creates threat to External validity


General Issues Summary
Assumption that participants understand survey materials
often questionable.
Social desirability responding


Inhibited responding threatens Internal Validity
May represent a confound if groups differ in desirability
set.
Foundations of Psychology 242, Dr. McKirnan.
Research
 Topic areas & formats
 Testing Hypotheses
 General issues in Survey research
 Sources of bias (or fraud…)
 Examples of surveys
62
Foundations of
Research


Bias / Fraud in survey research
Social research is increasingly important to political &
cultural debates.

Effects of gay marriage

Political “approval” ratings

Scientific consensus on global warming

Research on working mothers …
Pressure for confirmatory results encourages bias or
outright fraud (see this week’s article on Opinion Polls).


In the study structure

Items used

Sample
In the interpretation of results

“Cherry picking”

Simple distortion
63
Foundations of
Research

Opposition to gay marriage


Example



64
Example of fraudulent survey use
Judges have struck down religious doctrine as a basis for
disallowing gay marriage
Opponents have sought evidence of civil
harm to justify discrimination
Harm to children has emerged as key
issue in the debate
APA; multiple studies show no harm.
April DeBoer, left, and Jayne Rowse on
Friday after closing arguments in their
challenge to Michigan’s marriage
restriction. Credit Mandi Wright/Detroit Free Press,
Mark Regenerus’ Survey: Children in
via Associated Press
households with gay/lesbian parent
fare worse.
 Funded & cited widely by gay marriage opponents
 Study sample and interpretation of results wildly biased
 Has been disavowed by Renenerus’ Academic Department and
American Sociological Assn.
 Continues to be central “evidence” in court cases
Foundations of
Research

65
Source of survey fraud; question wording can elicit a
response desired by the researcher;


Bias in survey research: Leading or biased items
How much do you support the administrations’ actions to protect
you and your children from terrorists…
Wording can “normalize” a response, e.g.,

When do you feel that it is O.K. to cheat on an exam?
..when I really do not know the material
.. when others are doing it
.. when I think the exam is unfair

Vague wording can be interpreted in a biased fashion
Is there anything your husband could do to be more intimate with you?

“Push” polls: a survey can be used to actually create an
attitude.
Foundations of
Research
Push Polls:
66

An item “traps” the participant into endorsing a specific view.

When the “data” are released the biased wording is ignored.



Publicity about the “findings” (e.g., by politically biased news
organizations) are used to further create or change attitudes.
Many political & social organizations use this strategy to…

Ostensibly measure attitudes objectively

Use the “results” to influence popular opinion.
Example of a highly biased survey: The Republican National
Comittee health care survey.

This “survey” attempted to create fear of health care reform.

It got limited distribution, but is a great example of a Push Poll
Foundations of
Research


GOP health survey (2009 – 2010)
Some of these items are simple lies – or manipulations based
on lies – designed to induce anti-health care attitudes…
Other are powerful (and dishonest) emotional manipulations
✓
✓
✓
✓
67
Foundations of
Research


Forms of survey bias:
Provide leading or emotionally manipulative
information to induce an attitude rather than simply
measure it, to provide politically useful “data”..
Questions that, if you accept their assumptions, can
only be reasonably answered one way…
68
Foundations of
Research

Biased surveys: Democratic example
An example from
the Democrats,
that is also used
for fund-raising.
69
Foundations of
Research
Manipulative
presentation
of
questionable
information
Simple
emotional
manipulation
Distorted
description
that may be
changed in
presentation
of findings
“Who could
disagree”
item.
Democratic biased survey (2007)
70
Foundations of
Research
Summary: Manipulating attitudes by surveys
1. Ask manipulative or highly leading questions
Summary
2. Find high levels of agreement (and potentially
change participants’ attitudes).
3. Publicize – and often distort or overstate –
the “findings” via highly biased news
sources
4. News reports themselves lead to attitude
change among people who are uncertain or
uninformed.
71
Foundations of Psychology 242, Dr. McKirnan.
Research
 Topic areas & formats
 Testing Hypotheses
 General issues in Survey research
 Sources of bias (or fraud…)
 Examples of surveys
 By key elements of survey design
72
Foundations of
Research
73
Examples of surveys & data, 1
Consumer reports survey of mental health care
question
• Satisfaction
with therapy.
• Differences
between types
of therapy.
population
sample
data
findings
Self-Identified
group:
Self-selected
convenience
sample:
Attitudes &
behavior
Descriptive &
hypothesis
tests
U.S. mental
health care
users
Readers who
got therapy &
returned the
survey, n=4000
Self-report
questionnaire,
cross-sectional
High
satisfaction
for most
treatments
Foundations of
Research
74
Examples of surveys & data, 2
“Monitoring the future” youth studies
question
• Social
behavior
• Academics
• Alcohol &
drug use
• Health.
population
sample
data
findings
Demographic
group:
Random
sample:
Knowledge,
attitudes and
behavior
Mostly
descriptive
All U.S. youth,
15 -> 21 years
old.
Sample of High
School health
classes,
n=3000 -> 5000.
Face--to--face
interviews &
questionnaires,
longitudinal
(bi-yearly)
Assess yearly
trends/shifts in
drugs, grades,
emotional well
being
Foundations of
Research
75
Examples of surveys & data, 3
Gallup, Time/CNN, other polls
question
populations
samples
data
findings
Demographic:
National,
random
Knowledge,
Attitudes,
Behavior
Descriptive
Digit dial
telephone,
n=150 to >500
Brief interview,
cross-sectional
Ratings of
politicians,
Consumer
preferences
Approach to
Affordable
Care Act
- Eligible voters,
Political
opinions,
Lifestyle
information
Social
attitudes, e.g.,
managed care
- Target age
groups
Self-identified:
“Democrats”…
Behavioral:
Voters, ACA
users…
General:
- U.S. adult
population
Foundations of
Research
76
Examples of surveys & data, 4
Exit polls
question
Election
outcome,
possibly
stratified by
state / region
population
sample
data
findings
U.S. electorate
Probability
Self-reported
behavior
Descriptive/
predictive
National and/or
local electoral
district
Stratified
random sample
of electoral
districts.
Self-report
interview,
cross-sectional
Increasingly
inaccurate
predictions
See reading on shifts in use of polling data in U.S. politics
Foundations of
Research
77
Examples of surveys & data, 5
”Social Issues Survey” of Chicago gay / lesbian
community
question
- Stress &
coping
- Alcohol &
drug use
- Responses to
HIV / AIDS
population
sample
data
findings
Self-identified
Targeted multiframe
Attitudes &
behavior
Descriptive &
hypotheses
Self-identified
gay, lesbian, &
bisexual adults
in Chicago.
community
newspapers,
organizations,
& mailing lists,
n=3500
Self-report
questionnaire,
cross-sectional
- High
experience of
discrimination
- Less stress &
alcohol-drug
use than
expected
Foundations of
Research
78
Examples of surveys & data, 6
National Institute on Drug Abuse Household survey of
Alcohol and Drug use
question
Alcohol-drug
use and
problems,
treatment use,
health effects.
population
sample
data
findings
National
Random Multistage:
Knowledge,
attitudes &
behavior
Typically
descriptive
All U.S. adults
1. Census tract
2. Household,
3. Any adult
member
Face - to - face
Interview,
successive
cross-sectional
(each 5 years)
Age & regional
differences in
substance use,
trends over
time in use &
problems
n>4000
 Data often used for hypothesis-oriented 2nd
analyses (i.e., as archival data).