Transcript Data Collection Mode Effects Controlling for Sample

Data Collection Mode Effects
Controlling for Sample Origins in a
Panel Survey:
Telephone and Internet
Presented at the 2005 Annual Meeting fo the
American Association of Public Opinion
Research
Mike Dennis, Knowledge Networks
Cindy Chatt, University of Nebraska-Lincoln
Alicia Motta-Stanko, Knowledge Networks
Paul Pulliam, RTI International
Acknowledgements
The survey designed by:
RTI International
The Odum Institute at the University of North Carolina
The authors wish to thank RTI International for permission to
analyze the data used in this paper and disseminate the findings.
2
The Purpose of the Research
What matters more in accounting for variance in survey responses?
Mode of Data Collection (Telephone versus Internet)
or
Sample Origin (KN panel sample versus KN panel
rejecters/non-cooperators)
This is a contest of data collection mode versus sample quality.
The main purpose of this research is to help further our
understanding of the respective contributions of mode of data
collection (phone vs. web) and sample origin (cooperators vs. noncooperators) in explaining variance in survey response data.
3
Background of Study
The analysis is based on the Survey on Civic Attitudes and
Behaviors After 9/11.
Study was created with a non-response follow-up study
and implemented by Knowledge Networks from January to
March 2002.
The study is important because:
It is the first mode-effects study using KN data that
effectively controls for sample type (persons who
joined the KN panel and participate in KN studies,
versus those who refused to join the panel or who
refused to participate in the panel study).
4
Questionnaire: Public Policy & Civic
Attitudes & Behaviors
The survey questionnaire had several modules of
interests: ratings of Bush and Gore (early 2002!), attitudes
toward terrorism, the adequacy of governmental response
to bioterrorism and terrorism more generally, attitudes
toward what the government should be doing and would
do in the event of a terrorist attack, civic participation and
civic values questions, and background questions on
religious faith and other aspects.
The survey was approximately 25 minutes in length.
5
Main Finding
The mode of data collection is more often a significant
factor in accounting for variance in survey responses than
is the origin of the sample. When controlling for panel
experience and demographics, for every one question
where sample origin was a significant factor, mode of data
collection was significant for six questions.
MODE
Sample
6
Sample Design for the Study
Internet Sample
Panel Acceptors
Telephone Sample (Controls)
RDD Sample
Panel Rejecters
Non-Response Follow-Up Sample
(NRFUS)
Random sample for each of three groups.
7
Sample Composition, Mode, & Sample Size
Sample Size
Fielded
Completed
Interviews
Completion
Rate
KN Panel by Web
3627
2979
82%
KN Panel by
Telephone (Controls)
477
300
63%
Nonresponse1
(NRFUS) by
Telephone
2730
600
22%
Sample Group/Mode
1Nonresponse
sample consists of ‘panel rejecters’ and those
panelists that did not respond to the Internet survey.
8
Sample Representativeness:
Demographic Comparison
Each interviewed group was compared to the January
2002 Current Population Survey (Census)
Main Findings:
 Mean Error (Unweighted):
– Panel by Web:
– Panel by Phone:
– NRFUS by Phone:
2.8 percentage points
4.1 percentage points
3.6 percentage points
9
Summary of Multivariate Analyses: Count of Statistically Significant
Predictors of Answers by Mode and NRFUS (p < .05)
N
items
N
significant
for mode
N
significant
for NRFUS
Grade Bush's performance (attitudinal)
2
1
Worried about terrorism (attitudinal)
2
2
Information expected during bioterrorist event (attitudinal)
6
5
Sources from which anthrax information sought (behavioral)
6
6
Trusted source during bioterrorism event (attitudinal)
1
Feeling thermometers for Bush and Gore (attitudinal)
2
2
Important issues, politics, current events (behavioral)
3
3
Neighborhood statements (combination - attitudinal and behavioral)
13
9
1
Self-perception statements (attitudinal)
5
5
2
Volunteerism and/or donating behavior (behavioral)
4
2
44
34
6
100%
77%
14%
2
10
Results of Multivariate Analyses
Phone respondents more likely to:
 Disagree that bioterrorism is the most important problem
 Seek info on anthrax from …web, hotlines, national TV, own doctor, local
government, other
 Rate Pres. Bush and Al Gore higher on feeling thermometer
 Discuss politics
 Discuss community issues
 Help neighbors
 Be happy about their neighborhood
 Have pride in their neighborhood
 Have a sense of belonging in neighborhood
 Have a neighborhood that gives them pleasure
 Rely on neighbors
 Trust others
 Enjoy mixing socially with others
11
Freq Distributions by Group (Unweighted)
Question
Panel by Panel by
Internet Telephone
NRFUS by
Telephone
Rate George Bush for job done as
President (% ‘A’ grade)
42
40
40
Grade Bush on dealing with terrorism
57
57
51
(% ‘A’ grade)
No large differences across the groups for
grading political job performance.
12
Freq Distributions by Group (Unweighted)
Question: When a bioterrorism event
Panel by Panel by
happens, what info do you expect CDC Internet Telephone
to provide the public?
NRFUS by
Telephone
Type of GERM (% Yes)
93
94
96
Name CRIMINAL Suspects (% Yes)
38
60
60
No. of people dead, ill, etc. (% Yes)
71
85
88
Explanation of steps people should
take to minimize risk (% Yes)
91
97
98
Recommendations for treating illness
caused by germ (% Yes)
88
97
97
Travel advisories/restrictions (% Yes)
70
89
89
Telephone mode: Higher expectations for
information from CDC.
13
Freq Distributions by Group (Unweighted)
Question:
Sought info on ANTHRAX or other
SEVERE ILLNESSES from…
Panel by Panel by
Internet Telephone
NRFUS by
Telephone
Local radio and TV
46
43
40
WEB health and news sites
12
26
17
Phone HOTLINES
1
5
6
26
46
43
Own Physician or Health Care Professional
4
10
12
Local or State Health Department
3
10
11
Cable 24-hour News and Network News
Telephone mode: Higher report of
information-seeking behavior.
14
Freq Distributions by Group (Unweighted)
Question:
Within the past year, how often have
you…
Borrowed, loaned, etc, small things
with neighbors like sugar
Panel by Panel by
Internet Telephone
NRFUS by
Telephone
39
43
46
60
71
70
(% Sometimes & Often)
Helped people in this neighborhood or
they helped you with small favors
(% Sometimes & Often)
Telephone mode: Higher report of ‘good
neighbor’ behavior.
15
Freq Distributions by Group (Unweighted)
Civic Self-Perception Statements
(% Completely Agree)
Panel by Panel by
Internet Telephone
NRFUS by
Telephone
I am happy to live in this neighborhood
35
49
57
See myself as part of neighborhood
21
37
40
Feel sense of belonging to
neighborhood
22
38
42
Being in neighborhood gives me a lot of
pleasure
23
39
43
Would get my neighbors to fix problems
9
23
22
Telephone mode: Higher report of positive
feelings toward own neighborhood.
16
Freq Distributions by Group (No weight)
Sociability Self-Perception Statements
(% Completely Agree)
Panel by Panel by
Internet Telephone
NRFUS by
Telephone
I am trusting of others
9
23
22
I easily fit into groups
9
28
32
I like to mix with others
13
31
39
I tend to be a happy person
13
31
39
I enjoy helping others
23
41
46
Telephone mode: Higher report of positive
sociability self-perceptions.
17
Discussion
Major Differences in Response Caused by
Mode?
Major differences existing between Internet and telephone
modes of data collection include:

Presence or absence of interviewer

Dependence on visual or aural communication

Interviewer or respondent control of interview pace or
information sequence
Each difference can lead to multiple mode effects
seen in the data.
(Dillman, Sangster, Tarnai, and Rockwood, 1996)
19
Presence or Absence of Interviewer
Can lead to:
 Tendency to answer on extreme ends of response
scales
 Non-differentiation
 Social desirability / self-presentation bias
20
Presence or Absence of Interviewer:
Neighborhood Statements
1. I am happy to live in this neighborhood.
2. I really see myself as a part of this neighborhood
3. I feel a sense of belonging to this neighborhood.
4. Being in this neighborhood gives me a lot of pleasure.
5. If there are things in my neighborhood that need to be fixed or improved, I
would be able to get my neighbors to do something about it.
21
% Average Response by Sample Group:
Neighborhood Questions
100.0%
90.0%
79.6%
76.9%
% of Average Response
80.0%
70.0%
66.2%
Positive
60.0%
Negative
50.0%
Neither
40.0%
Don't Know /Refused
30.0%
20.0%
16.9%16.5%
10.6%11.6%
9.9% 10.3%
10.0%
0.4%
0.8%
0.2%
0.0%
Internet
Telephone
NRFUS
Sam ple Group
Very similar results for self-perception items (e.g., “I am trusting.”)
22
Evidence of Non-Differentiation and Recency
Effects on Phone (Neighborhood Statements)
Response: Completely Agree (+5)
Internet
Telephone
NRFUS
All same Answer
4.9%
16%*
13.8%*
4 out 5 Same
10.2%
14.8%*
17.3%*
Response: Neutral (0)
Internet
Telephone
NRFUS
All same Answer
2.1%
0.1%
1.4%
4 out 5 Same
2.5%
1.0%
2.4%
Response: Completely Disagree (-5)
Internet
Telephone
NRFUS
All same Answer
2.0%
1.1%
1.0%
4 out 5 Same
1.6%
0.4%
0.9%
*Significant differences at p<.05 between Internet and
Telephone and Internet and NRFUS for Completely Agree
23
Conclusions
Sample origins do not appear to be having much of an effect on
results compared to mode of data collection
Differences in the data are consistent with inherent differences
between Internet and telephone data collection, leading to patterns in
the phone-collected data reflecting:




Social desirability bias
Extreme positive responses on answer scales
Answer distribution differences due to communication type differences
Item Non-differentiation
Study limitation/Buyer Beware: Questionnaire potentially laden with
more questions (compared to many surveys) susceptible to social
desirability bias.
24
Thank you!
[email protected]
(650) 289-2160