Transcript Q2008 - ISTAT

Understanding the Decision to
Participate in a Survey and the
Choice of the Response Mode
Anders Holmberg and Boris Lorenc
European Conference on Quality in Official
Statistics Rome, Italy July 11, 2008
Content
• Introduction to the problem
• Background
• The design of the study
– goals
– setup
– results
• Summary and tentative conclusions
European Conference on Quality in Official Statistics Rome, Italy July 11, 2008
2 of 17
Mixed Mail-and-Web Surveys
• Swedish individual and household surveys
with Self-Administered Questionnaires (SAQ)
• The problem: low use of the web alternative
– belief in rationality of the web mode (editing, other
quality aspects, cost, environment)
– cannot publicly afford the Web mode only
– possibly, a mode switch (more of which later)
– goal: finding a solution with an increase in the
proportion of web respondents and a cost
reduction, without harming the response rate
European Conference on Quality in Official Statistics Rome, Italy July 11, 2008
3 of 17
Background
• PAP (mail Q) as the standard mode for SAQ
– cover letter, questionnaire, return envelope, in some cases
separate instruction, incentives.
– response rates – as elsewhere – fall gradually
• Web as an alternative mode for SAQ
– introduced in the 2000’s for ind/hh surveys in Sweden
– with the expectations of; easier input editing, higher
response rate, cost reduction, and greater acceptance
– carried out by including a paper sheet in the above, with an
URL and login data
• Reminders
European Conference on Quality in Official Statistics Rome, Italy July 11, 2008
4 of 17
Background (cont’d)
• Web as an alternative mode for SAQ (cont’d)
– its use is optional, according to preference of external
customer or internal statistical product
• The expectations were not fulfilled
– mostly due to the web’s low use: in mail-and-web mixed
mode surveys around 10-15% of the responses are collected
through the web
– no previous study of cost done, but impression of survey
managers is that the costs are higher in the mixed mode
than in the single paper mode. (Development cost of single
use SAQ:s)
• Reasons?
European Conference on Quality in Official Statistics Rome, Italy July 11, 2008
5 of 17
The Parent Survey
• Entry on the labour market after
•
•
•
graduation.
Proportion of NR as well as web
respondent were higher for men than
women, (37% vs. 28%) and (23% vs. 11%)
The proportion of NR is highest in the
lowest and highest income class.
No observed differences in choice of mode
with regards to age and immigrant
background.
European Conference on Quality in Official Statistics Rome, Italy July 11, 2008
6 of 17
Design of the Study
• A study was initiated to understand the
reasons for the low proportion of web
responses
– conducted as a telephone follow-up
– a simple random sample among the three groups
in the parent survey: web respondents (W), paper
respondents (P) and nonrespondents (NR)
– sample sizes of about 310 in each group; the
response rates 77%, 83% and 61% respectively
European Conference on Quality in Official Statistics Rome, Italy July 11, 2008
7 of 17
Data collection results from
interviews
NR
P
W
Refusals
8%
2%
2%
4%
Not
accessed
Response
proportion
32%
16%
21%
23%
61%
83%
77%
74%
Total
100%
100%
100%
100%
European Conference on Quality in Official Statistics Rome, Italy July 11, 2008
Total
8 of 17
Design of the Study, the
Interview questionnaire
•
Specific questions to each group of respondents
•
Was the web alternative observed?
•
Why/why not use the web questionnaire?
•
Preferences regarding choice of response mode
•
A follow-up question to the NR-group on their
status of occupation.
European Conference on Quality in Official Statistics Rome, Italy July 11, 2008
9 of 17
Results Web-mode visibility
• The main results
– 80% of paper respondents saw the web mode.
Roughly 65% of paper respondents saw the web
mode alt. in the first mail-out. Roughly 55% of the
NR-group saw the web mode in the first mail-out.
– 75% of the paper respondents who saw the web
mode alt. chose Simplicity of PAP as the reason to
hold onto the mail mode (as did 60% of those
paper respondents who, on another question,
ranked web as the mode of highest preference)
– 25% of the paper respondents who saw the web
alt. chose those alternatives that refer to
unavailability of computer or Internet (permanent
or temporal) as the reason
European Conference on Quality in Official Statistics Rome, Italy July 11, 2008
10 of 17
Results: Choice of paper or web
• In an optional open-ended question, 20% of
the paper respondents who saw the web
mode alt., gave circumstantial reasons for
using the PAP mode, namely immediate
availability of PAP
– security and privacy do not seem to be a concern
• Web respondents
– Over 90 % of the web responders had the
simplicity of the web as their main reason for
choosing the web mode.
European Conference on Quality in Official Statistics Rome, Italy July 11, 2008
11 of 17
Results: The Nonrespondents
• > 60 % of nonrespondents declared specific
circumstances as their main reason for not
responding, i.e. lack of time, forgetfulness, etc
• Approx. 8% decided not to take part in the
parent survey as they had (fallaciously)
interpreted themselves as being outside the
target population.
European Conference on Quality in Official Statistics Rome, Italy July 11, 2008
12 of 17
Results: Question on mode of
preference
NR
P
W
Paper
17 %
53 %
1%
Web
52 %
36 %
90 %
Telephone
18 %
2%
1%
Visit
3%
1%
0%
Other, does not
matter
Total
10 %
8%
8%
100%
100%
100%
European Conference on Quality in Official Statistics Rome, Italy July 11, 2008
13 of 17
Respondent Suggestions to
increase web use
• Send survey invitations through e-mail
• Discard the paper questionnaire
• Make the web-option obvious and the only
alternative
• Provide incentives for web responses
• Have functioning technical solutions
European Conference on Quality in Official Statistics Rome, Italy July 11, 2008
14 of 17
Summary
• Observations
– The web option is not visible enough
– The mail mode is considered simple (easily move
around, change entries, break and continue)
– Circumstantial reasons: the ‘what is at hand’
principle
• Conclusions/changes
– Strategy change needed when SAQ is mailed
– Marketing of the web alternative
– Other technical solution when receiving web
questionnaires
– Construction environment.
European Conference on Quality in Official Statistics Rome, Italy July 11, 2008
15 of 17
Thank You!
Anders Holmberg and Boris Lorenc
[email protected]