Transcript Mixed mode - NCRM EPrints Repository

Issues of Coverage, Sampling and Participation in
Mixed Mode Surveys
Peter Lynn, University of Essex
6th ESRC Research Methods Festival
Oxford, 08-07-2014
An initiative by the Economic and Social Research Council, with scientific leadership by
the Institute for Social and Economic Research, University of Essex, and survey
delivery by NatCen Social Research and TNS BMRB
Mixing Modes
“mixing modes gives an opportunity to compensate for the
weaknesses of each individual mode at affordable cost”
- de Leeuw (2005)
Involves an explicit trade-off between costs and (multiple
sources of) survey errors
Recent interest in mixed modes is particularly stimulated by the
(possibly false) notion that the marginal cost of data
collection by web is close to zero
Lynn | ESRC RMF, 8 July 2014
A Distinction
• Multi-mode (or multiple mode) data collection: Different
modes used for different survey items, e.g.
- CASI component within a CAPI survey;
- CATI follow-up to a mail questionnaire;
- etc
• Mixed mode data collection: The same survey items can be
collected by different modes for different sample members:
- sequential;
- concurrent selective;
- concurrent with respondent choice; etc
Lynn | ESRC RMF, 8 July 2014
Motivation for Mixing Modes
Motivation for mixed modes rather than a single mode:
• Cost reduction; or
• Coverage/ participation enhancement.
(Designs to achieve both simultaneously are proving elusive.)
Different motivations tend to suggest rather different designs:
• Combinations of modes;
• Sequence of modes (or non-sequential).
The coverage, sampling and participation issues may differ
between these types of design
Lynn | ESRC RMF, 8 July 2014
Sequential and Concurrent
Mixed Mode Designs
Sequential design:
•
Use a number of modes in sequence;
•
Get as much response as possible in one mode, before trying remaining
non-respondents in the next mode
•
Cheaper modes first if motivation is to reduce costs
•
Higher response rate modes first if motivation is to maximise participation
Concurrent selective design:
•
Offer a different mode to each of 2 or more subsets of sample members
Concurrent elective design:
•
Offer each sample member a choice of mode
Combination: e.g. Concurrent selective sequential design
Lynn | ESRC RMF, 8 July 2014
Sequential Design: Example
Phase 1:
•
Mail an invite to a web survey to all sample members;
•
Mail a reminder to those who have not responded after a week or two
Phase 2:
•
Mail a paper self-completion q’re to those who have still not responded
after a further week (or include this with the reminder above)
Phase 3:
•
Approach for a face-to-face interview those who have still not responded
after a further period
Key design choices: Which modes? Which order? Criterion for switching to
next phase?
Lynn | ESRC RMF, 8 July 2014
Coverage Issues with Mixed
Mode Surveys I
Mixed modes (of approach) can help address frame quality
problems. Examples:
• Inconsistent contact details on sampling frame: only address for
some people, only email for others, etc
• Dual-frame approaches: good coverage is provided only by the
union of multiple frames, which have different contact details
Note: Mixed modes of approach need not necessarily imply
mixed mode data collection
Lynn | ESRC RMF, 8 July 2014
Inconsistent Contact Details:
Example
Dutch GPSs (e.g. Labour Force Survey, ESS experiment)
•
Select addresses from Postal Address Register
•
Match phone numbers to addresses (via names)
•
70% match successfully: Can approach by phone
•
30% do not match: First approach face-to-face
•
Some of the addresses with phone numbers also require face-to-face
follow-up
Lynn | ESRC RMF, 8 July 2014
Dual-Frame: Example
•
RDD or list-assisted sampling to generate sample of phone numbers:
- These numbers are screened to identify households (with phones)
•
Supplementary sample of addresses:
- These addresses are screened (face-to-face) to identify households with
no (landline) phone
•
The two samples combined give good coverage
Lynn | ESRC RMF, 8 July 2014
Coverage Issues with Mixed
Mode Surveys II
Coverage issues can introduce constraints on mixed mode data
collection. Example:
• Desire is a sequential web → face-to-face design
• Not all sample members are web users
• Non-web users must skip the web phase, either explicitly or
implicitly
Lynn | ESRC RMF, 8 July 2014
Coverage Issues with Web
as a Primary Mode
A. Restrict survey to web users:
• Obvious cost advantages;
• Non-random under-coverage: requires evaluation and adjustment.
Or B. Include non-web users in web mode:
• Requires provision of hardware, software and training;
• Various models, e.g. LISS, KnowledgePanel, GIP, ELIPSS.
Or C. Include non-web users in a different mode…
Lynn | ESRC RMF, 8 July 2014
Coverage Issues with Web
as a Primary Mode, ctd.
Option C, including non-web users in a different mode:
• May have cost advantages compared to providing equipment –
depends on frequency/nature of data collection, etc;
• May have measurement disadvantages (see next two presentations!)
Various designs possible:
• Web + mail, based on mail-only approach;
• Interviewer-administered recruitment, followed by web+mail,
web+phone, or other mixes;
• More options are feasible in longitudinal context.
Examples: GESIS panel and Gallup panel are both web+mail
Lynn | ESRC RMF, 8 July 2014
Sampling Issues with Single
Mode Web
No general population frames with email addresses, therefore
first approach must be by a different mode.
For frames without names, mail approach requires selfadministered respondent selection:
• This is error-prone if paper-based and may cause dropout if
web-based.
Interviewer-administration preferred but costly.
Alternatively, use non-probability recruitment methods (opt-in
panels) and model-based inferential paradigm.
Lynn | ESRC RMF, 8 July 2014
Sampling Issues with Mixed
Mode including Web
Frames with partial information can be used (e.g. email
addresses for a subset) – but none yet in UK.
Instead, single-mode initial approach needed:
• Mail approach, with web+mail data collection, may offer lowcost solution of reasonable quality in some situations;
• In UK, this design may work with named-person frames such
as admin records, but is hampered by the need for
respondent selection for general population surveys;
• Interviewer approach and respondent selection may be
preferable, but only cost-effective for longitudinal surveys.
Lynn | ESRC RMF, 8 July 2014
Respondent Selection
Next / last birthday method:
•
European Social Survey (UK) Experiment (Villar, 2013)
•
Selection compared to birth date info from questionnaire;
•
Approx. 50% correct, 20% incorrect , 30% uncertain
•
(amongst households with 2+ adults)
Household roster / grid method:
•
Community Life Survey (Williams 2013)
•
Approx. 25% incorrect selections
→ Difficult / impossible to control who completes a selfcompletion questionnaire;
→ Chance of incorrect respondent may be greater with web
Lynn | ESRC RMF, 8 July 2014
Participation in Mixed Mode
Surveys
Response rates in single-mode surveys:
Face-to-face > telephone > self-completion
Amongst self-completion surveys:
Mail > web
– often, but not always
Composition of response:
Broadly similar between modes – typically
But some differences
Lynn | ESRC RMF, 8 July 2014
– typically
Participation in Mixed Mode
Surveys
Population
FTF Response
Web Response
Lynn | ESRC RMF, 8 July 2014
Participation in Mixed Mode
Surveys
Population
Mixed Modes Response?
Lynn | ESRC RMF, 8 July 2014
Increasing Response Rates
• Success requires:
- All (or most) people who would have responded in mode 1
continue to respond;
- Additional people respond too (in mode 2)
• But it is generally the case that:
- (Average) response propensity declines FTF → Phone →
Mail → Web
- Data collection costs decline in same order
- Refusal in one mode reduces propensity to respond in next
mode
Lynn | ESRC RMF, 8 July 2014
Examples of Increasing
Response Rates
• Examples are (all?) from surveys that use a subset of the
sequence FTF – Phone – Mail – Web
• And even then, it is also necessary to “exhaust” each mode
• E.g. 1 British Household Panel Survey: FTF → phone
- Approx. 93% response FTF alone; 96% FTF + phone
(amongst previous wave respondents)
Lynn | ESRC RMF, 8 July 2014
Examples of Increasing
Response Rates
• Examples are (all?) from surveys that use a subset of the
sequence FTF – Phone – Mail – Web
• And even then, it is also necessary to “exhaust” each mode
• E.g. 1 British Household Panel Survey: FTF → phone
- Approx. 93% response FTF alone; 96% FTF + phone
(amongst previous wave respondents)
• E.g. 2 British Crime Survey 2011 Follow-Up: Mail → web
- 60% response mail alone; 63% mail + web
(amongst BCS respondents)
Lynn | ESRC RMF, 8 July 2014
Examples of Failing to
Increase Response Rates
• All (?) surveys that use a sequence which is not a subset of
FTF – Phone – Mail – Web
• E.g. 1 UKHLS-IP 2009 wave 2: Phone → FTF
- 76% response FTF alone; 67% Phone + FTF (w1 resps)
Lynn | ESRC RMF, 8 July 2014
Examples of Failing to
Increase Response Rates
• All (?) surveys that use a sequence which is not a subset of
FTF – Phone – Mail – Web
• E.g. 1 UKHLS-IP 2009 wave 2: Phone → FTF
- 76% response FTF alone; 67% Phone + FTF (w1 resps)
• E.g. 2 UKHLS-IP 2012 wave 5: Web → FTF
- 84% response FTF alone; 79% Web + FTF (w4 resps)
Lynn | ESRC RMF, 8 July 2014
Examples of Failing to
Increase Response Rates
• All (?) surveys that use a sequence which is not a subset of
FTF – Phone – Mail – Web
• E.g. 1 UKHLS-IP 2009 wave 2: Phone → FTF
- 76% response FTF alone; 67% Phone + FTF (w1 resps)
• E.g. 2 UKHLS-IP 2012 wave 5: Web → FTF
- 84% response FTF alone; 79% Web + FTF (w4 resps)
• E.g. 3 UKHLS-IP 2012 wave 2: Web → FTF
- 85% response FTF alone; 81% Web + FTF (w1 resps)
Lynn | ESRC RMF, 8 July 2014
Examples of Failing to
Increase Response Rates
• All (?) surveys that use a sequence which is not a subset of
FTF – Phone – Mail – Web
• E.g. 1 UKHLS-IP 2009 wave 2: Phone → FTF
- 76% response FTF alone; 67% Phone + FTF (w1 resps)
• E.g. 2 UKHLS-IP 2012 wave 5: Web → FTF
- 84% response FTF alone; 79% Web + FTF (w4 resps)
• E.g. 3 UKHLS-IP 2012 wave 2: Web → FTF
- 85% response FTF alone; 81% Web + FTF (w1 resps)
• E.g. 4 NL-ESS 2009: Web → Phone/FTF (new sample)
- 52% response FTF alone; 46% Web + Phone/FTF
Lynn | ESRC RMF, 8 July 2014
Non-Response Bias
• Most (of the few) studies to date either found no effect or a
modest (assumed) positive effect
• Hope is that web may disproportionately add young, full-time
employed, busy people – who are generally underrepresented in surveys
• But very little evidence either way on this point:
- NL-ESS 2009 found almost identical sample composition in
the FTF-only and Web + Phone/FTF samples (age,
employment, education, etc)
- UKHLS 2012 too found no significant differences between
FTF-only and Web+FTF (age, gender, household type, etc)
Lynn | ESRC RMF, 8 July 2014
Mixed mode including Web:
UK Examples
Two cross-sectional surveys:
Community Life Survey (Williams 2013):
•
•
•
•
•
•
Random sample of addresses from PAF
Advance letter → Mail invitation → Mail reminder → Mail questionnaire
16% responded online + 11% mail = 27% response (no incentive)
19% responded online + 12% mail = 31% response (£5 conditional)
22% responded online + 13% mail = 35% response (£10 conditional)
25% responded online + 14% mail = 39% response (£5 unconditional)
European Social Survey (UK) Experiment (Villar, 2013):
•
•
•
Random sample of addresses from PAF
Advance letter → Mail invitation → Mail reminder → Face-to-face fieldwork
21% responded online + 18% face-to-face = 39% response
Lynn | ESRC RMF, 8 July 2014
Mixed mode including Web:
Longitudinal Examples
British Crime Survey re-contact study (Fong & Williams 2011):
•
Issued sample = 30% of BCS respondents who gave an email address;
•
Email invitation → Email reminder → Postal q’re → Postal reminder
•
35% responded online + 27% by mail = 62% response
Lynn | ESRC RMF, 8 July 2014
Web+Interviews: Examples
1958 Birth Cohort (Brown et al 2014):
• 9th wave (age 55) – 5 years after previous wave
• Sequential web → telephone
• 62% responded online + 21% phone = 83% response
Understanding Society (Jäckle et al, 2013):
•
•
•
•
•
Issued sample = wave 5/wave 2 of a household panel
Mail(+email) invitation → (email reminders) → Mail reminder → Face-to-face
21% responded online + 55% face-to-face = 76% response (w5 sample)
30% responded online + 50% face-to-face = 80% response (w2 sample;
higher incentives)
23% responded online + 55% face-to-face = 78% response (w5 sample, £10)
•
23% responded online + 51% face-to-face = 74% response (w2 sample, £10)
Lynn | ESRC RMF, 8 July 2014
Conclusions
Potential cost savings from mixed mode may erode if we:
- Aim for full population coverage
- Aim for response at least as high as could be achieved with
single-mode
Participation advantages of mixed mode likely to be costly
Most promising cost-quality trade-off in the UK currently may be:
- Web → Mail → Face-to-face
But … measurement concerns (next presentations!)
Lynn | ESRC RMF, 8 July 2014
Issues of Coverage, Sampling and Participation in
Mixed Mode Surveys
Peter Lynn, University of Essex
6th ESRC Research Methods Festival
Oxford, 08-07-2014
An initiative by the Economic and Social Research Council, with scientific leadership by
the Institute for Social and Economic Research, University of Essex, and survey
delivery by NatCen Social Research and TNS BMRB