Transcript Response Rates & Results of Advance Letter Experiment

Response Rates and Results of the
Advance Letter Experiment
2004 RRFSS Workshop
Toronto, June 23, 2004
David A. Northrup, Renée Elsbett-Koeppen and
Andrea Noack
ISR, York University
Outline
general comments on response rates
how response rates are calculated
 a very brief history of response rates
 what strategies have/are being put in
place to deal with declining response
rates

Outline
(continued)
response rates and RRFSS

what did it take to get the 62% rate for
2003 RRFSS
 number
of calls
 refusal conversions

results of the advance letter experiment
Calculating Response Rates
 Completions / estimate of number of eligible
households (HH)

eligible HHs include completions, refusals,
callbacks, and a % of the “never answered”
 ISR method same as BRFSS,

aka “CASRO 3”
 RRFSS 2003 = 62%, exclude callbacks =
71%
Response Rates for American
Election Study
100%
80%
60%
40%
20%
0%
1952
1956
1960
1968
1972
1976
1980
1984
Response Rates for BRFSS
80%
60%
median
mean
40%
20%
0%
1994 1995 1996 1997 1998 1999 2000 2001 2002
Strategies for Improving
Response Rates
 interviewer training
 increase call attempts
 “convert” refusals
 use advance letters
 payments (as a lottery, to completers, to the
whole sample)
Data Collection at ISR for
RRFSS Response Rates
 minimum number of 14 calls (more
when there is reason to think extra calls
might obtain a completion)
limitation of one month sample release
 costs about 3 to 7 points on response rate

 at least one attempt to convert almost all
refusals
390,106
RRFSS: Fun with
Numbers 1 (2003 Data)
 number of calls: 390,106
 percent of interviews completed first call:
21
 number of interviews completed on the
10th or subsequent calls: 3,158
 number of interviews completed after a
refusal: 2,678
RRFSS: Fun with
Numbers 2 (2003 Data)
 average number of calls per completed
interview: 4.65
 most calls made for a single completion:
33 (for two (different) interviews)
 response rate if 10 plus calls and refusal
conversions are dropped: 48.2%
 number of complaints about interviewer
calling registered at ISR: 13
Characteristics of Refusers:
2003 RRFSS Data
variable
standard converted
mean age
education: > than high school (%)
university (%)
% employed
47.99
16.7
47.8
61.7
53.86
23.0
39.2
52.4
% saying health fair or poor
11.8
16.5
% doctor told high blood pressure
22.3
27.1
% smoke 100 cigarettes
52.5
55.0
# of cases: standard = 24,700, converted = 2,640
all differences significant
Characteristics of Easy and Hard
to Reach: 2003 RRFSS Data
variable
mean age
education: > than high school (%)
university (%)
% employed
% saying health fair or poor
% doctor told high blood pressure
% smoke 100 cigarettes
easy to
reach
50.43
18.6
45.6
54.8
13.8
25.0
52.7
hard to
reach
43.50
12.5
52.8
73.6
8.1
16.8
50.6
# of cases: easy = 17,000, hard = 3,150
all differences significant
Letter Experiment: 1
 six Health Units participated (Durham, London,
Grey Bruce, Halton, Waterloo, Sudbury)
 test two versions of letter: ISR and HU
 needed to work with our monthly target and
wanted to acknowledge random variation in
response rates per HU per month
 used sample “replicates” to implement
experiment
Letter Experiment: 2
 Month one: replicate 1, ISR letter;
replicate 2, HU letter; replicate 3, and 4
(when used), control group
changed presentation in months 2 and 3
 copy of letter at the end of this set of
handouts
 exactly the same text, different letterhead,
signature & envelope

 Except
Halton
Letter Experiment: 3
 survey introduction exactly the same
except one additional sentence

“Recently, we sent a letter to your
household about an important research
project.”
 questions about the letter the same

except Durham
Why the Letter Might Improve
Response Rates to RDD Surveys
 reduces the possibility that the telephone call
catches people by surprise
 increases legitimacy of research project in
the eye of the potential respondents
 demonstrates social value
 improves the confidence of the interviewer
Why Advance Letters Might
Not Improve Response Rates to
RDD Surveys
 letter does not reach, or is not read by,
respondent
 ceiling effects
 survey topic & subpopulations
 they give “timid” participants a chance to
prepare to say “no”
Response Rates for Months
1 & 2 of the Experiment
66%
62%
63%
70%
70%
50%
30%
letter
no letter
ISR letter
HU letter
p value =.035 (for letter (1,200) versus no letter (1,345))
p value =.025 (ISR (600) versus HU (600))
Response Rates Months 1 & 2:
All Six Health Units
80%
60%
none
ISR
HU
40%
20%
0%
Durham London
Grey
Bruce
Halton Waterloo Sudbury
See next slide for numbers & p values
RR by HU and Treatment
treatment, RR (%)
P value
HU
none
ISR
HU
none/
ISR
none/
HU
ISR/
HU
Durham
58.9
63.6
65.3
.422
.275
.803
London
57.8
64.5
69.2
.254
.048
.475
Grey
70.5
73.2
78.7
.618
.128
.367
Halton
57.5
57.0
68.7
.925
.053
.087
Waterloo
64.7
60.2
65.8
.441
.851
.416
Sudbury
69.8
63.2
71.4
.255
.776
.213
Number of cases per HU: ISR = 50, HU = 50, none = 100
Mean Calls per Completion
mean # of calls
P value
Durham
None
5.89
ISR
5.69
HU
5.57
none/ none/ ISR/
ISR HU HU
.825 .721 .905
London
6.41
6.06
4.43
.751
.049
.060
Grey
4.36
4.58
4.69
.763
.622
.896
Halton
5.18
6.42
6.53
.157
.124
.935
Waterloo
4.82
6.42
4.38
.049
.510
.047
Sudbury
5.95
5.14
5.42
.445
.597
.809
Number of cases per HU: ISR = 50, HU = 50, none = 100
Mean Calls per Completion by
Letter Status
6
5
4
# of calls 3
2
1
0
letter
no letter
ISR
HU
not see
letter
saw
letter
Letter/no letter p = .890, ISR/HU p = .230, not see/saw p = .001
% of First Call Attempts Leading to
Completions & Refusals
15%
10%
comps
ref
5%
0%
letter
no letter
ISR letter
HU letter
letter/no letter p =.204, ISR/HU p =.008
At the Start
of the Interview
35%
38%
40%
27%
20%
0%
R told Interviewer
they saw letter
R asked about
letter
"standard" survey
introduction
Awareness of Letter
Variable
(based on 602 cases)
R indicated saw letter at intro
R indicated letter came to house
total respondents aware of letter
personally read the letter
got more info (web site, 1-800)
letter made a lot of difference to
decision to participate
total
30
21
51
40
1
26
Data Characteristics
Variable
ISR
HU
none
P
(n=300)
(n=300)
(n=620)
year of birth
1955
1954
1954 .820
male (%)
employed (%)
58
59
56
59
57
60
.800
.627
health excellent
22
22
22
.951
smoked at least
100 cigarettes (%)
50
56
56
.197
Costs: Month One
 cost of materials: $314; staff cost: $1,919
 total: $2,233
 per case cost: $3.62
 buys 72 interviews or 12 per HU
 need to estimate savings from making
fewer calls, and making fewer refusal
conversion calls
Conclusions
 HU letter (seems to) increase response and
warrants consideration as a tool to improve
RRFSS response rates
 affect on variable distributions minimal, but
small sample size limits scope of
examination
 social-political distance between respondent
and sender probably matters
 letters may have value other than just
increasing response rates
Questions