Transcript slides - ReStore

Non-response weighting at
NatCen/ScotCen
- a brief (and biased) history
Susan Purdon
Survey Methods Unit, NatCen
The case against
• 10 years ago non-response weighting not
the norm
Reasons:
• Response rates pretty high on govt
sponsored surveys
• Beyond age and sex, not many control totals
around
The case against
• Many surveys had long histories: weighting
would introduce discontinuities
• Non-response is subjective and cosmetic!
• No two statisticians would create the same
set of non-response weights. Unscientific.
• Weighting makes analysis more complex
and error prone.
The counter-arguments
• Non-response rates have now grown (c.
1%/year)
• The under-representation of certain groups
is a constant and clearly biasing
• Trends in response rates undermine the
argument against introducing discontinuities
• ONS have concluded that all national
statistics should be calculated on a
consistent basis (same age-sex-region distn)
The implications
• Most surveys now come with non-response
weights
• New industry of calculating weights for old
surveys
• Not too comfortable a position (what if new
method around the corner?)
How it’s done on gen pop
surveys
• Approach kept as simple as possible. Adjust
(standardise) for age-sex; no major attempt
to eliminate other biases.
• Usual approach = calibration weighting
where
(a) adjust to national age-sex totals; but
(b) give all household members same weight.
Calibration weighting
• Biases observed for individuals, but nonresponse is a household process (yes/no
determined by whoever answers the door);
• Implies that probability of response depends
upon who you live with (e.g. young men
living with parents v. other young men)
• So only under-represent young men in
certain types of households
Calibration weighting
• Calibration weighting attaches a probability
of selection per household that ‘explains’
difference between survey and population
age/sex.
• Assumes that non-response is not an
independent process
More elaborate approaches
• Non-response hierarchies internal to the
survey
• Panel attrition
• Studies with informative sampling frames
Suggestions for analysts
• Non-response weights are subjective. You
don’t have to trust them.
• Check that observed differences are not
attributable to weighting
• If you think the survey organisation has
missed a trick then tell them!