Transcript PPT - CORDIS

How representative are
the samples?
Sabine Häder and Peter Lynn
European Social Survey (ESS) –
Launch Conference
Brussels, 25/26 November 2003
Sampling: Important part of a survey!
Precondition for comparability of countries
What do you need for achieving high-quality
comparable samples?
In an ideal world you have
complete frames, from which you can select
randomly (with known probabilities)
many individuals
who all like to respond.
Häder/Lynn
Representativeness of Samples
2
..... Sounds easy.....
But: Could be found only in a minority of
ESS countries
In reality:
Considerable variation between countries
in constraints (e.g. availability of sampling
frames) and survey practice
Häder/Lynn
Representativeness of Samples
3
Task: Minimising the impact of this variation
Requirements for sample designs:
The same population (residents 15 years and
older)
Same precision of results (Effective sample
size of 1.500)
Highest achievable response rates
Recording sample design characteristics (e.g.
inclusion probabilities)
Häder/Lynn
Representativeness of Samples
4
Organisation of the work package
Expert Panel
Sabine Häder
(Centre for Survey Research and Methodology, Germany)
Siegfried Gabler
(Centre for Survey Research and Methodology, Germany)
Seppo Laaksonen
(Statistics Finland)
Peter Lynn
(University of Essex, U.K.)
Häder/Lynn
Representativeness of Samples
5
Guidance:
Identifying suitable sampling frames (e.g.
Austria)
Developing new sample designs (e.g.Greece)
Maximising response rates (e.g. Switzerland)
Implementing concept of design effects (e.g.
France)
Finally: ‚Signing off‘ the sample designs
Result: Best probability samples in all countries
—› As close to the ideal world as possible
Häder/Lynn
Representativeness of Samples
6
Variation in procedures to achieve
equivalence of outcomes: 2 examples
Advances in survey design and practice: 2
examples
Use of design weights in data analysis
Some lessons for the future
Häder/Lynn
Representativeness of Samples
7
Variation in procedures to achieve
equivalence of outcomes:
Example 1: Age range 15+ (no upper cut-off)
Address-based samples (e.g.UK, Greece)
Population register samples (e.g. Finland,
Denmark)
Electoral register sample (Italy)
Häder/Lynn
Representativeness of Samples
8
Variation in procedures to achieve
equivalence of outcomes:
Example 2: Effective sample size 1,500+
Equal probability unclustered samples (e.g.
Finland, Denmark)
Variable probabilities due to address-based
sampling (e.g. UK, Greece, Portugal)
Variable probabilities due to local aims (e.g.
Luxembourg)
Clustered samples (most, but to different extents)
Häder/Lynn
Representativeness of Samples
9
Example 2: Effective sample size, continued
Design effects
Clust
Add
Other
Tot
1.40
1.00
1.00
1.40
Equiv.
precision
2,100
Netherlands 1.00
1.19
1.00
1.19
1,785
Germany
1.38
1.00
1.10
1.52
2,280
Ireland
1.20
1.33
1.00
1.60
2,400
Israel
1.22
1.29
1.01
1.59
2,385
Sweden
1.00
1.00
1.00
1.00
1,500
Slovenia
Häder/Lynn
Representativeness of Samples
10
Advances in survey design and practice:
Example 1: National probability sample of
households in Greece
Probability samples previously used only by NSI
No list of persons or addresses available to other
organisations
Quota sampling common
Exclusion of rural areas and most islands
common
Häder/Lynn
Representativeness of Samples
11
Advances in survey design and practice:
Example 1: Greece (continued)
Area-based probability sampling approach
Census EAs as PSUs
Field listing of addresses
In-office selection of addresses from the list
Carried out successfully - and good response
rate!
Häder/Lynn
Representativeness of Samples
12
Advances in survey design and practice:
Example 2: Inclusion of 15-17 year-olds in
electoral register based sample in Italy
Equal probability sampleof electors (18+)
Interviewer lists all residents 15+ (r) at address
of selected elector and randomly selects one
Interviewer also asks number of electors at
address (e)
Selection probability is Ke/r: all persons 15+
have known selection probability
Häder/Lynn
Representativeness of Samples
13
Design weights
In most countries, selection probabilities were
unequal
We ensured that the probabilities are known
We have converted these into a „design weight“
that should be used in all analyses
e.g. address-based samples (Greece, UK,
Portugal ...): probability is inversely proportional
to number of persons in household
Häder/Lynn
Representativeness of Samples
14
Design weights: Example
CZ
ES
GR
FI
SE
V1, unw
25.7
14.9
18.7
22.9
21.1
V1, wtd
11.9
6.1
8.0
22.9
21.1
V2, unw
26.3
24.2
38.0
13.8
11.6
V2, wtd
23.4
22.7
35.8
13.8
11.6
V1 = Single-person household (%)
V2 = 3+ hrs/week watching TV (%)
Häder/Lynn
Representativeness of Samples
15
Lessons for the future
Central co-ordination of sample design and
close liaison with national teams necessary
Necessary not only to „agree“ the design, but
also to monitor and discuss implementation
Central team must be seen as „helping,“ not
„controlling“
Specification of effective sample size was
successful
Methodological research into sample design
Häder/Lynn
Representativeness of Samples
16