Transcript Introdution to Survey Measurement Quality

Introduction to
Survey Quality
Paul P. Biemer
RTI International
Lars E. Lyberg
Statistics Sweden
Course Content
• Concepts of data quality
• Survey measurement
process
• Coverage error and
nonresponse
• Survey instrument
• Interviewers and interviewing
• Mode and Setting
• Data processing
• Evaluation methods
• Practical implications
The Evolution of Survey
Process Quality
Chapter 1
Concepts
• Survey
• Survey methodology
• Quality
• Survey quality dimensions
• Survey process quality
• Quality assurance
• Quality control
• Error sources
• Mean squared error
The Concept of a Survey
• concerns a set of objects
comprising a population
• population under study has one
or more measurable properties
• goal is to describe the
population by one or more
parameters defined in terms of
the measurable properties
The Concept of a Survey
(con’d)
• access to the population
requires a frame is needed
• sample is selected in
accordance with a sampling
design specifying a probability
mechanism and a sample size
The Concept of a Survey
(con’d)
• observations are made in
accordance with a
measurement process
• based on the measurements
an estimation process is
applied to compute estimates
• purpose is to infer to the
population
Typical Shortcomings
• target population is changed
during the study
• selection probabilities are not
known for all selected units
• correct estimation formulas
are not used
Types of Surveys
One-time
Repeated or continuing
The survey environment
The survey infrastructure
A Brief History
• Biblical censuses
• Political arithmetic 1650-1800
• The 1895 ISI proposal
regarding representative
investigations
• Bowley argues for random
sampling 1913
• The 1934 Neyman paper on
the representative method
• Neyman develops theories
for sampling and confidence
intervals
• Nonsampling error theory in
the 1940s
• Interpenetration
• The US Census Bureau
survey model
• Developments in other
disciplines
 Questions and
interviewers
 The response process
The Quality Revolution
• Deming’s 14 points
• Juran’s spiral of progress
• Ishikawa’s 7 quality control
tools
• Joiner’s triangle (quality,
scientific approach, teamwork)
• Shewhart’s control chart for
process control
• Dodge and Romig’s
acceptance sampling
• A theory for statistical
process control
Definitions of Quality
Fitness for use
Quality of design
Quality of conformance
Quality dimensions in official
statistics (one of them is
accuracy)
Quality according to some
business excellence model
Performance indicators
Eurostat’s Quality
Dimensions
Relevance of statistical
concepts
Accuracy of estimates
Timeliness and punctuality in
disseminating results
Accessibility and clarity of the
information
Comparability
Coherence
Completeness
The Process View
Product characteristics are
established together with the
user
The quality of the product is
decided by the processes
generating the product
The processes are controlled
via key process variables
Measuring and
Documenting Quality
• Accuracy can be measured
• Other quality dimensions are
qualitative and can be seen as
constraints
• Quality profiles
• Quality reports
• Performance measures
Examples of Tools
• Self-assessment via excellence
model
• Checklists
• Quality management
• External auditing
• Customer satisfaction surveys
Improving Quality
• Changing processes
• Project teams
• Standardization via current
best methods documents
• Development of quality
guidelines
We Concentrate on
Accuracy
Data must be of sufficient quality for
decision-making
Other dimensions are constraints
Accuracy is much more difficult to
understand
It is important to convey information
on error sources and their
contributions to total survey error