Transcript Lilian_Survey Methodology for SOAR conf Nov 6 2014

Survey Methodology
Lilian Ma
November 6, 2014
Three aspects
•
•
•
•
1. How questions were designed
2. How data was collected
3. How samples were drawn
Probability sampling generally used.
Basic Characteristics/features
•
•
•
•
•
•
•
1. Their purpose
2. The population they try to describe
3. The source from which they draw samples
4. The design of the way they sample people
5. The use of interviewers
6. The mode of data collection
7. The use of computers in collection of answers
Thoughts to decide on features
adopted
• Think of survey as information source
• Compare various design features to see
how different survey design features
permit the surveys to achieve their
different purpose
Sampling Error
• Errors in statistics because of the omission
of some persons in the population
(1) Sample survey: CCAT AJC
Project on unrepresented Parties
•
•
•
•
•
•
•
•
•
•
•
•
Sponsor
CCAT AJC
Collector
CCAT AJC
Purpose
Main purpose: to self evaluate Access to Justice measure adopted
for the benefit of self represented parties appearing before tribunals and
agencies. To allow tribunals to check against checklist developed by attendees
at CCAT conference in 2012. To help agencies and tribunal to get information
needed for strategic planning
Year started
2014
Target population Tribunal and agencies chairs of all jurisdictions in Canada
Sample design: 64 questions grouped under 2 demographics and 14 areas
Sample size about 500 subjects
Mode of administration
Survey monkey, paper survey for testing and
promotion
Time dimension Fall 2014
Pilot project
Tested June 2014
Frequency
annual repetition
Levels of observation
tribunal/agency
(2)Sample survey: SJTO
Excellence Project
• Sponsor
SJTO
• Collector
SJTO
• Purpose
Main purpose: to self evaluate quality and performance
of SJTO and to collect information needed for strategic planning
• Year started
2014
• Target population
SJTO internal subjects and selected external
subjects
• Sample design COAT Excellence Framework combined with client
survey, stakeholder survey and staff survey
• Sample size about 500 subjects ( client survey about 500+ )
• Mode of administration
computer survey monkey, paper survey
• Time dimension Fall 2014
• Pilot project
March 2014
• Frequency
annual repetition
• Levels of observation
person
What is Survey Methodology SM
• SM seeks to identify principles about the design,
collection, processing, and analysis of surveys that are
linked to the cost and quality of survey estimates
• Quality is defined within a framework labeled the total
survey error paradigm. It is both a scientific field and a
profession.
• As a science, it requires multidisciplinary application of
– mathematics, statistics on sampling and inferences from sample
results to population results.
– Psychology re: memory, interview techniques
– Computer science for database design, file processing
Some important decisions
• How will the potential sample members be identified and selected?
• What approach to contact those sampled. And how effort devoted to
try to collect data from those who are hard to reach or reluctant to
respond?
• How much effort devoted to evaluating and testing questions that
are asked?
• What mode to pose questions and collect answers?
• How much effort devoted to checking the data file for accuracy and
internal consistency?
• What approaches used to adjust the survey estimates to correct fro
errors that can be identified?
The above decisions will affect the quality of estimates emerging from
the survey.
How surveys work to produce
statistical descriptions of population:
Inference and Errors in surveys, two
types of inference
• If -> mean inference
• And = means statistical computing,
• Then
Respondent answer to
questions -> characteristics of a
respondent = characteristics of the
sample -> characteristics of the population
Two inferential steps
• 1. Answers people give must accurately describe
characteristics of the respondents
• 2. The subset of persons participating in the
survey must have characteristics similar to those
of a larger population.
• When either of these conditions are not met the
survey statistics are subject to “error” meaning
deviation of what is attained to what is desired
outcome.
Errors
• “Measurement error” or “error of
observation” is the deviation of answer
given to a question from the underlying
attribute being measured
• “Errors of nonobservation” is the deviation
of a statistic obtained from a sample from
that of the full population.
SM is the study that makes survey
more or less informative
• Examine each error
• Study from a quality perspective
• Study all the survey design decisions