Transcript How to conduct your survey: Self

POLI10251
Getting Quantitative: using
surveys in social research
Mark Brown
Social Statistics,
Ground Floor Humanities Bridgeford Street
Statisticians are Cool
Jean-Paul Benzécri: inventor of Multiple Correspondence Analysis
2
Challenging preconceptions..

I hate/can’t do statistics.. quantitative data can often
be analyzed with relatively simple techniques – you
don’t need to be a statistician.. or even very good at
math’s

Quantitative methods are only relevant for
‘Quantitative researchers’ The qualitative versus
quantitative debate is unhelpful. In the social sciences
evidence comes in numerous forms and you need to be
able to work with a variety of data. Many research
questions are best answered with a mixed methods
approach.
3
and developing quants skills is good for your CV!
‘QM are the most marketable transferable skill available
Graduates consistently report that ‘my QM skills got me
the job’
(HEFCE: Social Science by Numbers)
A note on quantitative data

We focus in this lecture on the social survey as one of
the most important sources of quantitative data in social
science research

But there are other types.. Notably administrative data
collected by many organisations e.g
– University collects data on students
– NHS collects patient records
– Police collect crime statistics
Valuable for monitoring, policy evaluation and research

5
The ubiquitous survey
government
media
surveys
charities
political parties
academia
commercial
measuring characteristics, outcomes, behaviours & attitudes
6
8 out of 10 Cats
7
Surveys making
headlines...
British Crime Survey
Surveys making
headlines...
General Household Survey
Health Survey for England
Expenditure and Food Survey
Expenditure and Food Survey
What are (quantitative) surveys?


A form of systematic data collection from a welldefined population of interest
They usually
– draw a sample
– Involve systematic and standardised data
collection: all respondents asked the same thing in
the same way and answer using standard
categories
– generate quantitative (numeric) data that can be
analysed using statistical methods
10
Systematic and standardised data collection:
Using tick boxes
Why not just ask respondents to discuss it in their own words?
11
Some potential advantages of the closed format
question...

Greater specificity of question and answer (in this case a series of
questions measuring attitudes on different aspects of inequality) can
generate richer data than just asking ‘what do you think about…?

Answers can be added up to give a quantitative measure of attitudes
in a population e.g. What percent of respondents think Government
should increase public spending on welfare

Crucially we can compare responses for different groups in the
population
e.g. We could look at whether the percent who supported increased
spending on welfare varied by age of respondent.. or education
level…
12
POLI10251: Increase spending on welfare benefits
even if it leads to higher taxes
POLI10251
40%
Agree strongly
0%
35%
Agree
Neither Agree nor
Disagree
26%
30%
35%
20%
Disagree
28%
15%
Disagree strongly
10%
Don't know
2%
25%
100%
10%
5%
0%
Agree
strongly
Agree
Neither
Agree nor
Disagree
Disagree
Disagree
strongly
Don't know
(58 cases)
13
Making comparisons
Compare POLI0251 with the rest of the nation (BSA 2012)
Increase spending on welfare benefits even if it leads to higher taxes?
POLI10251
National (2012)
Agree strongly
0%
6%
Agree
26%
29%
Neither Agree nor Disagree
35%
33%
Disagree
28%
27%
Disagree strongly
10%
5%
Don't know
2%
0%
100%
100%
(58 cases)
(2799 cases)
14
Making comparisons
Compare POLI0251 with the rest of the nation (BSA 2012)
Increase spending on welfare benefits even if it leads to higher taxes?
40%
35%
30%
25%
20%
POLI10251
15%
National
10%
5%
0%
Agree
strongly
Agree
Neither
Agree nor
Disagree
Disagree
Disagree
strongly
Don't know
15
Making comparisons:
differences by age of respondent (BSA 2012)
Increase spending on welfare benefits even if it leads to higher taxes?
100%
90%
80%
70%
(5) DISAGREE Strongly
60%
(4) Disagree
50%
(3) Neither
40%
(2) Agree
30%
(1) Agree strongly
20%
10%
0%
17 to 34
35 to 54
55+
16
Making comparisons
Change over time (BSA 2010 and 2012)
Increase
35.00% spending on welfare benefits even if it leads to higher taxes?
30.00%
25.00%
20.00%
2010
2012
15.00%
10.00%
5.00%
0.00%
(1) Agree
strongly
(2) Agree
(3) Neither
(4) Disagree
(5) DISAGREE
Strongly
17
Would you say that the gap between those with high incomes and those
with low incomes is too large, about right or too small?
100.00%
90.00%
80.00%
70.00%
60.00%
50.00%
(3) Too small
(2) About right
40.00%
(1) Too large
30.00%
20.00%
10.00%
0.00%
18
Getting critical: a question of validity

Always ask how well does a survey question measure
the concept of interest (construct validity)

Choice of question wording (and the answer categories
provided) are very important.

Consider whether we are ‘collecting data’ or ‘creating’ it
19
The search for pattern

Sure tick box answers oversimplify the diversity of viewpoint but this
may be necessary to identify patterns and relationships in the data
(a key aim in survey analysis). How would you go about this if
respondents all answered in their own words?

Qualitative questions (interviews and maybe some focus groups)
would reveal a more complex picture and be better tool for
understanding the reasons behind the patterns.

Used together, a powerful research design
20
Survey data is normally collected from a sample of the
target population
Want to know something about a population?
..It only takes a sample
generalise results back to
population (inference)
The Population
The Sample (from
which data is collected)
21
The power of sampling!
Mori Final Election Poll 2010
Sample <2,000
Population > 40mill !
Source: http://www.ipsos-mori.com
22
Representative samples

One of the great strengths of quantitative surveys is that, if
well designed, they can generate results that can be
generalised from the sample to the wider population

But you can only do this if you have a sample that is
representative of the population. Otherwise your results
will be biased and potentially misleading

Unfortunately many survey designs fall short and produce
biased samples that are not representative of the population

Even more unfortunately many users of surveys (especially in
the media) ignore this issue and assume that results from all
surveys can be generalised – this is BAD SCIENCE
23
What is a representative sample?


One where the composition of the sample (e.g. the
share who are male and female, of different age groups,
of rich and poor etc) is the same as in the population
i.e. It resembles a miniature mirror-image version of the
population
sample includes same
share of males and
females as in population
population
sample
24
Why does it matter?



The things we study in surveys (behaviours, attitudes, etc) will vary
according to characteristics of individuals.
E.g. Consider a survey of student use of social media, Let’s
suppose females use facebook more than males
If so, a sample with a higher share of females than in the population
(below) will over-estimate the true average time spent on face book
share of male v female
not representative of
population
25
population
sample
Getting a representative sample – harder than you think

Suppose you were asked to design and carry out a
survey to investigate the study behaviour of social
science undergraduates in Manchester (looking at
hours studied, % of lectures attended etc)

How would you get the sample?

How representative would it be?
26
Possible sampling strategies
(for a sample of 200)
Pros and cons ?
Strategy 1
Identify 1 large first year
lecture (>200) and ask
lecturer to let you
handout questionnaires to
the class



Strategy 2
Stand with a clipboard at
entrance to Arthur Lewis
and interview every 10th
student (ask screening
question first to check a
SoSS UG) until you get
200
Strategy 3
Get a list of all registered
UGs in SoSS from UG
office – randomly select
200 from the list –
contact sample by email
Which one the easiest to get?
Which likely to give highest response rate?
Which one will give most representative sample? (Which most
susceptible to bias)?
27
The science of sampling
Random (probability) samples

The ability to make ‘statistical inference’ to the population
(generalise our results from the sample) really demands the use of
random sampling

This is as the name suggests (think National Lottery numbers
drawn out of a hat, where everyone has chance of being picked) also called probability sampling

Strategy 3 on previous slide describes a classic random sample

Strategy 1 and 2 were Non-random samples and subject to bias
28
Sample Size.. The bigger the better

The other big issue in sample design is sample size – how big does
it need to be?

Results from survey analysis will be much more reliable if based on
a large number of cases.

This is one of the advantages of using existing large scale surveys
like British Social Attitudes ...
...and a frequent weakness of doing your own survey (where the
samples are often too small to support meaningful data
analysis/generalisation of results)
29
And a word on data collection

Need to distinguish between the sample we design and
the achieved sample (those in the sample who actually
take part in the survey)

Unfortunately non-response is a massive problem in
survey research (many surveys struggle to get 50%
response)
Can result in serious bias (people who respond are
generally different to those who don’t, so sample is
unrepresentative)

30
Recap: the value of good description

Standardised measurement + ability to generalise
findings to the population (inference) make well
designed surveys powerful tools for accurate
description of patterns in society

don’t under-value the importance of good description in
research – We need to know the nature and extent of
differences in society before we can set about asking
why they exist or how to tackle them.
… particularly valued in the current climate of
evidence-based policy research.
31
Describe and compare – across groups and over time
Obesity in Scotland, by age and sex (1995-2005)
Source: Scottish Health Survey, Scottish Government
32
More than good description:
survey analysis can be used to test theory

Survey analysis can be much more than good description

e.g. consider contested theories on what factors are driving an
‘obesity crisis’ Lifestyle.. Related to culture… or deprivation?

Can develop hypotheses from these theories and then test them
using survey data. E.g. we could start by crosstabulating obesity
levels against income.. or any other variable you think may be
important

though be careful… a statistical relationship does not necessarily
imply cause and effect e.g. It could be another ‘third factor’,
perhaps education level that is separately influencing both income
33
and lifestyle factors related to obesity
Questions of Causality
Lifestyle
factors
related to
obesity
Income
Education
level
Should I do my own survey?
....or use someone elses.


Doing your own quantitative survey is hard to do well
Common problems
– Unrepresentative sample designs
– Inadequate sample size
– Questionnaires – much harder than you think

The good news is that we have fantastic secondary
resources of survey data...
(UK Data Service http://ukdataservice.ac.uk/)
– Large representative samples
– Rich data on topics you are interested in
– Never been more accessible
35
Access to survey data – it’s never been easier
and you don’t always need to do your own analysis

Re-purpose existing published tables and charts

Generating your own tables and graphs using
survey data on-line

Downloading the dataset and doing your own analysis
on your pc – perfectly possible but need training in data
analysis (modules in year 2 and 3)
36
Sourcing evidence from British Social Attitudes
(BSA) onlinewww.Britsoc.com
37
Sourcing evidence from British Social Attitudes
(BSA) onlinewww.Britsoc.com
Tomorrow’s practical class will show you how to access
tables of data for use in essays and project work

Please complete on-line registration to use site before
you come

Please come to the correct slot 4-5 or 5-6 (see list)
38