Transcript MSc ASR: SR04 Lecture 1, Introductory data analysis (part 1)

MSc ASR, SR06 Session 9
Quantitative methods of social
research for cross-national
comparisons
Paul Lambert, 5.2.02
http://staff.stir.ac.uk/paul.lambert/teaching.htm
1
Quantitative cross-national social
research
1) Introduction
2) Three traditions in Qn cross-national research
3) Seven themes in Qn cross-national research
Case study 1: Secondary analysis of cross-national
surveys
2
Introduction: Formats of
Quantitative Cross-National
research
Aside: cross-national  between country
cross-national  comparative
 But in quantitative methods, ‘XN’ &
‘comparative’ often used interchangeably
3
QDA: Analysis of patterns of relationships
between variables in the variable-by-case matrix
[Low # of vars; stats / graphical summaries]
Cases 
1
2
3
4
.
N
1
1
2
2
.
17
18
17
18
.
 Variables
1.73 A
.
1.85 B
.
1.60 C
.
1.69 A
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
4
A convenient distinction
Macro-social data
Micro-social data
Work and/or report Work and/or report
at level of
at level of
aggregated unit
constituent unit (eg
(country)
individuals)
Micro-macro distinction isn’t always important (&
can be confusing). But is widely used, & tends to
be associated with different research fields.
5
a) Macro-Social QnXR
Each case represents country, & aggregate
statistics are compared
Denmark
Ireland
Italy
Portugal
UK
Ideal family size
1979
1989 Religiosity ‘81
2.31
2.13
2.06
3.62
2.79
3.42
2.11
2.20
2.90
2.29
2.23
2.66
2.29
2.14
2.33
(eg from Coleman 1996:39)6
b) Micro-social QnXR
Cases (eg people) are grouped by country
Case id Country Indv. vars
1
1
17 1.73 A
2
3
4
5
6
.
1
1
2
2
3
.
18
17
18
18
19
.
1.85
1.60
1.69
1.65
1.84
.
B
C
A
C
B
.
Natl. var
56.2
56.2
56.2
50.8
50.8
260.3
.
7
Data Analytical techniques
Same core data analysis techniques as for
other social science applications, eg :
Critical issue is ‘level of measurement’
Univariate, bivariate, multivariate
Description v’s inference
Survey methodology issues
A few advanced extensions, eg ‘mixed
models’ to cater for hierarchical effects.
8
Key feature of QnXR:
Country as a categorical factor
Analyse within countries then compare
outcomes (‘case oriented’)
V’s
Analyse data pooled between countries, use
countries / country level factors as explanations
(‘variable oriented’)
9
Country as a categorical factor
Often criticised:
• Appears to be overly simplistic
However
• Same as other QDA factors, eg gender,..
• Critics forget qualified interpretations that good
QDA makes: [these patterns] are associated with
categories, all other things being equal.
• Bad QDA: forget controls for relevant other things
10
Quantitative cross-national social
research
1) Introduction
2) Three traditions in Qn cross-national research
3) Seven themes in Qn cross-national research
Case study 1: Secondary analysis of cross-national
surveys
11
A typology of quantitative crossnational research designs?
• Bryman 2001(p53): 4 types of cross-cultural
research
• Ragin 1987: 2 analytical orientations, one mainly
Qn, the other mainly Ql; proposed resolution with
Qn-style summaries of Ql research
 No typology is perfect – there is much overlap and
ambiguity in methods – but it can be useful to classify
patterns of modern social research…
12
A popular two-stage story:
Early quantitative researchers naively
attempted to measure national differences as
single variables. They badly misclassified or
ignored important national level differences.
Much more thoughtful considerations of
complex national contexts are needed, & often
these are more suited to qualitative research
methods.
Eg: Hantrais and Mangen 96: moves to interpretive methods;
Ragin 87: variable v’s case oriented approaches
13
This inaccurate simplification
implies a false Qn/Ql division:
• Doesn’t reflect variety of current practice
in QnXR (& indeed past practice)
• Doesn’t acknowledge multivariate QnXR
• Doesn’t do justice to many carefully
conducted / reported QnXR projects
• Tends to over-estimate QlXR capactity
14
A picture of QnXR under this
typology:
Crude variable oriented
Case oriented
15
Multitude of contemporary social
research examples don’t fit this
• There are a great many quantitative caseoriented designs
• It is unfair to describe all variable-oriented
designs as inadequate
• ..though to be fair, many variable-oriented
projects are genuinely weak!
16
A fairer typology of QnXR
Crude variable-oriented
Case
oriented
Sophisticated variable
oriented
17
Crude variable oriented
Early or recent, micro- and macro- research making
claims over country level differences, with:
• Insufficient exploration of relevant explanatory
factors
• Limited or poor quality variable
operationalisations & discussions
• Relevant national contexts not appreciated
• False assumptions of good harmonisation
Example: see the illustrated analysis using the ESS
18
Sophisticated variable oriented
Early or recent micro- and macro- research making
claims over country level differences, with:
• Sufficient exploration of relevant explanatory
factors
• Good quality variable operationalisations and
discussions
• Relevant national contexts suitably described
• Accurate assumptions of good harmonisation
Example: more applications than is often realised…
19
Case oriented
Qn analyses within countries, then outcomes
evaluated between countries by authors / readers
• Doesn’t require strong assumptions of data
harmonisation
• Expertise of report writer covers national context
Examples: Edited books; centrally coordinated
projects; end user reviews; …
20
Sophisticated variable oriented
• Attractive method:
– offers parsimony of XN summary
– uses large scale resources
• Methodology for good conduct necessary
– Reliability, validity, implementation, translation
– Sample design
– Reporting strategy and claims
• Boundary to crude research subjective / contested
• Existence often denied by anti-Qn sociologists…
21
Why not be over-cautious?
• Case oriented QnXR seems a safe bet?
Doesn’t make claims not justified
But doesn’t make much impact either
• Remains need for good variable oriented:
Offers a parsimonious summary of
national differences
Govt / media with utilise regardless
22
Quantitative cross-national social
research
1) Introduction
2) Three traditions in Qn cross-national research
3) Seven themes in Qn cross-national research
Case study 1: Secondary analysis of cross-national
surveys
23
3.1) Data availability
•
Massive increases in data resources
accessible to social researchers
–
–
–
•
•
Secondary survey datasets
Official statistics resources
Internet provision / communications
Many data resources under-exploited
Most data originates from survey sources
- but some exceptions
24
3.2) Dataset complexity
•
Secondary surveys tend to feature
–
–
–
–
–
•
Many variables and cases
Complex variable operationalisation choices
Complex structuring (eg multiple hierarchies)
Complex weighting / sampling information
Data analysis & management software needs
Aggregate statistics’ features
–
Difficulty understanding source derivation
25
3.3) Variable operationalisation
•
Single biggest issue in most QnXR conduct
–
–
–
•
Survey design
Dataset analysis
Result reporting
Models of comparability
–
–
–
Exact equivalence of measures
Relativistic equivalence of meanings
Wide literature on ‘reliability’, ‘validity’ of X-N
variable measures and aggregate statistics
26
Variable harmonisation ctd
• Choices over key variables allow use of
previous literatures (eg H-Z & Wolf 2003).
Eg measures of income; occupation; ethnic
group; education; region; crime; health; ..
• Choices over specific analytical variables
require new efforts
Eg, attitude harmonisations of Inglehart.
27
3.4) Survey design
Harkness et al 2003:
Ex post facto harmonisation
(more widespread, eg Eurostat, IPUMS, LIS)
v’s
Coordinated design, sampling, & implementation
(big money projects, eg ESS, ISSP)
Latter as preferable – but whilst many projects
attempt this model, far fewer succeed...
28
3.5) Conduct and logistics
•
•
•
High costs of coordinated surveys
Considerable efforts, and many errors, in ex
post facto harmonisation
Issues of cooperating with colleagues /
diverging academic traditions, eg
–
–
–
different views data access / confidentiality
Technical / software compatibility
different organisations involved in survey production
 QnXR can be very slow process
29
3.6) Temptation
Cross-national datasets nearly always look
simpler than they really are
 dangerous temptation to rush into
uncritical variable analysis
30
3.7) Prejudice
•
Prejudices against quantitative methods
pronounced in European sociology, especially
wrt cross-national comparisons
–
–
•
QnXR evidence often ignored
QnXR researchers portrayed as simplistic
Prejudices favouring quantitative methods
often seen in governmental and media
organisations
–
Mainly: uncritical acceptance of harmonisations
31
Quantitative cross-national social
research
1) Introduction
2) Three traditions in Qn cross-national research
3) Seven themes in Qn cross-national research
Case study 1: Secondary analysis of cross-national
surveys
32
Some leading secondary surveys:
(see handout for internet links)
ESS
ISSP
IPUMS
LIS / LES / LWS
ECHP / CHER / PACO
WVS / EVS
Eurobarometer
Education: PISA / TIMSS
Social Stratification:
CASMIN / CCAP / …
Health /Welfare: eg SVH
33
European Social Survey
• New annual attitudes / values / social
circumstances cross-sections, 2002
• Equivalence of design and survey
implementation between countries
• Extensive methodological resources
• Free access to data
34
Analysis (see SPSS syntax eg)
• Opens harmonised files from 15 countries in 2002
• Select variables measuring attitudes, age, gender
and educational levels
• Generate tables of patterns split by countries
• Use regression models to evaluate contribution of
mulitiple explanatory factors:
– Country specific ‘structural breaks’
– Country effects as variables / interactions
35
Liberal attitudes to homosexuality and
their associations with educational level
(national average and Cramer’s V to educ)
%
Switzerland
Czech Rep
Spain
Finland
UK
Greece
Hungary
Ireland
81
58
70
62
75
51
48
82
CV
10
11
20
14
7
23
5
8
%
Israel
Netherlands
Norway
Poland
Portugal
Sweden
Slovenia
59
88
77
46
71
82
52
CV
20
6
13
16
15
12
18
36
Log-regression prediction of liberalism
to homosexuality for ESS adults
(value & significance of coefficient estimate)
Age-squared
-1.72** Interactions:
Low educ
-0.31** Low educ*NW
0.19**
High educ
0.35** Female*NW
0.35**
Female
0.21** Female*South
-0.15*
North West
1.07** Contrast: medium education
Southern
male from eastern European
0.56** country.
37
This is ‘crude’ variable oriented
• Didn’t try out sufficient relevant explanatory
factors
• Didn’t check variable choices extensively
• Merged variable categories for convenience
• Didn’t use survey weights
• Didn’t contextualise reporting with sufficient
substantive national background and crossexaminations of data sources and measures
38
..but it could have been sophisticated
variable oriented
•
•
•
•
Could have evaluated variable meanings
Could have studied backgrounds
Could have added more explanatory factors
Could have reported more carefully
• .. Research consumption = understanding
how well the results were prepared
39
Summary on Quantitative crossnational research
Quant methods contribute to both
‘variable’ & ‘case’ oriented comparisons
Crude variable oriented widely criticised,
and many bad examples persist
Sophisticated variable oriented research
can be found, and represents most attractive
format of QnXR
40