Transcript EM7_10 - Graduate Institute of International and Development

Epistemology and Methods
Process-Tracing and Content Analysis
May 25 2010
Process-tracing
• Process-Tracing and Historical Explanation (George &
Bennett)
• It’s the Process Stupid (Checkel)
George & Bennett
• What is it about:
• “…to identify the process, one must perform the difficult
cognitive feat of figuring out which aspects of the initial
conditions observed, in conjunction with which simple
principles of the many that may be at work, would have
combined to generate the observed sequence of events”
(Goldstone)
• Applied in many theoretical contexts (rational choice, game
theory, institutionalism, constructivism, …)
George & Bennett
Why process-tracing?
• …may help narrow the list of potential causes
– “it might be difficult to eliminate all potential rival
explanations (…) especially when human agents are
involved (…) for they may be doing their best to conceal
causal processes
• …it forces to take equifinality into account (alternative paths
through which the outcome could have occurred), thus
mapping out various potential causal paths
– with more cases, focus on conditions under which causal
paths occur (typological theory)
George & Bennett
Why process-tracing?
• …that intervening variables, if truly part of a causal process,
should be connected in particular ways is what allows to
reduce the problem of indeterminacy
• …it complements other research methods
• …answers in deviant cases
• …finding omitted variables
• …role in theory development
George & Bennett
Varieties of process-tracing?
• Detailed narrative
– historical chronicles
• Use of hypotheses and generalizations
– parts of the narratives accompanied with explicit causal
hypotheses /attempting to engage in explaining general
patterns
• Analytic explanations
– explicit theoretical forms
• More general explanations
– Moving up the ladder of abstraction
George & Bennett
Forms of causal processes:
• Linear causality
• Complex causality
• Interacting causal variables / interaction effects (typological
theories)
• Path-dependency (locate key decision or branching points)
George & Bennett
To watch out for:
• Assessing alternative hypothesized processes
– Your interest in your hypothesis to be valid creates a strong
confirmation bias, and it can overstate the causal weight…
• Interruption in the causal path
– Which lead to provisional conclusions
• Equifinality
– If you have two alternative explanations / Cumulative
effects? Interaction effects?/ Or different interpretation of
facts
George & Bennett
Guidelines:
• Identifying and addressing factual errors, disagreements, and
misunderstandings
• Identifying all potentially relevant variables and hypotheses
• Comparing various case studies of the same events (using
various theoretical perspectives)
• Identifying additional testable and observable implications of
competing interpretations
• Identifying the scope conditions…
Checkel
• Process tracing: a new buzzword status
• Bringing theory closer to “politics”
• Making scholarship more policy-relevant
•
•
•
•
But watch out for:
Losing sight of the big picture
Be aware of data requirements
Epistemological traps?
Checkel
• Examples:
• in IR: “Democratic Peace” (law-like)
• In EU integration studies: “Bureaucrats Go-native in Brussels”
(claim)
• We know little about the process
• This could help us for instance to rule out certain alternative
explanations (e.g. those send to Brussels are already
committed Europeans), help us understand the intervening
variables (e.g. that lead to democratic peace)
Checkel
• What is a mechanism?
• “Mechanisms connect things. They are “recurrent processes
linking specified initial conditions and a specific outcome”
(Mayntz 2003)
• Link between cause and effect
• Mapping the process carefully
• Between the beginning and the end, the researcher looks for a
series of theoretically predicted intermediate steps
Checkel
• Does it coincide with prior, theoretically derived expectations
about the working of the mechanisms?
• Data is overwhelmingly qualitative (historical memoirs,
interviews, press accounts and documents)
• Strong on questions of how and interactions
• Weaker at establishing structural context (constructivist
concern)
• Challenge: Significant amount of time and data required
Checkel
• Empirical Application: The Socializing Power of European
Institutions
• Persuasion as one mechanism of socialization
• Scope conditions under which it occurred
• “How would I recognize persuasion if I were to walk through
the door?”
• Use of multiple data streams (interviews with Committee
members over five years, confidential meeting summaries,
secondary sources, triangulation)
Checkel
• Process-Tracing: The Good, the Bad and the Ugly
• Lesson 1 (The Good): Helping to Bring Mechanisms Back In
– and the method is here to stay!
• Lesson 2 (More Good): Coming to Grips with First Mover
Advantages
– balance theoretical favorites with alternative explanations
• Lesson 3 (Yet More Good): Promoting Bridge Building
– (e.g. between rational choice and social constructivism)
Checkel
• Lesson 4 (The Bad): Weak Theories
– Not conducive to develop parsimonious or generalizable
theories
– The mechanism-based approach to theory is “intermediate
between laws and descriptions”
– Problematic when it leads to over-determined and, in the
worse case, kitchen sink arguments where everything
matters
– Attention to research design can minimize the problem
Checkel
• Lesson 5 (More Bad): Proxies Are a Pain for Us, too.
– Persuasion can not be measured directly, so interviews
before and after, etc…
• Lesson 6 (Still More Bad): It Takes (lots of ) Time
• Lesson 7 (Yet More Bad): Losing the Big Picture
– Structural context and normative implications
• Lesson 8 (The Ugly): The Dreaded “E” Word
– PT is at odds with the interpretative epistemologies that
under gird may forms of constructivism
Content Analysis
• General comments:
• “the systematic counting, assessing, and interpreting of the
form and substance of communication”
• Precondition: physical record of communication
• Type of communication: books, magazines, newspapers,
documentaries, films, recordings, photographs, meeting
minutes, government documents, diplomatic communiqués,
cartoons, political advertisement, speeches, letters and diaries,
blog,….
Content Analysis
• Steps in using this method:
– Define the population of communications we want to study
(this is determined by the research question!)
– Once population is defined, one possibility is sampling
(still a large n, sufficient level in generalizing from results)
Content Analysis
• Focus on
–
–
–
–
–
Type of communication
Time period of communication
The size of communication (e.g. op-ed articles)
The frequency of communication (e.g. monthly memos)
The distribution of communication (e.g. home delivered
newspapers)
– The location of communication
– The parties to the communication (sender, receiver)
Content Analysis
• Define the unit of analysis
• Word (e.g. peace in speeches of presidents)
• Pitfalls:
• Different meanings of the word (e.g. we seek peace vs. we never allow
peace): counting the words in context
• Item/Theme: the communication itself taken as a whole
• Example: e.g. how many times GB senior diplomats referred to Saddam
Hussein as Hitler-like prior to 1991 Persian Gulf War
• Full-text search (e.g. NYT) for words “George Bush”, “Saddam Hussein”
and “Hitler”
• These words appear within 10 words in either direction
Content Analysis
• Create a “dictionary”: define each and every observation we
might make and allocate a particular coding
• What constitutes a salient reference: e.g. Cuban school books
(American, US, Imperialists, Outlaw Regime in Washington,
etc…)
• Evaluation of salient references (good or bad, pro or anti,
ranking, intensity)
• E.g. ranking statements to measure “intensity” of newspaper
endorsement of candidates
• The role of pre-testing!
Content Analysis
• Special problems:
• Communications are issues for a purpose “All Chinese people
believe that the new agricultural policy is a major step….”
• If we want to assess the impact of communication, we need to
know whom it reaches…
• What is the degree of our own access to communication (free
choice over material we analyze?)
• Is the research sample representative
• Intercoder reliability (human judgments): degree of consensus
among coders
Laver/Garry
• Estimating Policy Positions from Political Texts
• The Manifesto Research Group (MRG) / Comparative
Manifestos Project (CMP): biggest show on the road (started
in the early 1980s)
• Party manifestos are strategic documents written by politically
sophisticated party elite with many different objectives in mind
• Usually issues during elections campaigns, reflect party
positions
• Provide evidence party position changes over time
Laver/Garry
• MRG: measuring relative emphasis of an issue (salience, not
party position!), position # emphasis (attention paid to)
• MRG categories unipolar (e.g. law and order)
• MRG categories bipolar (e.g. social service expansion)
• In order to retrieve information on position, additional coding
of MRG raw data (e.g. Harmel-Janda Party Change Project
(PCP))
• Definition of 19 issues of interests, coding -5 to +5, few
experts engage in coding)
• Caveat: Prior knowledge (subjective placement of parties)…
Laver/Garry
• Estimating policy positions / expert coding:
• Data reduction (coding scheme)
• in MRG sentence count for 54 coding categories, in PCP transforming text
into scores on 19 policy scales
• New approach for systematic coding: an example of hierarchical decisionmaking process (Table 1):
• 1 ECONOMY (Role of state in economy)
– 11 Economy/+State+ (Increase role)
• 111 Economy/+State+/Budget (Budget)
• 1111 Economy/+State+/Budget/Spending (Public Spending)
–
–
–
–
–
11111 Health
11112 Education and Training
11113 Housing
11114 Transport
….
• 1112 Economy/+State+/Budget/Taxes (Increase Taxes)
• 1113 Economy/+State+/Budget/Deficit (Increase Budget Deficit)
Laver/Garry
•
•
•
•
Estimating policy positions / computer coding
Quantitative content analysis
Dictionary – mechanical criteria
But need to focus on less ambiguous words or phrases
(unipolar)
• Example on “taxes”…it’s possible to lower taxes and to raise
taxes, but in manifestos in favor of lowering taxes (question is
how often is one wrong in order to correct?)
Laver/Garry
• The validity costs are offset by gains in reliability (Computer
coding is reliable) and fewer costs
• Computer codes without knowledge of context (no preinformation…), computer could potentially more easily detect
policy changes…
• Laver/Garry on policy positions in UK and Ireland
• Cross-validation of techniques (Computer, revised expert,
MRG, expert surveys), high correlation on economic policy,
less so on social policy