Transcript Quantitative versus Qualitative Is a Distraction

Quantitative versus Qualitative Is
a Distraction: Variations on a
Theme by Brewer & Hunter (2006)
Methods @ Plymouth, 2007
W. Paul Vogt
1
The Theme
• Focusing on the distinction between
quantitative and qualitative, and on
methods of mixing them, “ignores a wider
range of methodological problems and
opportunities to solve them.”
2
The Variations (Overture)
•
•
•
•
•
•
•
•
•
•
To think in terms of quant and qual designs is a category mistake.
The q-q distinction diverts attention from other kinds of multimethod research.
It leads to stereotyping and tribalism among researchers.
It encourages us to accept our weaknesses.
It ignores the relation between indicators and concepts.
It is based on confusion about the nature of thinking.
It diverts attention from the nature of the phenomena being
studied.
It underemphasizes the extent to which researchers routinely
alternate between categorical and continuous variables.
It distracts us from other ways of thinking and handling data,
particularly graphic ones.
Mixed methods, by treating the q-q distinction as though it were
the most important one, may have the paradoxical effect of
reinforcing categories better abandoned or deemphasized.
3
1. Thinking in terms of quant and qual
designs is a category mistake.
All designs can accommodate quant
and/or qual data. This is an empirical
claim. It is easy to find examples of
quant or qual data collected using each
of the main design types: document
analysis, secondary analysis, naturalistic
observation, surveys, interviews,
experiments, participant observation.
4
2. The quant-qual distinction deflects
attention from other multi-method work.
• Document analysis and interviews (both
often verbal)
• Field observations coded numerically and
laboratory experiments coded numerically
• Surveys and focus groups
5
3. It leads to stereotyping and
tribalism among researchers.
• Correlations and tendencies are confused with logical entailment.
– Tashakkori & Teddlie (1998) can illustrate.
• Researchers using quantitative “methods” make time-free and
context-free generalizations. The number of exceptions is huge,
e.g., sociologists who use quant data to study social context, or
social historians who use quant data to show change over time.
• Researchers using qualitative “methods” do not believe it is possible
to discuss causes. Again, the number of counter examples is huge.
Causal analysis that does not rely on quant data is venerably old.
– Gibbons’ Decline and Fall
– Max Weber’s Protestant Ethic
– Durkheim’s Division of Labor
6
4. It encourages us to accept our
weaknesses; or provides an excuse for
not correcting them
• Social psychologists have shown how
easy it is for biased side-taking to emerge
in social situations.
• Ignorance of, say, grounded theory or of
regression analysis can be a mark of
cultural status among researchers.
7
5. The important relation of indicators to
concepts is not addressed
• Is the indicator a symptom? a cause?
a component? a predictor?
– Law and morality a la Durkheim
– Democracy =? fair elections, separation of
religion and state, free speech
– Principal components analysis and factor
analysis (e.g., health and intelligence)
8
6. It leads to confusion about
the nature of thought.
• Numbers are words.
• One of the most basic of language distinctions, in all languages, is
the distinction between singular and plural, one and many, which is
fundamentally a quantitative distinction.
• Even the most verbal-intensive research contains relations of
quantity—such as more or less, mild or strong, and so on.
• Even the most number-intensive research contains qualitative
distinctions: cause, influence, predict, present/absent, and so on.
• Rank order relationships are the somewhat unrecognized meeting
ground of qualitative and quantitative data.
• Researchers using quantitative data often split continua into
categories, such as high, medium, or low on the self-efficacy scale.
9
7. It diverts attention from the nature of
the phenomena being studied.
• Height of Mendel’s peas (cat or cont?).
– Could evolution could accommodate
continuous variation or did it require
discontinuous variation?
– The battle mattered because it concerned the
nature of the phenomena; it had to be
resolved on substantive grounds. The issue
was the nature of reality, not how to code it.
10
8. Researchers often alternate between
categorical and continuous variables
• Falklands War and First Gulf War and suicide rates in
the U.K. and U.S.
• recessions and the ability to afford university tuition in
the U.S.
• Quantifying subjective judgments: Pain
• Gleason scoring of tumors. Here we have the subjective
judgments of physicians about objects.
– interrater reliability—inter-subjective.
– Quantified subjective judgments are used to make a categorical
decision about treatment.
– To talk about whether this kind of analysis and decision making
is quant or qual is irrelevant. It must inevitably be both.
11
9. It distracts from other ways of
thinking about data, particularly graphic.
• Venn diagrams are essential for thinking
about overlapping categories.
• Change over time in quantitative data is
often hard to describe without line graphs.
• SEM without graphic means of depicting
the measurement and causal models.
• Causal models are almost inevitably
graphic—not quant, not qual, but graph.
12
10. Mixed methods may not help
• Mixed methods may have the
paradoxical effect of reinforcing the
categories they were meant to bridge—
categories that in many contexts are
better abandoned.
13
Conclusions
• The characteristics of one’s evidence and
how one codes it will constrain one’s
analysis strategies—at some point. This
point should come relatively later, not
earlier. It should be one of the later, not
one of the earlier, branches in the decision
tree.
• What should come earlier?
14
Prior Questions 1
• How can I best gather the kind of evidence
I need to answer my research question?
(Design)
• Who or what do I sample or select for
study using that design? (Sampling)
• What are the ethical implications of those
design and sampling choices? (Ethics)
• Only then: Should I code my data using
words, numbers, pictures, or all three?
15
Prior Questions 2
Questions about research questions that have
priority over the quant-qual distinction include:
• Does the problem involve making causal inferences?
• Is it necessary to generalize from the cases studied
(sample) to a broader group (population)?
• Does the problem include the study change over time?
• Is it necessary to interact with subjects/participants?
• Must one find one’s own data sources and/or generate
one’s own data?
• Is it necessary to use more than one design?
16