Transcript London Medical Sociology Group 2011

Can Qualitative
Research Have Impact?
Is It Even EvidenceBased?
Martyn Hammersley
The Open University
Seminar, London Medical Sociology Group,
Kings College, London, October 2011
The Role of Sociology
Sociologists have long been concerned with the
social value of their work, though they have
taken different views about their role.
We can distinguish the following broad options, the
sociologist as:
a)Technician;
b)Professional;
c)Critic;
d)Engaged activist.
Impact and Evidence-based
Policymaking and Practice
• Recent pressure for social science to maximise its
impact: institutionalised in ESRC requirements, and
in government demands that universities contribute
directly to the national and local economies.
• The notion of impact employed here is close to that
championed by the evidence-based policymaking
and practice movement.
• From this point of view, the primary task of research
is to demonstrate ‘what works’, in other words to
evaluate the effectiveness of policies and practices,
or to generate new, more effective ones.
The Concept of ‘Impact’
• One problem with this notion of impact, from
the point of view of qualitative researchers, is
that it privileges quantitative work designed to
control variables and measure ‘effect size’.
• In its classical form, the gold standard for
research serving evidence-based medicine,
and evidence-based practice more generally,
was the randomised controlled trial.
• Such research was seen as demonstrating
‘what works’, what does not, and what causes
unacceptable side effects.
Liberalisation
In many quarters, over time, there has been
a liberalisation of this classical model, as
regards:
1.The relationship between research and
practice;
2.The sorts of research that can inform
policymaking and practice.
However, this shift has been untheorised
and amounts to little more than a
defensive strategy
What Does ‘Impact’ Mean?
We need to think carefully about the term
‘impact’.
It is a metaphor whose prototype is: one
object hitting another and causing it to
change position or trajectory; whether a
mere ‘nudge’, to use a currently influential
term, or a more dramatic transformation.
The Problems With Impact
There are problems with this metaphor:
a)The influence of research findings on
policymakers and practitioners is not
direct, it is mediated by many factors;
b)It is always an interpretative not a
mechanical process;
c)We should not assume that high impact is
always good. To do so is to adopt a naïve
Enlightenment perspective.
Engineering Model
The notion of impact corresponds with what has been
called the ‘engineering model’ of the relationship
between research and policy/practice (Janowitz,
1972; Bulmer 1982; Finch, 1986; Weiss 1977;
Hammersley 2002). This assumes that:
1. Social science can make the same dramatic
contribution to practice as natural science.
2. It can do this through producing theoretical or
instrumental knowledge that is then 'applied to' or
'translated into’ policies or practices; and/or
through social science methods being applied
within policymaking and practice, rendering these
more rational and effective.
The Enlightenment Model
The usual contrast to the engineering model is the
enlightenment model. There are different versions:
1. What we might call strong enlightenment sees
social science as providing a comprehensive
theoretical perspective that facilitates the
transformation of policymaking and practice;
1. Weak enlightenment assumes that research shapes
policymaking and practice in a range of indirect and
piecemeal ways, percolating through social
networks: it may throw doubt on current
assumptions, suggest alternative perspectives,
point to unknown consequences, etc.
Problems With ‘Strong
Enlightenment’
It implies that somebody is in the dark, and that only
researchers have a light. But:
• A comprehensive theoretical perspective is of
questionable practical value.
• There are other valuable sources of knowledge
besides research.
• There are serious doubts about the capacity of
social science to produce all of the kinds of
knowledge required to make good policy and
practice.
Is Qualitative Research Evidencebased?
There are two ways in which qualitative research is
faulted as regards its capacity to serve
evidence-based policymaking and practice:
1.It frequently does not address the question of
‘what works?’
2.It is alleged that it cannot provide the quality or
reliability of knowledge required.
The Evidence-based Model
of Research
What is involved here is a crude positivist conception
of research in which findings are derived by means
of determinate inference (logical or calculative)
from observational givens. In short, research is to
be carried out on the basis of explicit procedures.
However:
• Neither medical research nor natural science
approximates closely to this ideal in practice.
• The philosophical rationale for this ideal has been
comprehensively demolished over the past 50
years within the philosophy of science.
Problems With the Procedural
Model
• There are no observational givens, in the sense
required: all observation is assumption-laden and
disposition-relative.
• Relying upon procedural rules may correct some
errors but can also generate new error as well.
• There are no logical or calculative means by
which we can infer conclusions from data. The
latter always have to be interpreted, and
competing interpretations evaluated through
informed judgment.
A Wider Concept of Impact
There has been an interesting ambivalence about the
demand for impact among qualitative researchers.
Indeed, many wish their research to have impact, in the
sense of challenging and changing policy/practice.
As I noted at the beginning, they may even see
themselves as social critics or activists; for instance,
recent discussion of ‘public sociology’ (Burawoy
2005; Holmwood 2007; Brewer 2011)
Of course, a positivist conception of research practice
is rejected here; and perhaps also the notion of
strong enlightenment, in favour some model of the
‘specific’ intellectual.
Problematic Assumptions About
Social Science
What is the relationship between research and
policymaking/practice assumed here, if it is not either
that of the engineering or strong enlightenment
model?
Is there not retained the same excessive estimate of
the capabilities of social research that is
characteristic of those two models? There are two
aspects to this. It seems to be assumed that:
1.Social science can answer evaluative or value
questions.
2.It can provide reliable factual evidence about all the
questions that are relevant to key policy or practice
issues.
Problematic Assumptions
about the Wider Society 1
These arguments about how sociology can
have impact seem to assume that a
deliberative democracy is in operation. But:
a)This is not the case (see Weiss 1983;
Hammersley 2006);
b)There are major tensions between experts
providing knowledge and many conceptions
of deliberative democracy.
Problematic Assumptions
about the Wider Society 2
Where impact is conceptualised as being upon
the individual consumer or client, two
problematic assumptions are involved:
a)That this consumer is sufficiently informed
and rational (in the appropriate sense) to
make good use of any knowledge provided.
b)That sociological knowledge can be
preserved in the relatively brief and simple
messages that are required if impact of this
kind is to take place (Hammersley 2006).
Conclusion
The notion of impact involves misconceptions
about the nature of social research, and what
it can offer policymaking/practice, and about
the wider society.
This is true not just in relation to notions of
research for evidence-based practice, but
also as regards the idea of a public sociology.
The weak enlightenment model forces us to
adopt a more modest role, and to try to
defend this in the face of excessive external
and internal demands for ‘impact’.
References
Burawoy, M. (2005) ‘For public sociology’, American Sociological Review, 70, pp4-28.
Brewer, J. ‘The new public university, 19th January 2011, ‘A new kind of public university and a
new kind of social science’, 11th February 2011’, and ‘The New Public Social Sciences 2’ 25
March 2011. Available at (accessed 6.10.11):
http://sociologyandthecuts.wordpress.com/category/john-brewer/
Bulmer, M. (1982) The Uses of Social Research, London, Allen and Unwin.
Finch, J. (1986) Research and Policy: the uses of qualitative methods in social and educational
research, Lewes, Falmer.
Hammersley, M. (2002) Educational Research, Policymaking and Practice, Paul Chapman/Sage.
Hammersley, M. (2006) Media Bias in Reporting Social Research? The case of reviewing ethnic
inequalities in education, London, Routledge, 2006.
Hammersley, M. (2011) Methodology, who needs it?, London, Sage.
Holmwood, J. (2007) ‘Sociology as Public Discourse and Professional Practice: A Critique of
Michael Burawoy’, Sociological Theory, 25, 1, pp46-66
Janowitz, M. (1972) Sociological Models and Social Policy, Morristown NJ, General Learning
Systems.
Weiss, C. (1977) ‘Research for policy’s sake: the enlightenment function of social science
research’, Policy Analysis, 3.
Weiss, C. (1983) ‘Ideology, interests, and information’, in D. Callahan and B. Jennings (eds.)
Ethics, The Social Sciences, and Policy Analysis, New York, Plenum Press.