Transcript Tim Leighton





It is impossible to present the philosophy of
scientific realism and its application in social
science in a 30 minute presentation
It is possible to identify crucial problems in using
evidence to inform policy recommendations
And to explain how a realist perspective
stresses the need to understand mechanisms
and contexts in order to explain outcomes.
The presentation will present examples of
research that emphasise the importance of
mechanisms.
EMPIRICISM
REALISM
INTERPRETIVISM
•description and
measurement give us the
‘facts’ about the world.
•objects in the physical,
social & psychological world
exist & have properties
independently of our
theoretical concepts and
discourse about them.
•hermeneutic
•the world exists
independently of us.
•But knowledge is reduced
to the level of events.
•theoretical descriptions
must be defined in terms
of observables
(operationalisation)
•relies mainly on
quantitative methods
•maintains the linguistic
objectivity of scientific
theories.
•social constructionist
•relativist
•everything we know is
socially constructed
(including science)
•layered ontology
•tends reduce existence
to texts, discourses and
forms of consciousness
•Makes use of qualitative &
quantitative methods.
•relies mainly on
qualitative methods.
•accepts that theories should
be subject to empirical
validation.
From: BYGSTAD, B. & MUNKVOLD, B. E. (2011) In Search of Mechanisms. Conducting a Critical
Realist Data Analysis. 32nd International Conference on Information Systems, Shanghai 2011.
Related in various ways in Social Science (e.g.
Giddens’ Structuration theory)
Conflation and Reduction abound! (e.g. ‘Brainism’,
Rational Choice Theory, class reductionism of (some)
Marxism)
Critical Realism attempts to engage with “the
complexity, openness, and ambiguity of social action.”
Are they ‘technologies” like
drug treatments or
surgical interventions?
Are they more akin to social
or educational programmes
which attempt to persuade
people to change their
behaviour?
They consist for the most
part of conversations –
perhaps they are more like
rhetoric (the art of
persuasion).
The question “what works?” should be replaced by “what works for whom in
what context?”
Social programmes (such as programmes to reduce crime, increase health
and well-being, or to promote “recovery”) are complex, often with multiple
outcome goals.
Notoriously, such programmes are evaluated (including with RCTs) as
efficacious in one setting, but disappoint when transferred to another.
“The outcome of a mechanism is contextual, i.e. dependent on other
mechanisms. Thus, a mechanism may produce an outcome in one context,
and another in a different context. This contingent causality (Smith 2010) is
inherent in all open systems.”
(Bygstad, B., & Munkvold, B. E. (2011). In Search of Mechanisms. Conducting a Critical Realist Data Analysis.)
Key Principles (from Nick Tilley’s presentation Conducting Realist Evaluations)
• Programs are theories incarnate
• Evaluations are tests of theories
• Theories need to comprise context-mechanism-outcome
conjectures
• Mechanisms refer to the ways in which effects are brought about
• Contexts refer to the conditions for the operation of mechanisms.
They are seldom closed.
• Outcomes refer to the effects of mechanisms activated in context
• Mechanisms generally (though not always) involve reasoning and
resources
• Programs work differently amongst different subgroups
“Social programmes, as discussed in detail in Chapter 2, offer
resources (material, social, cognitive) to subjects, and whether
they work depends on the reasoning of these individuals.
Subjects may seek out programmes (or not), volunteer for them
(or not), find meaning in them (or not), develop positive feelings
about them (or not), learn lessons from them (or not), apply the
lessons (or not), talk to others about them (or not). It is within this
interpretative process - or mechanism - that the causal powers of
programmes reside. Any method of systematic review that omits
such a vital agent from its core hypotheses automatically sets up
a depleted inquiry.”
Pawson R. (2006) Evidence-Based Policy: A Realist Perspective (Chapter 3)
A realist framework:
Outcome: Transformative change (provisionally defined as durable change in a client’s
frame of reference /emotional commentary, evidenced by their metacognitive accounts of attitude and
relationship changes)
Mechanisms: causal processes generating change - powers emergent
from relations (for example modification of emotional commentary resulting from interactions in a
therapeutic relationship)
Contexts: structural powers bearing on the actualisation of mechanisms
(including countervailing mechanisms, and powers of institutional and other social structures)
From: Pawson R. (1996) Theorizing the Interview. British Journal of Sociology 47/2,
p.300
Koski-Jännes, A. (2005) On the Problems of Randomized Clinical Trials as Means
of Advancing Clinical Practice. Evidence Based Practice: Challenges in
Substance Abuse Treatment. Helsinki: Nordic Council for Alcohol and Drug
Research (NAD).
Morgenstern, J. & Mckay, J. R. (2007) Rethinking the paradigms that inform
behavioral treatment research for substance use disorders. Addiction, 102/9,
1377-1389.
Orford, J. (2008) Asking the right questions in the right way: the need for a shift in
research on psychological treatments for addiction. Addiction, 103/5.
Tucker, J. A. & Roth, D. L. (2006) Extending the evidence hierarchy to enhance
evidence-based practice for substance use disorders. Addiction, 101/7, 918-32.
Tudor Hart, J. (1997) What evidence do we need for evidence based medicine?
Journal of Epidemiology and Community Health (1979-), 51/6, 62.
Weak theories:
If interventions work through specific effects then some should be
superior due to the efficacy of their ingredients.
Theory-based matching effects should appear.
Very little evidence for these.
Markedly inconsistent findings in efficacy, moderator and mediator effects
with all the ‘evidence-supported’ interventions.
Important differences between the outpatient and aftercare arms of
Project MATCH.
(Morgenstern, J., & McKay, J. R. (2007). Rethinking the paradigms that inform behavioral treatment research
for substance use disorders. Addiction, 102(9), 1377-1389.)
Many trials
Various meta-analytic reviews
Some recent investigations into mechanisms
Apodaca, T. R., & Longabaugh, R. (2009). Mechanisms of change in motivational interviewing: a
review and preliminary evaluation of the evidence. Addiction, 104(5), 705-715
Morgenstern, J., Kuerbis, A., Amrhein, P., Hail, L., Lynch, K., & McKay, J. R. (2012). Motivational
interviewing: A pilot test of active ingredients and mechanisms of change. Psychology of Addictive
Behaviors. 26(4), 859-869.
Results show significant main effects for student residence, with more
drinking and more problems reported by students living in fraternity
and sorority houses. For students who live in the Greek system, it is
possible that a house-based intervention (rather than the individual
interviews conducted in this study) would be an effective prevention
strategy. Current research conducted by our group is investigating the
effectiveness of such a group-based intervention for these students.
What is the mechanism here? And what context facilitated its activation?
“This is a field where there is no lack of
randomised controlled trials. Perhaps it is time to
move from only studying whether MI works to
also studying how it works, that is to study the
mechanisms behind MI.”
(Smedslund et al 2011 p.28, emphasis in original).
Morgenstern, J., Labouvie, E., McCrady, B. S., Kahler, C. W., & Frey, R. M. (1997). Affiliation with Alcoholics
Anonymous after treatment: A study of its therapeutic effects and mechanisms of action. Journal of consulting and
clinical psychology, 65(5), 768.
Morgenstern, J., Bux, D., Labouvie, E., Blanchard, K. A., & Morgan, T. J. (2002). Examining mechanisms of action in
12-step treatment: The role of 12-step cognitions. Journal of Studies on Alcohol and Drugs, 63(6), 665.
Morgenstern, J., & Longabaugh, R. (2002). Cognitive–behavioral treatment for alcohol dependence: A review of
evidence for its hypothesized mechanisms of action. Addiction, 95(10), 1475-1490.
Morgenstern, J., Bux, D. A., Labouvie, E., Morgan, T., Blanchard, K. A., & Muench, F. (2003). Examining mechanisms
of action in 12-Step community outpatient treatment. Drug and alcohol dependence, 72(3), 237-247.
Longabaugh, R., Donovan, D. M., Karno, M. P., McCrady, B. S., Morgenstern, J., & Tonigan, J. S. (2005). Active
Ingredients: How and Why Evidence‐Based Alcohol Behavioral Treatment Interventions Work. Alcoholism: Clinical and
Experimental Research, 29(2), 235-247.
Longabaugh, R. (2007). The search for mechanisms of change in behavioral treatments for alcohol use disorders: a
commentary. Alcoholism: Clinical and Experimental Research, 31(s3), 21s-32s.
Huebner, R. B., & Tonigan, J. S. (2007). The Search for Mechanisms of Behavior Change in Evidence‐Based
Behavioral Treatments for Alcohol Use Disorders: Overview. Alcoholism: Clinical and Experimental Research, 31, 1s3s.
Apodaca, T. R., & Longabaugh, R. (2009). Mechanisms of change in motivational interviewing: A review and preliminary
evaluation of the evidence. Addiction, 104(5), 705-715.
Longabaugh, R., & Magill, M. (2011). Recent advances in behavioral addiction treatments: focusing on mechanisms of
change. Current psychiatry reports,13(5), 382-389.
Longabaugh, R., & Magill, M. (2012). Commentary on Michie et al.(2012): The lid is off the black box. Addiction, 107(8),
1441-1442.
Morgenstern, J., Kuerbis, A., Amrhein, P., Hail, L., Lynch, K., & McKay, J. R. (2012). Motivational Interviewing: A Pilot
Test of Active Ingredients and Mechanisms of Change. Psychology of Addictive Behaviors, 26(4), 859-869.
In the last of these, the authors distinguish between “active ingredients”, which are things the programme or
therapist does, and “mechanisms of change” which are processes that go on with the client, e.g. changes in
reasoning or take-up of resources.
The Palo Alto group (Moos, Finney etc) were historically interested in contexts and have attempted more
recently to characterise treatment programmes according to theory-based identification of ‘active ingredients’:
Bromet, E., & Moos, R. H. (1977). Environmental resources and the post-treatment functioning of alcoholic
patients. Journal of Health and Social Behavior, 326-338.
Cronkite, R. C., & Moos, R. H. (1980). Determinants of the post-treatment functioning of alcoholic patients: A
conceptual framework. Journal of Consulting and Clinical Psychology, 48(3), 305.
Holahan, C. J., & Moos, R. H. (1987). Personal and contextual determinants of coping strategies. Journal of personality
and social psychology, 52(5), 946.
Moos, R. H. (2007). Theory-based active ingredients of effective treatments for substance use disorders. Drug and
alcohol dependence, 88(2), 109-121.
Moos, R. H. (2007). Theory-based processes that promote the remission of substance use disorders. Clinical
Psychology Review, 27(5), 537-551.
Problems in relying solely or mainly on an experimental research design to provide
evidence about how alcohol or drug dependence treatment works (let alone recoverypromoting systems of care) have been recognised for 35 years.
The EBM movement encouraged an almost exclusive emphasis on RCTs as the gold
standard to try to identify efficacious treatments along the drug testing model.
Longitudinal outcome studies have been sometimes grudgingly included but they
have problems of their own.
In the view of many (e.g Morgenstern & McKay 2007, Orford 2008,) the results have
been disappointing or have reached a dead end.
Morgenstern, J., & McKay, J. R. (2007). Rethinking the paradigms that inform behavioral treatment research for
substance use disorders. Addiction, 102(9), 1377-1389.
Orford, J. (2008). Asking the right questions in the right way: the need for a shift in research on psychological
treatments for addiction. Addiction, 103(6), 875-885.
*according to Prof. Nick Tilley, realist evaluation “refuses to provide (meaningless) simple (minded) answers that
are sometimes wanted” but this of course does not mean that critical realist researchers are against evidencebased practice or reject RCTs as valueless.
Timko C., Moos R.H., Finney J.W., Lesar M.D. (2000) Long-term outcomes of alcohol use disorders:
Comparing untreated individuals with those in Alcoholics Anonymous and formal treatment. Journal of Studies
on Alcohol. 6(4):529–540.
Kaskutas L. A., Bond J., Humphreys K. (2002) Social networks as mediators of the effect of Alcoholics
Anonymous. Addiction 97: 891–900.
Timko, C., Billow, R., & DeBenedetti, A. (2006). Determinants of 12-step group affiliation and moderators of the
affiliation–abstinence relationship. Drug and alcohol dependence, 83(2), 111-121.
Kelly, J. F., Magill, M., & Stout, R. L. (2009). How do people recover from alcohol dependence? A systematic
review of the research on mechanisms of behavior change in Alcoholics Anonymous. Addiction Research &
Theory,17(3), 236-259.
Kelly, J. F., Stout, R. L., Magill, M., Tonigan, J. S., & Pagano, M. E. (2011). Spirituality in recovery: a lagged
mediational analysis of Alcoholics Anonymous’ principal theoretical mechanism of behavior change. Alcoholism:
Clinical and Experimental Research, 35(3), 454-463.
Kelly, J. F., Stout, R. L., Magill, M., & Tonigan, J. S. (2011). The role of Alcoholics Anonymous in mobilizing
adaptive social network changes: A prospective lagged mediational analysis. Drug and alcohol
dependence, 114(2), 119-126.
Kelly, J. F., Hoeppner, B., Stout, R. L., & Pagano, M. (2011). Determining the relative importance of the
mechanisms of behavior change within Alcoholics Anonymous: a multiple mediator analysis. Addiction, 107(2),
289-299.
Kelly, J. F., & Hoeppner, B. B. (2012). Does Alcoholics Anonymous work differently for men and women? A
moderated multiple-mediation analysis in a large clinical sample. Drug and Alcohol Dependence.
Mediators and moderators do not quite get us to mechanisms but they are a good
start. (this is beginning to be explicitly recognised in the discussion sections of
some of the papers cited)
Emerging clues:
•Changes in social network
•Self-efficacy (social for men, negative affect for women)
•Spiritual practices, reduction in negative affect for the most impaired (apparently
less important for the less impaired)
The “mechanisms” in realist terms are the way in which an individual uses the
above to change their reasoning (including “emotional reasons”) and make use of
resources.
The “contexts” are the contingencies which inhibit or facilitate the activation of
mechanisms (e.g. stigma > reduced occupational opportunities > reduced
confidence and sense of purpose)
“Our results suggest that the process of recruiting young drug users into a study
and focusing their attention on substances before a brief intervention may be, in
itself, sufficient to induce positive reactive effects on behaviour.” (p.1024)
Cognitive/experiential
1. Consciousness raising: Gaining information that increases awareness about the current behavior
pattern or the potential new behavior
2. Emotional arousal: Experiencing emotional reactions about the status quo and/or the new behavior
3. Self-reevaluation: Seeing and evaluating how the status quo or the new behavior fits in with or
conflicts with personal values
4. Environmental reevaluation: Recognizing the positive and negative effects the status quo or new
behavior have upon others and the environment
5. Social liberation: Noticing and increasing social alternatives and norms that help support the status
quo and/or change and initiation of the new behavior
Behavioral
1. Self-liberation: Making choices, taking responsibility for, and making commitments to engaging in a
new behavior or a behavior change
2. Stimulus generalization or control: Creating, altering, or avoiding cues/stimuli that trigger or
encourage a particular behavior
3. Conditioning or counterconditioning: Making new connections between cues and a behavior or
substituting new, competing behaviors and activities in response to cues for the "old" behaviors
4. Rcinforcement management: Identi fying and manipulating the positive and negative reinforcers for
current or new behaviors. Creating rewards for new behaviors while extinguishi ng (eliminating
reinforcements) for current behavior.
5. Helping relationships: Seeking and receiving support from others (family, friends, peers) for current
or new behaviors