Transcript The Door Opens - The Practical Radical

Matching Evaluation
to the Nature &
Conceptualization of
the Problem
Simple, Complicated and Complex Problem
Framings and the Implications for Evaluation
Brenda Zimmerman, Schulich School of Business
York University, Toronto, Canada
Presentation to the Joint Canadian Evaluation
Society/American Evaluation Association Conference
Toronto Oct 29, 2005
Social Transformation – our definition

An alteration of what is established by the
introduction of new elements or forms
(including new ideas, practices, or resource
flows). In particular the alteration of social
relationships to allow for an improvement in
or transformation of intransigent and broadly
based social problems
Our approach to social innovation


Connecting understanding of large scale
patterns or dynamics to action at the level of
individuals or groups
Concerned with how to understand these
interactions in such a way that we could help
those trying to make a difference
The Decision to Use a Complexity
Lens




Complexity theory leads us to look at the
relationship between the micro and the macro
It suggests how individuals act effectively in
arenas which they cannot control (complex is
more than complicated)
Complexity theory points to what is
happening “in the between”; in the
relationships between things.
Implications for evaluation??
Simple
Complicated Complex
Following a Recipe
The recipe is essential
Recipes are tested to
assure replicability of
later efforts
No particular
expertise; knowing how
to cook increases
success
Recipe notes the
quantity and nature of
“parts” needed
Recipes produce
standard products
Certainty of same
results every time
A Rocket to the Moon
Child
Raising a
Simple
Following a Recipe
The recipe is essential
Complicated Complex
A Rocket to the Moon
Child are critical
• Formulae
and necessary
•
Sending one rocket
increases assurance
that next will be ok
•
High level of expertise
in many specialized
fields + coordination
•
Recipes produce
standard products
Separate into parts and
then coordinate
•
Certainty of same
results every time
Rockets similar in
critical ways
•
High degree of
certainty of outcome
Recipes are tested to
assure replicability of
later efforts
No particular
expertise; knowing how
to cook increases
success
Raising a
Simple
Following a Recipe
The recipe is essential
Recipes are tested to
assure replicability of
later efforts
No particular
expertise; knowing how
to cook increases
success
Recipes produce
standard products
Certainty of same
results every time
Complicated Complex
A Rocket to the Moon
• Formulae are
critical and
necessary
Raising a Child
•
Formulae have only a
limited application
•
Raising one child
gives no assurance of
success with the next
• Sending one rocket
increases
assurance that next
will be ok
• Expertise can help
but is not sufficient;
relationships are
key
• High level of
expertise in many • Can’t separate parts
from the whole
specialized fields +
coordination
• Every child is unique
• Rockets similar in
critical ways
• High degree of
•
Uncertainty of
outcome remains
Simple
Following a Recipe
Complicated Complex
A Rocket to the Moon
Raising a Child
The recipe is essential
•
Recipes are tested to
assure replicability of
later efforts
Formulae are critical •
and necessary
Formulae have only a
limited application
•
Sending one rocket
increases assurance
that next will be ok
Raising one child
gives no assurance of
success with the next
No particular
expertise; knowing how
to cook increases
success
Recipe notes the
quantity and nature of
“parts” needed
Recipes produce
standard products
Certainty of same
results every time
•
High level of expertise
in many specialized
fields + coordination •
Expertise can help
but is not sufficient;
relationships
are key
•
Separate into parts
and then coordinate
•
Rockets similar in
critical ways
•
High degree of
certainty of outcome •
•
•
Can’t separate parts
from the whole
Every child is unique
Uncertainty of
outcome remains
• Sources for charts of next three slides
Begun, Zimmerman and Dooley, “Health Care
Organizations as Complex Adaptive Systems”,
in S. M. Mick and M. Wyttenbach (eds.), 2003
Advances in Health Care Organization Theory
San Francisco: Jossey-Bass, pp 253-288
Also can be found at www.change-ability.ca
Complexity Science
Established Science --- “complicated”
Holism
Reductionism
Indeterminism
Determinism
Relationships among entities
Discrete entities
Nonlinear relationships
– critical mass thresholds
Linear relationships
– marginal increases
Quantum physics
– influence through iterative nonlinear
feedback
– expect novel and probabilistic world
Newtonian physics
– influence as direct result of force from
one object to another
– expect predictable world
Understanding; sensitivity analysis
Prediction
Focus on variation
Focus on averages
Local control
Global control
Behavior emerges from bottom up
Behavior specified from top down
Metaphor of morphogenesis
Metaphor of assembly
Adapted from Dent 1999
Established Perspectives*
Complexity Science Perspective
VIEW OF THE
FUTURE
Relatively knowable
Relatively unknowable
RELEVANCE OF
HISTORY
None (transaction cost) to
high (institutional). When
high, history is deterministic.
High, but history may or may not be
deterministic.
DOMAIN OF
STUDY
Reified organization in the
environment
Relationships among individuals,
subsystems, systems
VIEW OF THE
ENVIRONMENT
Outside the organization;
evolves separately
Part of the domain of study; coevolves
with the organization
LEVELS OF
ANALYSIS
Single to few, relatively
independent
Multitude of nested levels
STRATEGY
Relatively designed
Relatively emergent
STRUCTURE
Equilibrium; relatively
centralized
Non-equilibrium; relatively decentralized
KEY INFO. FOR
THE
ORGANIZATION
External environmental
intelligence
Functioning of relationships
TEMPORAL
FRAMING
SPATIAL FRAMING
CONSTRUCT
FRAMING
Complexity Science
Perspective
Implications for Research
VIEW OF THE FUTURE
Relatively unknowable
Patterns may repeat, but without
predictive power. Anticipate
surprise. Study emergence.
RELEVANCE OF
HISTORY
High, but history may or may
not be deterministic.
Requisite to study history (vs. crosssectional only); conduct longitudinal
analysis
DOMAIN OF STUDY
Relationships among ind.,
subsystems, systems
Study patterns of interaction among
agents.
VIEW OF THE
ENVIRONMENT
Part of the domain of study;
coevolves with the org.
Study co-evolution of organization
and environment
LEVELS OF ANALYSIS
Multitude of nested levels
View issue from multiple, nested
levels of systems
STRATEGY
Relatively emergent
Study changes in strategy and
conditions that facilitate change
STRUCTURE
Non-equilibrium; relatively
decentralized
Assess flexibility of structures;
simple rules; min specs
PURPOSE OF
ORGANIZATIONAL
RELATIONSHIPS
Learning; co-creation of
meaning; functioning of
relationships
Assess degrees of co-participation,
learning, sharing; study quality of
relationships
TEMPORAL FRAMING
SPATIAL FRAMING
CONSTRUCT FRAMING
The Brazil Case Study





annual per capita income is less than
$5000
In the 1980s, Brazil’s AIDS problem was
worse than South Africa’s
By 2000, South Africa’s HIV infection
rate was 25% whereas Brazil’s was 0.6%
In 1992, the World Bank predicted that
Brazil would have 1.2 million AIDS cases
by 2000
…but the actual count was 0.5 million.
Brazil Analysis is from Begun, Zimmerman and Dooley, 2002
The Brazil Complex Problem


How do you respond to an AIDS epidemic in a
developing country?
No money for expensive drugs


A very iffy health care system


problems of treatment
High levels of illiteracy


problems of affordability of drugs
problems of compliance even if there were drugs
High levels of poverty and hunger

problems of nutritional needs for drugs
Making the Brazil Problem Complicated




“What will drug costs be for our infected
population?”
“What resources are needed to manage drug
therapies for illiterate patients?”
“What resources are needed to assure
compliance with drug associated nutrition in
this population?”
“What are the resources needed for an
effective prevention program?”
World Bank Responds to AIDS as
Complicated





Meaningful solutions require sophisticated,
integrated national health care systems
We cannot provide treatment to all when the drug
costs are so high
We cannot afford resources to manage treatment
compliance
With our limited resources, we should focus more
on prevention than treatment
It will therefore take a long time for the problem
to work itself through
Brazil Implicitly Recognized the Complexity

Began to address the AIDS issue head-on in 1994

Were unwilling to accept the answers of the World
Bank

Hence – had to change the questions…
 Changing the questions changes the focus,
changes what is “analyzed”, changes what is
seen as possible/impossible
The Brazil Questions Assume Complexity
World Bank Questions

“What will drug costs
be for our infected
population?”
Brazil Questions

so that we can provide
treatment to all who need
it?”


“What resources are
needed to manage
drug therapies for
illiterate patients?”
“How can we reduce costs
“What methods of
communication will work
to convey the drug therapy
routine to a patient – even
a homeless, illiterate
patient?”
The Brazil Questions
World Bank Questions


“What resources are
needed to assure
compliance with drug
associated nutrition in this
population?”
“What are the resources
needed for an effective
prevention program?”
Brazil Questions

“If food is an issue, how
can we ensure greater
compliance with the routine
by linking up with charities
that can provide food at the
right times of day?”

“How can we achieve our
prevention goals while
treating all of those
currently infected?”
The Brazil Conclusions
World Bank Conclusions





Meaningful solutions require
sophisticated, integrated
national health care systems
We cannot provide treatment
to all when the drug costs are
so high
We cannot afford resources
to manage treatment
compliance
With limited resources, focus
more on prevention than
treatment
It will therefore take a long
time for the problem to work
itself through
Brazil Conclusions

Find ways to use the
resources we have to
respond to the problem

Provide drugs to all by
finding ways to reduce drug
costs

Use our informal system to
train people to care for
themselves

Prevention will be part of the
treatment

Seek short and long term
results
The Brazil Responses

A stable container: Brazil built on existing
infrastructure (natural network existed)
 A somewhat shaky health system of
hospitals and clinics
 Added to by 600 NGOs, churches (hubs)

Free drugs to all AIDS patients: faced
down drug companies in pursuit of national
interest
The Brazil Responses - cont’d

Patients managed their own drugs
 Illiterate people were taught by local
“trustworthy” folks (hubs were sought)

Free treatment spread prevention ideas
 No labeling of those affected
 Prevention information readily available
Brazil saw the value in the
relationships as the key to change
The Brazil AIDS story - optimism

COURAGE– to challenge
WTO, USA, large
pharmaceuticals, World Bank


COMPLEXITY – lived with
the complex nature of the
society, and the AIDS
problem
RELATIONSHIPS – used the
power of existing relationships to
learn and enhance their
connectedness
Matching Evaluation
to Cycles of
Intervention &
Program Development
Ecocycle & Panarchy as a
Framework
The ecocycle as a metaphor for
successful social innovation


Based on the work of ecologists studying
natural systems (C.S. Hollings in particular)
Suggests four distinct stages in a “life cycle”,
linked to two key dynamics: connectedness
or sameness and stored capital or potential
MATURITY
GROWTH
BIRTH
Released
POTENTIAL Stored
The birth, growth, destruction and renewal of a forest
1
Exploitation
Variety
CONNECTEDNESS
Sameness
exploitation
4/8/2016
The Psychosocial place called
“exploitation”

Triggers: Choice! Groups of people have converged on some promising
products or processes and are devoting time and energy to these. Decision to
invest in one or more and build capacity

People: the implementers come into their own. Invest in people who love to take
ideas and develop them into reality, the team-builders and the engineers come
into their own.

Experience: Excitement, flow, high energy, doors opening, long hours, time
flying by….often correlated with peak experiences. A time of intense activity,
flow and productivity with a steep organizational “learning curve”

Evaluation: “Formative” – look for a continuous feedback systems as the startup phase moves towards greater goal clarity and efficiency in delivery

Management: Supporting capacity building. Careful attention to the need for
training, hiring and developing systems to support bringing a program “to
market” and moving it up the scale. Evaluation geared to providing feedback for
increasing effectiveness and efficiency.

Traps: Failure to build sufficient capacity or sufficient “lock-in” to achieve
economies of scale and delivery efficiency
Released
POTENTIAL Stored
The birth, growth, destruction and renewal of a forest
1
Exploitation
Variety
CONNECTEDNESS
Sameness
conservation
4/8/2016
The Psycho-social place called
“conservation”

Triggers: Reaching a level of efficiency and learning where
improvements become incremental. The “mature” phase, productivity
high and measurable, peak “earnings” and efficiency.

People: Invest in those excellent at management, system design excel,
others may be getting restless, or even a little bored.

Experience: satisfaction at success, the pride in the well oiled machine,
anxiety about mushrooming demand, loss of momentum

Evaluation: Summative evaluation – success based on goals should
be measurable by quantitative or qualitative means.

Management: awareness of vulnerability, standing still (again),
openess to creative destruction. Encourage “letting go”

Traps: rigidity, loss of peripheral vision, over-commitment. Fear of
change.
Released
POTENTIAL Stored
The birth, growth, destruction and renewal of a forest
Conservation
1
Exploitation
Variety
Creative
Destruction
CONNECTEDNESS
Sameness
Creative destruction






Not devastation of “soil”
Crisis…. from root word “to sift”
Creation requires destruction
To honor creation… open to destruction
Externally imposed
Internally driven
Creative
destruction
4/8/2016
Creative
destruction
4/8/2016
The psychosocial space called
“creative destruction”

Triggers: most clearly signaled by the discontinuation, destruction or
collapse of a given program or process and the simultaneous release of
resources and energies for other things.

People: those who thrive on crisis, on new beginnings are happy here –
others may be depressed or in mourning

Experience: of confusion, identity crisis,, change in relationships,
anxiety, elation.

Management: providing reassurance, “standing still”, ‘active” reflection,
contemplation, listening.

Evaluation: organizations should be “harvesting”: identifying lessons
learned and thinking about implications for the future.

Traps: Fear and guilt can act to isolate – communication is needed.

Outputs: Focus on new ideas, investment of capital in exploration.
Renewal
Also known as
Exploration or
Reorganization
Renewal/exploration
4/8/2016
The psychosocial space called
“exploration”

Triggers: The need for innovation. Development of multiple “random walks”,
experiments, initiatives which lead to little in the way of measurable outcomes
for considerable time

People: people who learn by doing are happy here; invest in individuals who are
entrepreneurs/ innovators with a track record.

Experience: Reflection moving to experimentation, lots of false starts and
sometimes frustration and mounting anxiety about inputs/output ratios

Management: Generate resources and connections and exchange of
information, encourage experimentations and learning, have faith in the selforganizing properties of information and groups. Introduce new ingredients as
required. Exert increasing pressure for output.

Evaluation: geared to capturing learning, building transparent accounts of
innovation.

Traps: “spinning wheels”, the incapacity to select a course and commit
resources to it to the required degree. Ungovernable competition.
4
2
Renewal
Conservation
3
1
Exploitation
Creative
Destruction
Little
CAPITAL STORED
Much
The birth, growth, destruction and renewal of a forest
Weak
CONNECTEDNESS
Strong
Released
POTENTIAL Stored
The birth, growth, destruction and renewal of a forest
Renewal
Conservation
1
Exploitation
Variety
Creative
Destruction
CONNECTEDNESS
Sameness
Stored
RIGIDITY
TRAP
POVERTY
TRAP
Released
Variety
Sameness
The psycho-social space of
poverty traps

Organization, processes, initiatives: a sense
of spinning wheels, initiatives and interactions
are infrequent or are not generative

Experience: “chronic disaster”, increasing
hopelessness or helplessness, as sense of
the poverty of the very ground of being
Psycho-social experience of
rigidity traps

Organizations, systems initiatives: rulebound, unresponsive, resistant to change, too
may resources tied up in non-productive
systems.

Experience: a kind of rigor mortis, nothing
new seems to happen, creativity
discouraged, emphasis on evaluation and
performance
Stored
Released
Variety
Sameness
Do we evaluate Resiliency?

Connectedness is an indicator of how much
similarity vs. variety characterizes the
behavior/composition of the system

Potential (or capital) is an indicator of the degree
to which resources (time, energy, money, skill,
knowledge) are committed to existing projects as
opposed to available to attach to new projects

Resilience is a measure of the ability of the
system to avoid “traps”, i.e. getting stuck in any
phase
Developmental
Evaluation
Formative Evaluation
Summative
Evaluation