Transcript EVALUATION OF COMMUNICATION PROGRAMS

Measuring the Impact of
Entertainment-Education
Programs
D. Lawrence Kincaid, Ph.D.
Center for Communication Programs
Bloomberg School of Public Health
Johns Hopkins University
4th International Entertainment-Education Conference
September 26-30, 2004 at the Lord Charles Hotel
in Somerset West, South Africa.
Overview
• Between a “rock and a hard
place” . . . without a paddle
• Association or cause and effect?
• From experimental control by
means of research designs
• To 8 conditions for causal
attribution
The biggest limitation to the
evaluation of communication
• Members of the population who watch the
program are always different from those
who do not watch. (Self-selection bias)
• These differences are usually related to
the expected outcomes. (behavior, etc.)
• So, a simple comparison of outcomes by
exposure is biased by the differences
between those exposed and not exposed.
The Counter-Factual Dilemma
• How much does a communication program
change the behavior of audience members who
are exposed to the program compared to what
they would have experienced if they had not
been exposed?
• Problem: A perfect solution would require a
parallel universe. Used only as an ideal.
• Randomized experimental design cannot be
used for full-coverage programs, so the problem
has to be solved with measurement and theory.
Solutions for Full-Coverage
Programs
1. Statistical Controls: Use multiple regression
to control for all confounding variables that
might affect exposure and the outcome.
2. Simple Matching: Limited to 1-3 variables.
3. Propensity Score Matching: Unlimited
number of variables that might affect
exposure and the expected outcomes.
4. Theory-based evaluation
A Practical Alternative
• The goal is to estimate how much change
can be attributed to communication.
• Establish the criteria for measuring
how much impact can be causally
attributed to your program.
• Obtain empirical evidence to support
each criterion.
• Draw an appropriate conclusion.
First, how do you know who was
really exposed? (implementation)
The South African TV Drama, Tsha Tsha
1. No. of characters audience can
correctly name from photographs (k=4)
2. No. of correct items of knowledge about
the drama (k=8)
3. No. of specific episodes they watched
that they can identify (k=26)
What is the name of the character
shown in each photo below?
Knowledge of the Drama Independent of
the Expected Outcomes (unaided)
1.
2.
3.
4.
5.
6.
7.
8.
Who is studying for a UNISA degree?
Who steals money from Viwe’s father?
Who is Joy’s father?
What gift does Viwe give to Andile?
Who gets a sexually transmitted infection?
What item does Unathi sell to get money for
food?
Who gives Mimi R1,000 for her business?
Who does Cedric send to Johannesburg to
deliver dagga?
Number of the 26 episodes watched
Mark which episodes you have seen:
1. Prince opens a hair salon and interviews girls in
the town
2. Mimi and Andile have sex, and she is scared he
will wake her grandmother
.
.
13. Funeral of Andile’s mother
.
.
26. A party is organized for Andile to dance in Joburg
A Continuous Measure of Recall
of Tsha Tsha
Distribution of the Index of Recall
0
0
.1
.02
Density
Density
.2
.04
.3
.06
Distribution of the Index of Recall with Non-viewers
40
50
60
Recall of Tsha Tsha Round 3 (episode3 + knowtsh3 std)
70
40
50
60
70
Recall of Tsha Tsha Round 3 (episode3 + knowtsh3 std)
Breaking recall into levels
for the analysis of impact
80
Percent
68
60
40
80
Percent
60
32
40
20
20
0
0
No
Yes
Watched the Drama
32
34
34
None
Low
High
Level of Recall
1. Did the outcome change over time?
Percent
40
30
20
10
8
10
Wave 1 2003
Wave 3 2004
0
Abstinence for a Year or More
N = 754; not statistically significant
2% of 10,000,000 = 200,000 youth
5. Was the observed change
large and abrupt?
Wave 1
Wave 3
Percent
40
33
28
30
17
20
10
8
10
9
0
Abstinence for a
Year or more
Faithful to My
Partner *
Abstinence for a
Month or more *
* Statistically significant: p<0.01
Note: A one-year time interval
2a. Was the change associated with
exposure to communication?
Percent
50
40
40
30
20
17
10
0
Not Exposed
Exposed
Faithfulness to My Partner
* Statistically significant: p<0.01
2b. Was the change associated
with exposure to communication?
Mean Percent
Percent Agreement
1
.5
39
50
Recall of Tsha Tsha, Round 3
Regression of AIDS attitude on level of recall (r=.32)
72
3a. Did exposure occur before the
observed change?
Wave 3
No Yes
Wave 1
Total
No
Yes
405
103
140
105
545
208
Total
508
245
753
Decided to Remain Faithful to My Partner
Net Increase = 37
Chi-square = 5.63; p<0.05
3b. Did exposure occur before the
observed change?
Unexposed
Exposed
Percent
80
60
70
46
40
20
13 14
18
5
22
12
0
Stayed No Yes to No
Stayed
Yes
New Yes
Decided to Remain Faithful to My Partner
N=753; Chi-square = 46.8, p<.001
4a. Multiple Regression to Estimate the Independent Effect
of Exposure after Controlling for Other Influences *
South Africa, 2004
Learned about
AIDS on TV
.13
.11
.26
Female
Lagged AIDS
attitude (wave 1)
Education
Frequency of TV
viewing
Kwazulu province
residence
AIDS
Attitude
.13
.09
.33
.12
.09
.31
Recall of the
Drama
-.01
* Including lagged attitude means
that the impact of other variables
is on change in attitude.
4b. Propensity Score Matching to Estimate Effects
of Exposure of a Drama on AIDS Attitudes
Wave 3 South Africa, 2004
Positive AIDS Attitude
Variables Used to Construct the
Propensity Score for Matching:
Percent
100
82
1. Age
73
80
2. Education
3. Gender
60
4. Income Level
40
6. Lagged Aids Attitude
7. Learned about AIDS on TV
20
9
0
Treatment
Group
Matched
Control
Group
Net
Difference
8. Province (KwaZulu Natal)
6. Is there evidence of a dose
response?
AIDS Attitude by Level of Recall
100
90
Mean percent
agreement
78
80
70
73
68
60
None
Low
Level of Recall
High
7. Is the causal inference
justified theoretically?
What is a theory?
• How communication works.
• A tool for thinking and action.
Theory-Based Evaluation
• Identify causal pathways
(mediating variables for effect)
• Demonstrate:
• Impact of communication on
mediating variables
• Impact of mediating variables on
behavioral outcome
IDEATION MODEL OF COMMUNICATION AND BEHAVIOR
C
O
M
M
U
N
I
C
A
T
I
O
N
INSTRUCTION
SKILLS & KNOWLEDGE
IDEATION
DIRECTIVE
Dissemination
Promotion
Prescription
COGNITIVE
Beliefs
Attitudes
Values
Perceived Risk
Subjective Norms
Self-Image
INTENTION
EMOTIONAL
NONDIRECTIVE
Dialogue
Counseling
Entertainment
Social Networks
PUBLIC
Advocacy
Regulation
Emotional Response
Empathy
Self-Efficacy
confirmation
BEHAVIOR
SOCIAL
Support & Influence
Personal Advocacy
ENVIRONMENTAL
SUPPORTS & CONSTRAINTS
Source:
Adapted from
Kincaid (2000)
A predictive model of communication & change
Knowledge
Personal
Advocacy
Social
Support &
Influence
Attitudes
Implies
simultaneous
effect of all
influences.
SelfImage
BEHAVIOR
Perceived
Risk
Emotions
SelfEfficacy
Norms
Implies
communication
can effect all
influences.
Direct and Indirect Impact of
Communication on Behavior
Mediating
Effects
Behavioral
Outcome
Communication
Program
A Path Model of the Effects of Recall of
the Drama on Identification with Boniswa
and AIDS Attitudes
Identification with
a character in the
drama
indirect
effects
Attitude
towards
HIV/AIDS
direct
effects
Level of
exposure to the
drama
direct
effects
7. Path Model of the Effects of Recall of the Drama on
Identification with Boniswa and AIDS Attitudes at Wave 3
South Africa, 2004
Watches “Days of
Our Lives”
.06
Abstained for a
month or more
.09
.34
Learned about
AIDS on TV
Lagged AIDS
attitude
Frequency of TV
viewing
Kwazulu province
.13
.13
.11
.26
Female
Education
Identification
with Boniswa
.12
AIDS
Attitude
.13
.33
.12
.09
.31
-.01
.09
Recall of the
Drama
7. Is the causal inference
justified theoretically?
YES
8. Are the results consistent with
previous program research?
YES
Eight Criteria for Causal Attribution
1. Change over time in the expected
outcome is observed.
2. An association between that change and
program exposure is observed.
(correlation)
3. Exposure occurs before the observed
change is measured. (time-order)
Eight Criteria for Causal Attribution
4. No evidence of confounding variables that
may have accounted for the change.
5. The observed change is abrupt and large.
(immediacy and magnitude)
6. The impact increases in proportion to level
or duration of exposure. (Dose response)
Eight Criteria for Causal Attribution
7. A causal connection is justified.
(causal pathways and theoretical coherence)
8. Consistency with previous program
research. (replication with variation)
Unfinished Work
We can still not adequately measure many of
the important effects of EE, especially drama.
1. Expanding the perspective and
ways that people make decisions
2. Emotional Involvement (caring, fear)
3. Measuring effects of narrative
Measuring the effects of a story
The Eagle and the Cocks
• Two cocks in the farm yard fought to decide who
should be master.
• The loser withdrew to a dark corner, while the victor
flew to the roof top and crowed lustily.
• An eagle spied him from high up in the sky,
swooped down, and carried him off.
• The other cock came out and ruled without a rival.
PRIDE COMES BEFORE THE FALL.
Aesop’s Fable (620 B.C.)
FIN
Thank you!
Why Programs Fail or Appear to Fail
• Inappropriate theory
•
but inadequate
application of the theory to the program
•
Appropriate theory, adequately designed,
Appropriate theory,
but poorly implemented
•
Appropriate theory, adequately designed,
well implemented,
•
but ineffective research
Appropriate theory, design, implementation,
but inadequate
measurement of the degree of impact
and research design,
Sources of Program Success
• Appropriate theory
• Correct application of the theory
to the design of the program
• Successful implementation
• Adequate evaluation research
• Adequate measurement of impact
Implications
• Five sources are interrelated.
• If not effective, it’s difficult to know why.
• Program may be effective, but the
research cannot show it.
• You can only improve if you know what
happened and why.
• There are realistic limits to what we can
learn from program evaluation.