Transcript Training on Setting-up Awareness Campaigns

Training on Setting-up Awareness
Campaigns
Prof. dr. Tom Brijs
Transportation Research Institute
Hasselt University (BE)
Disclaimer: Significant parts of the presentation were taken from the European Manual for Designing, Implementing,
and Evaluating Road Safety Communication Campaigns (FP6 project CAST)
Outline
1. Intervention mapping as a stepwise approach
for creating awareness campaigns
2. Practical implementation of the stepwise
approach
Part I:
Intervention mapping as a stepwise
approach for creating awareness
campaigns
Introduction
• Campaigns are all about changing behavior. So a
crucial point is to understand the determinants
of behavior
– Predisposing factors
– Reinforcing factors
– Enabling factors
Predisposing factors:

“learning-related factors within the
persons of the target population that
facilitate or hinder motivation for
change”
e.g., awareness of the problem, beliefs about the
problem, attitudes toward the problem, perceptions
about wearing helmets

Reinforcing factors:

“factors that encourage or discourage
continuation of the behaviour”

e.g., social support, peer pressure, incentives,
punishment (fines, demerit points)
Enabling factors:

“factors that can help or hinder the
desired behaviours as well as
environmental changes”

e.g.
 Skills: how to correctly wear the helmet
 Resources: money to buy a helmet
 Barriers: hot weather
Theory of planned behaviour
Explaining Human Behaviour
Skills
Barriers
Campaigns
Changing needs understanding
but
Understanding is not sufficient
for changing
Changing behaviour is a science
It involves applying theories, learning from
earlier empirical studies done on the topic or
related topic, and collecting data from your
target population and the facilitators of the
intervention. Foremost, changing behaviour
needs a systematic and planned approach.
Changing behaviour is a science
Evidence-based public health programmes
are based on empirical data and theory.
Theory and Practice
Applying theory and research
Knowledge of Theory
especially knowledge of the process and conditions
Example: Fear Appeals
Theory and Research
High Personal Relevance
Difficult Behavior Change
Fear
+
= Defensive reaction
Theory and Research
High Personal Relevance
Difficult Behavior Change
Skills
+
= Constructive reaction
Part II:
Practical implementation of the
stepwise approach
Intervention Mapping
Step 1: Needs Assessment
Step 2: Defining Campaign Objectives
Step 3: Selecting Methods and Strategies
Step 4: Designing the Campaign
Step 5: Planning for Campaign Implementation
Step 6: Planning for Campaign Evaluation
Intervention Mapping
Step 1: Needs Assessment
Step 2: Defining Campaign Objectives
Step 3: Selecting Methods and Strategies
Step 4: Designing the Campaign
Step 5: Planning for Campaign Implementation
Step 6: Planning for Campaign Evaluation
Step 1: Needs Assessment
 Task 1: Establish a planning group that includes
potential program participants and plan the needs
assessment
 Task 2: Conduct the needs assessment to analyze
traffic and mobility problems and behavioral and
environmental causes
 Task 3: Define the target population
 Task 4: Link the needs assessment to evaluation
planning by establishing desired program outcomes
Step 1: Needs Assessment
Task 1: Putting together a work group
Representatives of the resource group – the program
developers and possibly funders
Representatives of the potential program adopters and
implementers
Representatives of the priority population – ultimate
end users and managers or leaders from community
organizations
Step 1: Needs Assessment
Representatives of the resource group
• Public authorities
– Ministries
– National Road Safety Council
– World Bank, ADB, …
•
•
•
•
•
Road authorities
Ngo’s: HIB, CRY, GRSP, …
Insurance companies
Vehicle and safety equipment manufacturers
Charities
Step 1: Needs Assessment
Representatives of the potential program
adopters and implementers
• Enforcement authorities
– Cambodian National Traffic Police
– Local police
• Schools
• Driver education training centers
• Private organizatons, e.g.
–
–
–
–
–
Petrol stations
Automobile associations, car manufacturers
Festival and event organizers
Restaurant owners
Supermarkets, …
• Volunteer organizations
• Hospitals, doctors
• Drink producers
Step 1: Needs Assessment
Representatives of the priority population
• Action groups, e.g.
– Motorcycle action groups
– Road victims association
– Local road safety ambassadors
• Private organizations
–
–
–
–
Sport clubs
Youth organizations
Taxi drivers
…
• A selection of individuals from the target audience
Step 1: Needs Assessment
• What we know from the international literature
– A strong coalition increases the chances of success
of the campaign: public-private partnership
involving state organizations (from top level political
commitment to local commitment from police
departments, local road safety ambassadors), legal
departments, insurance companies, …
– Early involvement of the different stakeholders is
needed
– Strong police leadership needed to ensure that
strong enforcement is accepted by police officers
Step 1: Needs Assessment
Task 2: Conduct a needs assessment
1. What is the problem? Why? Is it a serious
problem? What people are involved?
2. What behaviors and environmental
conditions are involved?
3. Brainstorm key determinants of behaviors
and environmental conditions
4. Build a theory- and evidence-based
explanation
Step 1: Needs Assessment
 Where do we get the necessary information?
- crash statistics (RTAVIS)
- hospital data
- field observations
- field surveys
- past campaigns
- international literature
- interviews with experts
- focus group discussions
Step 1: Needs Assessment
•
Selection of a theoretical model underlying the
target behavior in the campaign
–
–
–
–
Theory of planned behavior
Health-belief model
Protection motivation theory
Deterrence theory
Step 1: Needs Assessment
Theory of Planned Behaviour
Behavioural
beliefs
Attitude
Normative
beliefs
Subjective
norm
Control
beliefs
Perceived
behavioural
control
skills
Behavioural
intentions
Behaviour
barriers
Step 1: Needs Assessment
• Examples
– subjective norm, perceived behavioral control (drink
driving campaign)
http://www.youtube.com/watch?v=qsY_Co-p8Bw
– Subjective norm, perceived behavioral control (drink
driving)
http://www.youtube.com/watch?v=23xr3JF-tz0
Step 1: Needs Assessment
Protection Motivation Theory
Intrinsic benefits
Extrinsic benefits
+
Severity
Vulnerability
=
Threat appraisal
Protection
motivation
Self-efficacy
+
Response cost
=
Coping appraisal
Step 1: Needs Assessment
• Examples
– Severity, vulnerability, response cost (helmet wearing
campaign Vietnam)
http://www.youtube.com/watch?v=3X46UoXpAME
– Self-efficacy (child restraint)
http://www.youtube.com/watch?v=j4-zRcm38oM
Step 1: Needs Assessment
Health Belief Model
Perceived
Severity
Cues to action
Perceived
Threat
Perceived
Susceptibility
Perceived
Benefits
Perceived
Barriers
Likelihood of
Action
Cost-benefit
Analysis
Step 1: Needs Assessment
• Examples
– Benefits (seatbelt wearing campaign)
http://www.youtube.com/watch?v=23xr3JF-tz0
– Action cues (campaign on fatigue)
http://www.youtube.com/watch?v=LR4NK_xz7y8
– Action cues (call a “responsible young driver”)
http://www.youtube.com/watch?v=Vq93NCBp9Ik
Step 1: Needs Assessment
Deterrence Theory
– People tend to change their behavior more easily when
campaigns are associated with stronger enforcement.
– Communication about enforcement must focus on:
• Immediacy: you will be fined immediately
• Severity: you will pay a high fine
• Likelihood: you have a high chance of being caught
– Example drink driving
http://www.youtube.com/watch?v=iwKCjIaT_Xs
Step 1: Needs Assessment
Task 3: Describing the individuals who are the
potential program recipients (target audience)
 The recipients of campaign benefits may not
necessarily be the population at-risk
– E.g., car drivers and pedestrians (pedestrian accidents)
– E.g. parents and children (child protection)
 There are often multiple groups targeted by an
intervention, some of whom are populations at risk and
others, those who influence the environment (e.g.
police, school teachers)
 Program participants could be environmental agents,
such as the road safety managers, police, …
Step 1: Needs Assessment
• Need for segmentation = separating the
audience in distinct, relative homogeneous
subgroups, called segments.
• Why?
– The idea is that since a segment shares key
characteristics, individuals within a segment will react
similarly to the same stimuli (e.g. campaign)
– Carefully selecting the target group increases
significantly the effect of a campaign
Step 1: Needs Assessment
• Criteria for target segment selection
– Severity: how much is the target group at risk, what
is the potential for improvement?
– Accessibility: how can they be identified and
reached?
– Responsiveness: are they willing to listen and are
they responsive to our message?
– Segment size: is the segment large enough to be
cost-effective?
Step 1: Needs Assessment
• Segmentation variables
– Demographic: age, gender, education, occupation,
family size, social class, …
– Geographic: rural versus urban, province, …
– Psychographic: personality, beliefs, values, interests,
lifestyle
– Behavioral: user rate (high offenders versus low
offenders)
– State of change
Step 1: Needs Assessment
•
Segmentation variables
–
State of change:
•
•
•
•
•
•
pre-contemplation: individual has no intention to change (believers
/ non-believers)
contemplation: individual starts to become aware of the problem.
Highlight discrepancy between individual needs and general public
needs (traffic safety)
preparation: individual has begun preparations to change. This can
be supported by facilitating change.
action: change has occurred but risk is still high for fallback. The
first actions must be rewarding in order to prevent fallback. Peer
pressure, subjective norms play a very important role here. Giving
gifts, demerit point systems etc may help.
maintenance: new behavior has started to become a habit
termination: new behavior is established and return to old behavior
is not likely
Step 1: Needs Assessment
Task 4: Objectives from the needs assessment
Type of objective
Safety outcomes
Definition
What will change in terms of the
safety problem? By how much?
Among whom? By when?
Safety-related behavior
outcomes
What safety-related behavior will
change? (e.g. helmet wearing)
Safety-related
environmental
outcomes
What environmental conditions will
change? (e.g. helmet lockers)
Step 1: Needs Assessment
Task 4: Objectives from the needs assessment
• What will be the target variable of our campaign?
–
–
–
–
Knowledge about risks associated with not wearing the helmet
Attitudes with respect to helmet wearing
Behavioral intentions with respect to helmet wearing
Changed behavior (helmet wearing)
• Observed
• Self-reported
-> the closer to behavior, usually the harder it is to prove the
effectiveness of the campaign)
Intervention Mapping
Step 1: Needs Assessment
Step 2: Defining Campaign Objectives
Step 3: Selecting Methods and Strategies
Step 4: Designing the Campaign
Step 5: Planning for Campaign Implementation
Step 6: Planning for Campaign Evaluation
Step 2: Campaign Objectives
• At this stage, the general aim of the campaign is translated into
specific objectives
• What are the expected effects of the campaign in terms of the
target variable for the specific target group?
– E.g. double the helmet use during peak hours for young people between
16-25 years old
• Other possibilities
– The number or severity of accidents involving non-helmet use
– The number of offences
• Alongside the primary objectives, there can also be secondary
objectives
– E.g. to influence the attitudes towards helmet use
– E.g. to influence perceive behavioral control (PBC) by reminding drivers
that they are in charge of their own decision to wear the helmet
Intervention Mapping
Step 1: Needs Assessment
Step 2: Defining Campaign Objectives
Step 3: Selecting Methods and Strategies
Step 4: Designing the Campaign
Step 5: Planning for Campaign Implementation
Step 6: Planning for Campaign Evaluation
Step 3: Methods and
Strategies
Theoretical methods
– general technique for influencing change
– E.g. use of testimonials
Practical strategies
– Specific technique for the application of the method
– E.g. accident victim goes to schools to talk about
his/her experiences and makes young people aware
of the dangers
Step 3: Methods and
Strategies
• Methods to influence knowledge
–
–
–
–
Discussion (e.g. in class rooms, on TV, radio, …)
Images
Active learning
…
• Methods to increase awareness/risk perception
– Information about personal risk
– Persuasive communication (e.g. role model stories)
• E.g. 6-12 years old: http://www.vtm.be/zeppezikki/#movie07
– Fear arousal
– …
Step 3: Methods and
Strategies
• Methods to increase self-efficacy
– Skill training (e.g. short courses)
– Guided practice with feedback (e.g. additional road education
courses)
– Reinforcement
(http://www.youtube.com/watch?gl=NL&hl=nl&v=CBZSF
5Q4B5c)
• Methods to change habits
– Presenting substitute behaviors
– Presenting cues (I buckle up before I start the engine)
Step 3: Methods and
Strategies
• To change attitudes
– Self-reevaluation (discussing about one’s own behavior)
– Increasing direct experience (e.g. testimonials)
– Anticipated regret (e.g. imagine you are disabled for the rest
of your life, how would you feel about that?)
• To change social influence
– Building skills for resistance to social pressure
(http://www.youtube.com/watch?gl=NL&hl=nl&v=NTHN
8JguKAU)
– Showing positive visible expectations
Intervention Mapping
Step 1: Needs Assessment
Step 2: Defining Campaign Objectives
Step 3: Selecting Methods and Strategies
Step 4: Designing the Campaign
Step 5: Planning for Campaign Implementation
Step 6: Planning for Campaign Evaluation
Step 4: Designing the
campaign
• Develop the campaign strategy
– Task 1: Define the strategy
– Task 2: Develop the message
– Task 3: Choose the media and media plan
Step 4: Designing the
campaign
Task 1: Define the strategy
•
•
Isolated campaign: a single campaign
Combined campaign: accompanied by increased
enforcement, changed legislation and/or educational
activities
Integrated campaign:
•
–
–
Horizontal integration: across sectors (involvement of
different stakeholders)
Vertical integration: across levels (from National to
municipal)
E.g. THINK! (UK) & STEP (USA)
–
•
Multi-year campaign: several waves of the same
campaign carried out over a longer period of time
Step 4: Designing the
campaign
Task 1: Define the strategy
• Mass-media communication (non-personal)
– TV, radio, newspapers, …
• Selective communication (reach target groups
simultaneously)
– Cinema, direct mail, flyers, brochures
• Interpersonal communication (two or more people
communicate directly)
– Focus groups
– Telephone
– Face-to-face
Step 4: Designing the
campaign
• What we know from international research
– Combined and integrated campaigns are more effective than
isolated campaigns
– Increased enforcement in combination with enforcement
awareness campaigns are very successful
• 2 weeks of advertizing about increased enforcement: 1 week before
and 1 week during the increased enforcement period
– Campaigns should be accompanied by measures to assist the
change process
– The use of campaign waves is very successful: 2-3 times per
year
– Feedback of the results at the end of the campaign towards
the general public increases the acceptability of campaigns
– Inform road users about intermediate results: information
panels indicating the number of offences or the number of
correct users
Step 4: Designing the
campaign
Task 2: Develop the message
• General principles about the campaign message
–
–
–
–
Clear
Consistent
Unambiguous
Well understood by the target audience
• Content of the message versus style of the
message
• You might want to involve an advertising agency
to propose different communication concepts
Step 4: Designing the
campaign
• The message content: what will be said
– Structure of the message
• One sided message: only arguments in favour of the
theme of the campaign are presented
– E.g. wear a helmet because it reduces your probability of serious
injuries or death in an accident
• Two-sided message: arguments in favour and against the
theme are presented, then arguments to counter the
opposing view are included
– E.g. wearing a helmet reduces your probability of serious injuries.
We know that it is bad for your looks, but do you really think that
it is worth driving without a helmet knowing that after having an
accident your good looks will be gone forever.
Step 4: Designing the
campaign
• When to adopt one-sided messages
– When the target audience is sympathetic to the message
– When the aim is an immediate/short term opinion change
• When to adopt two-sided messages
– When the target audience is unsympathetic to or initially
disagrees with the campaign message. I.e. when the negative
aspects of the wanted behavior are well known among the
target audience
– When it is likely that the target audience will be exposed to
contradictory messages
Step 4: Designing the
campaign
• The message style: how it will be said
– Emotional and non-factual versus cognitive and rational
approach
– Emotional messages can be either positive (love, hapiness,
pleasure) or negative (fear, anger, sadness)
– One must be cautious using fear appeals. Two types of
reactions can take place:
• Cognitive reaction: the respondent wants to reduce the danger by
adapting his/her behavior ( = control the danger)
• Emotional reaction: rejection of the message, i.e. activation of
various mechanisms to protect the person from the threat (deny,
defensive avoidance or reactance) (= control the fear)
Step 4: Designing the
campaign
• Examples
– Informational approach
• Drink driving (testimonial, social cost)
http://www.youtube.com/watch?v=9kplg8O6GPQ
• Helmet wearing (celebrity endorsement)
http://www.youtube.com/watch?v=BmZBDBjCh2o
• Speed
http://www.transport.qld.gov.au/resources/file/eb6d2f00d77fddb/st
opping_distances_tvc.mpg
– Emotional approach
• Drink driving (social cost)
http://www.youtube.com/watch?v=JdCTxTM_GtU
Step 4: Designing the
campaign
• Examples
– Fear appeal approach
Step 4: Designing the
campaign
•
Message framing
–
Problem avoidance: the message is directed at
avoiding getting into trouble by adopting the
correct behavior
E.g. (seatbelts)
http://www.youtube.com/watch?v=Xx6v9CNcQ04
–
Promotion oriented: the message is directed at
showing the benefits of adopting the correct
behavior
Step 4: Designing the
campaign
• Message execution: who’s saying it (crucial for
credibility of the message)
– Organizations instead of an individual
– The use of testimonials
– Celebrities
Step 4: Designing the
campaign
• Campaign identifiers
– Mascots
– Branding
– Logo
– Celebrities
-> they increase acceptance and/or credibility of the
message
-> they are useful as a cornerstone of a complete
campaign package
Step 4: Designing the
campaign
• What we know from international research
– Persuasive and emotional messages work better than rational
and informative (educational) messages
– Message must be relevant to the target group in order to
increase systematic processing by the target audience. E.g.
playing on family values, family e
– Systematic processing increases by exposing the audience to
the message several times
– Fear appeals can work in specific circumstances and when
they are not too aggressive. Otherwise, a defensive reaction
will take place. Therefore, it is very important to provide
efficient mechanisms to avoid the threat and that the target
audience finds itself capable of carrying out the
recommended actions.
Intervention Mapping
Step 1: Needs Assessment
Step 2: Defining Campaign Objectives
Step 3: Selecting Methods and Strategies
Step 4: Designing the Campaign
Step 5: Planning for Campaign Implementation
Step 6: Planning for Campaign Evaluation
Step 5: Planning for campaign
implementation
• Choice of media
• Media planning
• Pre-testing (if possible)
Step 5: Planning for campaign
implementation
• Choice of media and mediators
– Depends on campaign budget
– Depends on the target audience’s media habits
• Time most likely to be exposed to the message (e.g. young
males watch football matches)
• Time most likely to adopt the problem behavior (e.g. in
bars ~ drink driving)
– Depends on geographic coverage of the campaign
– Depends on advantages and disadvantages of
different media types
Step 5: Planning for campaign
implementation
• Strengths/weaknesses different media types
– TV
• S: large reach, high frequency, accessible to everyone
• W: high production cost, short lived messages (seconds)
– Radio
• S: large reach, also local reach, dynamic, low production cost
• W: low attention, short lived messages, not suitable for complicated
messages
– Cinema
• S: selective, high attention, allows for more complexity
• W: high production cost
Step 5: Planning for campaign
implementation
• Strengths/weaknesses different media types
– Newspapers
• S: large reach, geographical selectivity, high credibility, high
information capacity, short lead time
• W: short lifespan, poor demographic selectivity, poor reproduction
quality, high noise ration
– Magazines
• S: excellent for segmented audience, relatively long lifespan, good
reproduction quality
• W: high insertion cost, low frequency
– Flyers, leaflets, brochures
• S: high selectiveness, allows for complex messages, low cost
• W: low attention
Step 5: Planning for campaign
implementation
• Strengths/weaknesses different media types
– Direct mailings
• S: selective communication, for small target groups, high information
capacity
• W: ‘junk mail’, need for address lists
– Billboards, small-size posters/banners
• S: high exposure, on-the-spot presence, geographically selective, long
life span
• W: low attention, low information capacity
– Face-to-face contacts
• S: effective, interaction with target group
• W: low exposure, high cost per contact
Step 5: Planning for campaign
implementation
• Media planning
= Scheduling the distribution of campaign materials
(spots, billboards, website, …) and supportive
activities (e.g. education, enforcement, …)
– Duration and frequency
• single-phase versus multi-phase
– Time of campaign implementation
• Seasonal issues (e.g. fatigue in summer period)
• Event related (e.g. drink driving around Khmer New Year)
Step 5: Planning for campaign
implementation
• If possible: Pre-testing !!
–
–
–
–
Is the message appropriate for the target audience?
Is the message understood as intended?
Is the message clearly stated?
Is the message perceived as useful to the target
group?
– Is the message well recalled or remembered?
– Is the message provoking unexpected feelings or
reactions in the target group?
Step 5: Planning for campaign
implementation
• How to pre-test a campaign?
–
–
–
–
Interviews
Questionnaires
Focus groups
Thought listing task: record immediate verbalised
responses
Intervention Mapping
Step 1: Needs Assessment
Step 2: Defining Campaign Objectives
Step 3: Selecting Methods and Strategies
Step 4: Designing the Campaign
Step 5: Planning for Campaign Implementation
Step 6: Planning for Campaign Evaluation
Step 6: Planning for program
evaluation
• Types of evaluation
– Process evaluation
– Outcome evaluation
– Economic evaluation
• Evaluation designs
– Non-experimental
– Quasi-experimental
– Experimental (randomized control trials)
• Statistical issues
– Sample size
– Data analysis
Step 6: Planning for program
evaluation
• Types of evaluation
– Process evaluation:
• Did the implementation of the campaign take place as planned?
• Where there any external factors disrupting the campaign process? If
yes, what and how to prevent next time?
• How many paid and unpaid stories came out in the media?
• Objective exposure
– Number of TV/radio spots, audience figures, number of billboards, …
– TARP = Target Audience Rating Points (TARP>100)
• Subjective exposure
– Was the target audience aware that the campaign took place (campaign
awareness and message recall)
– Did the target audience understand the campaign message?
Step 6: Planning for program
evaluation
• Types of evaluation
– Outcome evaluation:
• Road crashes, injuries, fatalities (e.g. from RTAVIS).
However, beware that these may not be the direct result
of your campaign!
• Behavior as outcome measure
– V85 speed, median or average speed
– BAC levels
– Helmet wearing
How to measure: Observed behavior, self-reported behavior,
number of offences. Beware of confounding factors: use of
control group!
Step 6: Planning for program
evaluation
• Types of evaluation
– Economic evaluation:
• Cost of the campaign
– Development of the campaign
– Cost of media placement (e.g. TV advertising)
– Beware: direct vs. indirect costs
• Benefits of the campaign
– Benefits to the society due to reduced number of fatalities,
serious injuries and slight injuries
» E.g. willingness-to-pay method
» E.g. value of a victim (e.g. in Europe: 2.5 million
dollar/fatality)
Step 6: Planning for program
evaluation
• Types of evaluation
– Economic evaluation (continued):
• Method of evaluation
– Cost Effectiveness Analysis (CEA) = when monetary evaluation
of the benefits is not possible.
– E.g. Campaign or activity ‘X’ is most cost-effective in reducing
the speed per km/h
– Cost Benefit Analysis (CBA) = when it is possible to value the
benefits in a monetary way.
– E.g. For each dollar spent in campaign or activity ‘X’ society’s
benefits are ‘$’ (cost-benefit ratio)
Step 6: Planning for program
evaluation
• Evaluation designs
– Non-experimental = simple before-after comparison
for a specific outcome measure in the treatment
group without use of comparison group
Before
Campaign
After
Target audience
– Limitations:
• Several confounding factors: you are not sure that what
you measure is really the effect of your campaign
Step 6: Planning for program
evaluation
• Evaluation designs
– Quasi-experimental = a before-after analysis using treatment
(exposed to treatment, e.g. 16-25 years old) and comparison
group (not exposed to treatment, e.g. 40-45 years old)
Before
Campaign
After
Target audience
Comparison group
– Limitations:
• Individuals are not randomly allocated to either group. Ideally, the
comparison group should contain 16-25 years old who were not
exposed to the campaign, but sometimes this is practically not
possible.
Step 6: Planning for program
evaluation
• Evaluation designs
– Experimental (=randomized control design) = before-after study
where the subjects are randomly allocated to either the
treatment group or control group
Before
Campaign
After
Target audience
Control group
– Limitations:
• Theoretically the most powerful evaluation method, however
sometimes hard to implement in practice. For instance, when the
target audience is the entire population of a particular age group.
Step 6: Planning for program
evaluation
• Some more comments about evaluation designs
– When to evaluate?
• Immediate after-period analysis
• Longer-term after-period analysis (e.g. a number of months after
the campaign has finished) so evaluate the long-term effect of the
campaign
– How to isolate effects of several campaign activities?
Pre/post
measurement
Group 1
Group 2
Group 3
Group 4
Media
Enforcement
No program
X
X
X
X
X
Step 6: Planning for program
evaluation
• Statistical issues
– Sample size depends on a number of issues
• Statistical issues
– Power of the test (we want to maximize this probability)
» Null hypothesis states there there is no observable
difference between the outcome in the treatment group
versus the control group
» Power = the probability, assuming that the null hypothesis is
false (i.e. effect of campaign is significant) of obtaining a
result that will allow the rejection of the null hypothesis. So,
if there is an effect, we want to maximize the probability of
observing it in the data
» Typically, power >= 0.8
Step 6: Planning for program
evaluation
• Statistical issues
– Sample size depends on a number of issues
• Statistical issues (continued)
– Significance of the test (we want to minimize this probability)
» Probability to measure an effect if it is not there (false positive)
» Typically, significance <=0.05
– Difference: what difference between the treatment group and
comparison/control group does the test need to detect?
» E.g. we want the test to find a difference of 10% between both
groups
» Decreasing this number will increase the number of respondents
that you need in both groups
• Practical issues
– Budget available for carrying out the evaluation study
Step 6: Planning for program
evaluation
• Example of sample sizes
Power
Significance Difference
0.8
0.05
5%
Sample size
each group
1,237
0.8
0.05
10%
310
0.8
0.05
20%
78
Step 6: Planning for program
evaluation
• Statistical issues
– Data analysis
• Descriptive statistics
– Comparing sample mean, standard deviation
– Simple graphs: histograms, box plots, scatter diagrams, line charts
• Inferential statistics
– Hypothesis testing: testing whether the observed difference in the
outcome measure between the treatment group and the control group is
statistically significant
» E.g. T-test
– Advanced methodologies
» Time series analysis
» Structural equation modelling
Step 6: Planning for program
evaluation
Example structural
equation model