Transcript Document

Communication Strategy Seminar
Evaluation of communication
activities
Glenn O’Neil
24 September 2014
Schedule
• 30 minute presentation
• 20 minute case study
• 20 minute discussion on case study and Q&A
What is communications?
Programmes, projects, campaigns and
activities that are dedicated to the
management of communications
between an organisation and its publics
Source: Grunig, J. (ed.) (1992). Excellence in Public Relations and Communications
3
Communication components
– A programme is an organised set of communication
activities based on target audiences, themes or
functions running continuously or for long periods
– A campaign is an organised sub-set of
communications activities, directed at a particular
audience usually within a specified period of time to
achieve specific outcomes
– A product is an individual object, such as a
publication, website or video created to support a
communication activity
4
What is evaluation?
“Evaluation is the systematic assessment of the operation and/or
the outcomes of a program or policy compared to a set of explicit
or implicit standards, as a means to contributing to the
improvement of the program or policy”
source: “Evaluation”, by Susan Weiss, (1998)
Communication evaluation?
“A form of research that determines the relative effectiveness of a public
relations campaign or program by measuring program outcomes against
a predetermined set of objectives that initially established the level or
degree of change desired”
source: “Stacks, D. (2006). Dictionary of Public Relations Measurement and Research. Institute for Public Relations.”
5
Aim of communications?
6
Why evaluate communications?
• Effectiveness: understand what has
changed - did you change awareness,
attitudes and behaviors?
• Efficiency: understand what works best
- what channel, tool or activity – or
combination – worked best with your
audiences?
7
Process of communication evaluation
• Evaluation needs to be considered in
the planning phase but it is mostly a
post-activity action
• Theory is good to know:
– Communication / media / audience effects theories
– Theory of change
• Practical tools are also good to know:
– Communication plan of action / Logical frameworks / resultsbased management systems
– Measurement tools and methods
8
Communication plan of action: Example!
Goal: Our supporters and stakeholders understand how we are
using our funds in our programmes in developing countries
Objective 1: 80% of our regular supporters understand better our
use of funds and maintain their donations at the current level
Objective 2: Key political figures improve their understanding and
speak out in our support
Target audiences: current and potential supporters (mainly
women in their 40s), media, politicians
Activities: mini-campaign of e-news, postcards, travelling
exhibition, social media, celebrity endorsement, presentations in
parliament.
Messages: Focus on human element (“Czechs abroad” and
southern “success stories” of people supported); share
breakdown of funds and how used.
Evaluation indicators:
For objective 1: level of understanding of regular supporters on
use of funds; level of donations received
For objective 2: Number of political figures who publicly support
our organisation and our work
9
When to evaluate?
Communication activities
Before
During
10
After
What and how to evaluate
Framework
Level of effect
Evaluation tools
Quality and appropriateness
of activity, message and
channel
Case studies, focus groups, expert reviews,
secondary data
Number of communications
produced, distributed and
received
Distribution statistics, event attendance,
website visits and downloads
Number of communications
visible in the media
Media monitoring (software or manual), web
metrics software, media distribution statistics,
content analysis (software or manual).
Changes to
knowledge, attitude &
behaviour
Surveys, interviews, focus groups, tracking
mechanisms, web metrics software.
Changes to policies,
activities & practices of
targeted institutions
Case studies, interviews, observation studies,
tracking mechanisms.
Changes at national or
sector level
Surveys, case studies, secondary data.
Input
Formative
evaluation
Output
Process evaluation
Outcome
Summative
evaluation
Impact
Impact
evaluation
Evaluation framework
Objectives
Indicators
Objective 1: 80% of our
regular supporters
understand better our
use of funds and
maintain their donations
at the current level
Level of understanding of Online survey of regular
regular supporters on
supporters
use of funds
Objective 2: Key political
figures improve their
understanding and speak
out in our support
Evaluation tools
Source
Email list of regular
supporters
Level of donations
received
Donation tracking (online Supporters donations
and offline)
(available from
Fundraising Unit)
Number of political
figures who publicly
support our organisation
and our work
Tracking log to monitor
politicians’ statements
12
Media reports,
parliament records,
politicians statements
and websites
What can we find out? Example!
Reach: Estimated 250,000 people reached by our
communications
Visibility: Media reports in 5 online media and 3
newspapers – focus on Czech staff working in crises;
most articles positive or neutral, one negative (“money
wasted”)
Knowledge: 60% of supporters surveyed said they had
understood better how funds are used
Attitudes: Preference for human stories – “success
stories” over “hard facts”; dislike of picturing people as
victims; need to see where money donated is going
Action: 20% of people reached increased their donations
Action: 2 out of 10 people on social media engaged with
us
Support: 3 political figures spoke up in our favour
Tools: 9 out of 10 people heard of the issue through social
media; highest donations received at exhibition; no one
mentioned celebrity involvement
as a message
13
How can we find out?
Reach: Estimated 250,000 people reached by our
communications
Visibility: Media reports in 5 online media and 3
newspapers – focus on Czech staff working in crises;
most articles positive or neutral, one negative (“money
wasted”)
Knowledge: 60% of supporters surveyed said they had
understood better how funds are used
Attitudes: Preference for human stories – “success
stories” over “hard facts”; dislike of picturing people as
victims; need to see where money donated is going
Action: 20% of people reached increased their donations
Action: 2 out of 10 people on social media engaged with
us
Support: 3 political figures spoke up in our favour
Tools: 9 out of 10 people heard of the issue through social
media; highest donations received at exhibition; no one
mentioned celebrity involvement
as a message
14
Output / reach statistics
– A lot of statistics exists (or you have to collect
them) to measure your output and reach, for
example:
- Number of people attending events
- Number of people reached through media
in which you receive coverage
- Number of people visiting your website or
social media pages
Note: this can be interesting and useful
information but remains superficial!
15
Media visibility tools
– These tools help you to measure the coverage you receive
in the media – online and offline
– Many commercial tools exists (subscription-based) but you
can do a lot for free, e.g. Google News alerts on your
organisation or issue
– What you can measure, for example:
- Number of mentions in our targeted media
- Tone (negative, positive, neutral) of media coverage
- Importance of media coverage, e.g. type of media, place
in newspaper or website, etc.
- Comparison with similar organisations and their coverage
- How issue/organisation is being covered in media
Note: Media coverage is an indication of the visibility in the
media of the issue and/or your organisation - but it doesn’t
tell you what people know or think!
16
v.
0
de 7
c.
ja 07
nv
.0
fe 8
b
m .08
ar
s.
0
ap 8
ril
.0
m 8
ay
.
ju 08
ne
.0
ju 8
ly
.0
au 8
g
se .08
pt
.0
8
oc
t.0
no 8
v.
0
de 8
c.
0
ja 8
nv
.0
9
no
Media monitoring - example
%
of coverage
35
30
25
20
15
10
5
0
UDHR
Human Rights
Geneva Conventions
17
Online surveys
– Online surveys are an effective way to measure what
people know, think or do about an issue or organisation
– Many survey tools exist, are easy to use - with some free of
charge for basic surveys (e.g. Surveymonkey - 10 free
questions and 100 free responses per survey)
– What you can measure, for example:
- What people know/think/feel about an issue/organisation
- What people have done or will do
- Feedback on a communication programme, campaign
or product
It is easier to survey audiences that you already know – i.e.
a supporter base or partners (you have their emails); it is
harder (more expensive) to survey an audience you don’t
know, e.g. the general public
18
Focus groups / interviews
– These tools can be used to go in-depth on an issue and
understand “why” and “how”
– Focus groups are usually 8-12 people gathered together
for a guided discussion – guides are available as to how to
run them; Interviews are normally 1-1 discussion
– What you can measure, for example:
- What an audience understands in your communication
messages
- What people know/think/feel about an issue/organisation
- Feedback on a communication programme, campaign or
product
Often these methods can be combined with other methods
such as surveys or web metrics to provide more insights
and detailed feedback
19
Web metrics
– Web metrics are for measuring online activity
– Most websites have web metrics tools (e.g. Google Analytics) as
do most social media (e.g. Facebook Insights page)
– You can also measure general trends online, e.g. using Google
Trends or visitor trends on comparative websites, e.g. Alexa.com
– What you can measure, for example:
- Level of interest in your issue/organisation (no. of page visits)
- Differences in interest in issues/content
- Demographic information of your online audiences
- Increase in online membership/subscriptions/donations
- Level of online engagement (no. of share/likes/comments)
- Trends over time concerning an issue/organisation (google
Trends)
20
Web metrics - example
21
Tracking mechanisms
– Tracking mechanisms record actions taken on issues,
policies, legislation, etc.
– Tracking mechanisms are usually manually tracked on
standard forms in a systematic manner
– What you can measure, for example:
- Recording how many partners join a campaign
- Tracking and recording the number of business leaders
that speak out on an issue
- Tracking how an issue is acted upon by partners/target
audiences, e.g. Its appearance on meeting agendas
22
Tracking mechanisms - example
23
Making sense of it all
– You’ve collected some information – what does it tell you?
– Look back at your original goal and objectives – how did
you do?
– OR you’ve finished a major communication activity but
collected no information systematically – what should you
do?
– Look at what feedback you can collect now, gather the
team and assess together what you achieved or not
– Two points to consider in both scenarios:
- Focus on what your conclusions mean for future
activities
- Make a short report of your conclusions – a written
record is important for colleagues and future actions
24
Case study
– Read the case study
– Complete the evaluation matrix
– Then we will discuss the possible solutions…
25
Contacts & further resources
Contacts:
[email protected]
@glenn_oneil
glennoneil
www.owlre.com
www.intelligentmeasurement.net
26