Transcript Document
Session C: Evaluation James Farnham 3 June 2009 1 Content Context for evaluation Principles of evaluation Stepped approach to evaluation 2 The context for evaluation 3 Evaluation benefits Continuous learning and improvement ‒ improved effectiveness of marketing programmes ‒ improved marketing efficiency Better knowledge and insight for how to influence consumer behaviour Accountability More informed budgeting process Better forecasting of outcomes ‒ better management of expectations re results ‒ input to ‘policy levers’ ‒ enhanced credibility of social marketing 4 Moving from a static, linear approach to a more dynamic, continuous improvement journey Evaluation Evaluation Planning Planning Test-Learn-Refine (TLR) 5 Evaluation perspectives Accountability Vs Test-Learn-Refine ‘What gets measured is what matters’ Scientific realists Vs Social constructionists ‘No single accepted best practice’ Theory Vs Practice 6 A ‘line of sight’ to behaviour change is required across a range of measures Intermediate measures eg attitudes and response Outtake measures eg understanding & knowledge Behavioural change measures Reduced consumption Outcome/impact measures Reduced ARHA Output measures eg awareness Input measures eg toolkits, TVRs 7 Principles of evaluation 8 Ten marketing evaluation principles Better evaluation will improve both our marketing results and our accountability – it is essential, not a ‘nice to have’ Five themes ‒ Culture ‒ Process ‒ Measurement ‒ Influencing behaviour ‒ Test-Learn-Refine 9 Ten marketing evaluation principles Culture 1. Hard wire evaluation into every stage of the marketing process, focusing on active learning and rapid application 2. Evaluate every piece of activity at a proportionate level between 5% and 10% of total spend 10 Ten marketing evaluation principles Process 3. Develop a formal evaluation plan at the outset of the marketing process, including clear objectives. Each plan will address the following questions: What will be evaluated? (i.e. what is the marketing activity about and in what Policy context does it exist?) What aspects of the marketing activity will be considered when judging its performance? What standards (i.e. type or level of performance) must be reached for the marketing activity to be considered successful? What evidence will be used to indicate how the marketing activity has performed? What relevant lessons from previous marketing activities can be applied? What conclusions regarding marketing activity performance are justified by comparing the available evidence to the selected standards? How will the lessons learned from the evaluation be used to improve public health effectiveness? 11 Ten marketing evaluation principles Measurement 4. Collect robust, consistent data on marketing activity 5. Ensure all marketing evaluations focus on a range (c. 10) of KPI measures that are critical to success – both in terms of behavioural influence and ROI/cost-effectiveness. Avoid using awareness as an isolated measure of success Influencing behaviour 6. Put behaviour change at the heart of the evaluation thinking, capitalising on existing behavioural theory 12 Ten marketing evaluation principles Test-learn-refine 7. Use a rigorous test-learn-refine process within and between channels 8. Try to test locally prior to full PCT exposure, establishing control areas if possible 9. Evaluation should be used to encourage innovation, not create risk aversity 10. Question activities that cannot be evaluated 13 Stepped approach to evaluation 14 Is it a step? NSMC Total Process Planning Model 15 PHASE 1: SCOPE - gaining early clarity Step 1 Form an Evaluation Team Step 2 Lessons learned from previous evaluations Step 3 Clarifying objectives Step 4 Setting the evaluation budget Step 5 Evaluation plan 1 16 PHASE 2: DEVELOP – working up the overall approach Step 1 Using behavioural theory Step 2 Defining the behavioural goals Step 4 Writing the Evaluation Plan Step 5 Briefing the implementation agencies Step 3 Defining the KPIs 17 PHASE 3: IMPLEMENT – iteratively designing the evaluation detail with all agencies Step 1 Evaluation development with implementation agencies Step 2 Designing and setting up the measurement approaches Step 3 Getting the most out of evaluation 18 PHASE 4: LEARN – short term and long term Step 1 Immediate post-launch reporting Step 2 Interim Evaluation Step 3 Final Evaluation Report Step 4 Sharing and capturing the learnings from the Evaluation Report 19