Transcript Slide 1

June 24, 2009
MODULE II
Monitoring and Evaluating
Gender-based Violence
Prevention and Mitigation
Programs
Final
Anastasia J. Gage
Melissa Dunn
Learning Objectives
 By the end of this session, participants will be
able to:
 Differentiate between monitoring and evaluation
 Write goals and smart objectives for GBV
programs
 Design a logic model for a GBV program
 Identify criteria for indicator selection and
information sources for GBV indicators
 Discuss factors to consider when choosing an
evaluation design
At Least 4 Different Categories of GBV
Interventions
 Community mobilization
 Behavior change communication
 Service delivery
 Laws and public policy
M&E Challenges of GBV Programs (1)
 Hard to separate contribution of multiple
strategies
 Lack of investment in rigorous evaluation
 Limited evidence of effectiveness
 Non-standardized GBV definitions and indicators
 Enhancing comparability of data
 Climate of urgency around results but behavior
change is long-term
M&E Challenges of GBV Programs (2)
 Sensitive nature of the information and hidden
forms of violence (femicide, trafficking, FGC)
 Changing violence levels hard to measure and
interpret
 Ethical and methodological issues
Monitoring versus Evaluation
MONITORING =
• Tracking changes in program performance over
time
EVALUATION =
• Assessing whether objectives have been met
• Assessing extent to which program is
responsible for observed changes
Illustration of Program Monitoring
Program
indicator
Program
start
TIME->
Program
end
Illustration of Program Impact
With program
Without program
Change
in
Program
Outcome
Program
impact
Program
start
TIME->
Program
end
Activity 1: Is it Monitoring or
Evaluation?
 The Ministry of Women’s Affairs wants to know if
programs carried out in Province A are reducing the
prevalence of intimate partner violence (IPV).
 USAID wants to know how many villages in Region B
have been reached with anti-GBV messages by your
program this year.
 A country director is interested in finding out if the
care provided to rape victims in public clinics meets
national standards of quality.
Purpose of Monitoring and Evaluation
The purpose of monitoring and
evaluation is to measure program
effectiveness.
Monitoring and Evaluation Questions
 Were resources made available to the program in the
quantities and at the times specified by the program plan?
 Were the program activities carried out as planned?
 Which program activities were more effective and which
were less effective?
 Did the expected changes occur? How much change
occurred?
 Can improved health outcomes be attributed to program
efforts?
 Did the target population benefit from the program and at
what cost?
M&E Across Program Life Cycle
ASSESSMENT
What is the nature of the
(health) problem?
1
EVALUATION
How Do I know that the strategy is working?
How do I judge if the intervention is making a
difference?
5
4
IMPLEMENTATION/MONITORING
How do I know the activities are being implemented as
designed? How much does implementation vary from site
to site? How can be program become more efficient or
effective?
2
STRATEGIC PLANNING
What primary objectives should my
program pursue to address this problem?
3
DESIGN
What strategy, interventions and
approaches should my program
use to achieve these priorities?
Discussion: Key Elements of M&E Plan
 Do participants’ programs have M&E plans?
 What are the components of the plans? What
sections do the plans include?
 Distribute Handout 1: Template for M&E plan
 What is missing from their programs’ M&E plans?
 What additional components do their programs’
M&E plans include?
Activity 2: Identifying the Problem
 Get into 4 groups representing the different categories of
GBV intervention:
 Community mobilization
 Behavior change
 Service delivery
 Laws and public policy
 Review Handout 2: Gender-Based Violence Case Study
 What is the problem from your group’s perspective?
 What are 2-3 barriers your GBV intervention group might
face when trying to help women like Marie?
Program Goals
A goal is a broad statement of a desired, longterm outcome of a program. Goals represent
general big picture statements of desired results.
 Examples:
 To improve the lives of women who experience
gender-based violence
 To decrease injury and mortality due to genderbased violence
 To eliminate gender-based violence
Program Objectives
 Objectives: statements of desired, specific, realistic
and measurable program results
 SMART
 Specific: identifies concrete events or actions that will take place
 Measurable: quantifies the amount of resources, activity, or
change to be expended and achieved
 Appropriate: logically relates to the overall problem statement and
desired effects of the program
 Realistic: Provides a realistic dimension that can be achieved
with the available resources and plans for implementation
 Time-bound: specifies a time within which the objective
will be achieved
Source: GAP 2003
How to Write Program Objectives
To
[action]
The
[specify what knowledge, attitudes, skills, behaviors]
Among
[specific population or segment]
From – to
[from baseline to desired level]
By
or by
[ x percent ]
or to
[specific level
[time frame]
Activity 3: Is it a Goal or an Objective?
 To reduce GBV
 To increase the % of men and women in
beneficiary population who believe that violence
is not an acceptable way of dealing with conflict
from 40% in 2002 to 80% by 2007
 To increase the number of domestic violence
clients recommended for counseling who utilize
counseling services by 30%
Activity 4: Defining Program Goals and
Objectives
 Return to small groups from previous activity
 Develop one goal and two objectives for a
program that will address problems identified
in Handout 2 (GBV Case Study)
 After 15 minutes, a member of each group
will share the program goal and objectives
Developing Program Strategies
Role of a Conceptual
Framework
Conceptual Frameworks
Diagram that identifies and illustrates the relationships between all relevant
systemic, organizational, individual, or other salient factors that may influence
program/project operation and the successful achievement of program or project
goals.
M&E Purpose:





To show where program fits into wider context
To clarify assumptions about causal relationships
To show how program components will operate to influence outcomes
To guide identification of indicators
To guide impact analysis (causal pathways)
Illustrative Conceptual Model for IPV
Society
Community Relationship
Individual
Perpetrator
Society
•Norms
•Laws
•Notions of
masculinity
•Gender roles
•Poverty
•Unemployment
•Family
isolation
•Community
acceptance of
violence
•Marital conflict
•Male control of
wealth
•Male control of
decision-making
in family
Source: Heise 1998
•Being male
•Witnessing
marital violence
as child
•Being abused as
child
•Absent or
rejecting father
•Alcohol use
Activity 5: Developing Program
Activities
 Return to small groups from previous activity
 Develop 3 program activities for the goal and
objectives that the group had formulated
earlier to address the problems identified in
the GBV case study
 In 15 minutes, a member of each group will
share the program strategies and activities
Logic Models
What is a Logic Model?
Logic models are diagrams connecting
program inputs to processes, outputs,
outcome and impact as they relate to a
specific problem or situation. Logic
models show what resources your
program will need to accomplish its
goals, what your program will do, and
what it hopes to achieve, emphasizing
the links between all these things.
Logic Model Components
 Input – Resources used in a program (e.g., money, staff,
curricula, and materials)
 Process – Activities conducted by program to accomplish
its objectives (e.g., outreach, advocacy, training, BCC,
etc.)
 Output – Direct results of program activities (e.g., people
trained; materials distributed; couples counseled, etc.)
 Outcome – Program results at population level (e.g.,
knowledge, attitudes, behavior, coverage)
 Impact – Long-term change in morbidity, mortality, and
fertility
Activity 6: Identifying Logic Model
Components
 Distribute Handout 5
 Each of the scenarios in the handout
corresponds to one of the five components in the
logic model: Input, process, output, outcome, and
impact.
 As a group, decide which component the
scenario illustrates.
Illustrative Logic Model for Provider
Training Program
INPUT
PROCESS
OUTPUT
OUTCOME
IMPACT
•Human and
financial
resources to
develop
training
materials &
implement
training
program
•Develop
GBV clinical
training
curriculum
•Providers
trained in
GBV
•Increased
awareness of
GBV as a health
issue
•Improved
health and
safety of GBV
victims
•Conduct
TOT
workshops
•Conduct
GBV training
for providers
•Improved
provider
attitudes
toward GBV
•Increased
provider
ability to
identify,
counsel, care
for, and refer
GBV victims
•Increased
disclosure of
GBV
•Increased
knowledge and
utilization of
GBV services
Role of Logic Model
 Program planning
 Helps one think through program strategy – where
you are & where you want to be
 Program management
 Helps one track and monitor operations to better
manage results
 Communication
 Shows stakeholders at a glance what program is
doing and what it expects to achieve
Role of Logic Model
 Consensus building
 Promotes common understanding about what
program is, how it works and what it is trying to
achieve
 Fundraising
 Can structure and streamline grant writing
 Monitoring and evaluation
 Provides thorough understanding of what resources
program has to work with, what it is doing, and what it
hopes to achieve.
Activity 7: Developing a Logic Model
 Return to small groups from previous
activities
 Develop a logic model for two of the
activities of your program
 After 20 minutes, a member of each group
will share the logic model
Results Frameworks
Results Frameworks
Diagrams that identify steps or levels of results and illustrate the
causal relationships linking all levels of a program’s objectives.
Purposes:
 Provides a clarified focus on the causal relationships that connect
incremental achievement of results to the comprehensive program
impact
 Clarifies project/program mechanics and factors’ relationships that
suggest ways and means of objectively measuring the
achievement of desired ends
Other terms used:
 Strategic frameworks
Illustrative Results Frameworks
SO1: Increased Utilization of Reproductive Health Services
IR1 Strengthened sustainability
of GBV programs
IR1.1 Improved policy
environment for GBV
prevention and response
IR2 Expansion of high-quality health
services for GBV victims in the public and
private sectors
IR2.1 Increased availability of
GBV screening and referral
IR2.2 Improved provider attitude
towards GBV
IR1.2 Strengthened NGO
advocacy for GBV prevention
IR2.3 Improved provider competence in GBV
screening, referral, care, & counseling
Activity 8: Develop Monitoring and
Evaluation Questions
 Return to your previous group activities where
you developed your program goals and
objectives and logic model
 Come up with at least 1 monitoring question and
1 evaluation question
 After 10 minutes, a member of the group will
share the M&E questions
Selecting Indicators
What is an Indicator?
An indicator is a specific, observable and
measurable characteristic that can be used to
show changes or progress a program is making
toward achieving a specific outcome.
Characteristics of Good Indicators
 Valid: accurate measure of a behavior, practice or task
 Reliable: consistently measurable in the same way by different
observers
 Precise: operationally defined in clear terms
 Measurable: quantifiable using available tools
and methods
 Timely: provides a measurement at time intervals relevant and
appropriate in terms of program goals and activities
 Programmatically important: linked to the program or to
achieving the program objectives that are needed for public
health impact
Indicators do not specify a
particular level of achievement -the words “improved”,
“increased”, or “decreased” do
not belong in an indicator.
Common Indicator Metrics
 Counts
 No. of legal and service organizations for VAW/G survivors
 No. women and children using VAW/G social welfare services
 Calculations: percentages, rates, ratios
 % of health units with at least one service provider trained to care
for and refer survivors
 % of women aged 15-49 who have ever experienced physical
violence from an intimate partner
 Index, composite measures
 Gender-equitable men (GEM) scale
 Sexual relationship power scale
 Thresholds
 Presence, absence; pre-determined level or standard
Common Challenges in Indicator
Selection
 Choosing an indicator that program activities
cannot affect
 Choosing an indicator that is too vague
 Indicators that do not currently exist and cannot
realistically be collected
 Selecting an indicator that does not accurately
represent the desired outcome
 Too many indicators
How Many Indicator Are Enough?
 At least one or two indicators per key activity or result
(ideally, from different data sources)
 At least one indicator for every core activity (e.g., training,
BCC)
 No more than 8-10 indicators per area of significant
program focus
 Use a mix of data collection strategies/source
Factors to Consider When Selecting
Indicators
 Logic/link to framework
 Programmatic needs/information for
decision making
 Data availability
 Resources
 External requirements (government, donor,
headquarters)
 Standardized indicators (if available)
Activity 9: Selecting Indicators
 Return to small groups from previous activities
where you developed goals, objectives, activities,
and M&E questions
 Select 3 indicators that your group might use to
measure progress towards your program goals and
objectives
 Into which logic model component does each
indicator fall?
 After 15 minutes, a member of each group will
share the indicators
Activity 10: Selecting Indicators Cont’d
 As a class, look at the indicators that the groups selected
in the previous activity
 Discuss the following:
 To what extent are the indicators logically linked to the
group’s M&E framework?
 Do the proposed indicators share the characteristics of good
indicators?
 How will the indicators be used for program decision-making
and for what decisions?
 Are data available to measure the indicators?
 Can the indicators be realistically collected given available
resources?
 Are there government or donor requirements for measuring
the proposed indicators?
Where to Find Standardized Indicators
for GBV Programs
Bloom, Shelah S. 2008. Violence Against Women and Girls: A
Compendium of Monitoring and Evaluation Indicators. Ms.08-30.
Chapel Hill, NC: Carolina Population Center, the University of North
Carolina and Chapel Hill.
http://www.cpc.unc.edu/measure/publications
UN Division for the Advancement of Women (UNDAW), UN
Economic Commission for Europe (UNECE) and UN Statistical
Division. 2008. Indicators to measure violence against women.
Report of the Expert Group Meeting, 8 to 10 October
2007, Geneva, Switzerland; UN Human Rights Council. 2008.
Report of the Special Rapporteur on violence against women, its
causes and consequences by Yakin Ertürk. Indicators on violence
against women and State response. Advance edited version (28
January 2008).
Illustrative Indicators: Community
Mobilization
 Proportion of people who would assist a woman
being beaten by her husband or partner
 Proportion of people who say that wife beating is an
acceptable way for husbands to discipline their wives
 Proportion of people who agree that rape can take
place between a man and woman who are married
 Proportion of youth-serving organizations that include
training for beneficiaries on sexual and physical
VAW/G
Illustrative Indicators: Behavior
Change Communication
 Proportion of people who have been exposed to
VAW/G prevention messages
 Proportion of girls who say that they would be willing
to report any experience of unwanted sexual activity
 Proportion of girls that feel able to say no to sexual
activity
 Proportion of individuals who know any of the legal
sanctions for VAW/G
Illustrative Indicators: Service Delivery
 Proportion of health units that have documented and
adopted a protocol for the clinical management of
VAW/G survivors
 Proportion of health units with at least one service
provider trained to care for and refer VAW/G
survivors
 Proportion of women who were asked about physical
and sexual violence during a visit to the health unit
 Proportion of rape survivors who received
comprehensive care
Illustrative Indicators: Law and Public
Policy
 Proportion of law enforcement units following a
nationally established protocol for VAW/G
complaints
 No. of law enforcement professionals trained to
respond to incidents of VAW/G according to an
established protocol
 No. of VAW/G complaints reported to the police
 Proportion of VAW/G cases that were
investigated by the police
Measuring Indicators
Indicator 1: # of service providers trained (in the past
year or other period) to identify, refer, and care for VAW/G
survivors
“providers” include any clinician providing direct clinical
services to clients at public health facilities
“trained” refers to attending every day of a five-day training
course on VAW/G
Measuring Indicators
Indicator 2: % of women who were asked about physical
and sexual violence during a visit to a health unit
Numerator: Number of women who were asked, during the
course of their service provision at the health unit, about
any violence that had ever occurred, either physical or
sexual, in the geographic area of study (nation, province,
state, community).
Denominator:
Record review: Total number of women’s records that
were reviewed at the health unit.
Exit interview: Total number of women interviewed.
Activity 11: Operational Definition of
Indicators
 Distribute Handout 8
 Get into 4 groups
 Write an operational definition for one of the following
indicators:
 Intimate partner physical violence
 Appropriate care for rape survivors
 Gender-norm related attitudes that reinforce VAW/G
 After 15 minutes, a member of each group will share the
operational definitions
Measuring Composite Indicators
 Example: How to measure, “attitudes toward GBV” ?
 What question do you ask?
 How do you know any one question is a good
representation of general attitude?
 How do you scale each question (agree/disagree;
strongly disagree to strongly agree; yes/no)?
 How do you combine answers to the questions to
come up with one indicator?
 What value or weight do you assign to any one
question on GBV attitude in the survey?
Example of Measuring Attitudes
Toward Wife Beating in the DHS
 Sometimes a man is annoyed or angered by things that
his wife does. In your opinion, is a man justified in hitting
or beating his wife in the following situations:
 If she goes out without telling him?
 If she neglects the children?
 If she argues with him?
 If she refuses to have sex with him?
 If she burns the food?
Activity 12: Specifying Indicator
Metrics
 Return to small groups from previous
activities
 Look at the indicators that your group has
selected to measure progress towards your
goals and objectives
 Define the metrics for each indicator
Indicator Reference Sheets
Distribute Handout 9
Setting Indicator Targets: Useful
Information Sources
 Past trends
 Client expectations
 Donor expectations
 Expert opinion
 Research findings
 What has been accomplished elsewhere
Activity 13: Setting Indicator Targets
Based on Past Trends
One of the objectives of the Ministry of Women’s Affairs in
Country X is to reduce the prevalence of intimate partner
violence. The 2000 DHS showed that 30% of ever
married women aged 15-49 years in the country had ever
experienced intimate partner emotional, sexual or physical
violence. A comparable DHS conducted in 2005-2006
showed that the prevalence of intimate partner violence
(emotional, physical, or sexual) was 25% among evermarried women aged 15-49. Based on these trends, what
would be a realistic national target for this indicator in
2010?
Types of Information Sources
 Quantitative data
 Useful for tracking trends accurately and
highlighting differences
 Qualitative data
 Useful for understanding the context in which the
trends/differences occur and to interpret
quantitative data accurately (e.g. focus groups; indepth interviews)
Quantitative Information Sources
 Population-based surveys
 DHS (domestic violence module); VAW survey
 Facility surveys
 Client exit interviews; provider surveys; clinic
observations
 Health service statistics
 Program statistics
 Crime statistics (of poor quality in many LDCs)
 Behavioral surveillance systems
Triangulating Data Sources: Soul City
Community
6. Cost – Effectiveness Study
Individual
1. National Survey
2. Sentinel Site Studies
3. (National) Qualitative
Impact Assessment
4. NNVAW Partnership
Study
5. Media Monitoring
and Analysis
Society
Activity 14: Identifying Information
Sources for Indicators
 Return to small groups from previous activities
 Look at the indicators that your group has selected to
measure progress towards your program goals and
objectives
 For each indicator, identify the potential source of
data needed for its calculation
 What challenges might you face in using these
potential information sources?
 After 10 minutes, a member of the group will present
Guiding Questions for Choosing an
Evaluation Design
 What question does your program need to answer?
 What do you want to measure (indicators)?
 How sure to you want to be? (VERY SURE = GOOD )
 What is the cost of making a mistake (low,
medium, high)?
 When do you need the results? (FAST)
 How much are you willing to pay? (CHEAP)
 Has the program already started?
Choosing an Evaluation Design
GOOD, FAST, CHEAP:
PICK 2!
HIRE SOMEONE TO HELP YOU
OUT!
NO EVALUATION DESIGN IS
IMMUNE TO PROBLEMS!
Impact Evaluation Needs Specific Design
Evaluating program
impact; interested in
attribution/causality
Experimental (with
randomization) or quasiexperimental design
Program already
implemented; no
baseline
Post-test only design (then
multivariate analysis)
Program not yet
implemented
Pretest-post-test design;
time series design
When monitoring and evaluating
GBV programs, the
confidentiality, privacy, safety,
and well-being of GBV survivors
must be top priority at all times.
References
 Bloom, Shelah S. 2008. Violence Against Women
and Girls: A Compendium of Monitoring and
Evaluation Indicators. Ms.08-30. Chapel Hill, NC:
Carolina Population Center, the University of North
Carolina and Chapel Hill.
 Bott S, Guedes A, Claramunt MC, Guezmes A.
Improving the Health Sector Response to GenderBased Violence: A Resource Manual for Health Care
Professionals in Developing Countries. 2004. IPPF.
 IGWG. 2004. Addressing Gender-Based Violence
from the Reproductive Health/HIV Sector: A Literature
Review. Washington, DC: POPTECH.
References Cont’d.
 Soul City 4. Impact Evaluation; Violence Against
Women, Vols. I and II. 2001.
 WHO. 2007. WHO Ethical and Safety
Recommendations for Researching, Documenting
and Monitoring Sexual Violence in Emergencies.
Geneva, Switzerland: World Health Organization.
http://www.who.int/hac/network/interagency/news/ethica
l_and_safety_recommendations/en/index.html