Transcript Slide 1

MODULE II
Monitoring and Evaluating
Gender-Based Violence
Prevention and Mitigation
Programs
Anastasia J. Gage
Melissa Dunn
Learning Objectives
 By the end of this session, participants will be
able to:
 differentiate between monitoring and evaluation
 write goals and smart objectives for gender-based
violence (GBV) programs
 design a logic model for a GBV program
 identify criteria for indicator selection and
information sources for GBV indicators
 discuss factors to consider when choosing an
evaluation design
At Least 4 Different Categories of GBV
Interventions
 Community mobilization
 Behavior change communication
 Service delivery
 Laws and public policy
M&E Challenges of GBV Programs (1)
 Hard to separate contribution of multiple
strategies
 Lack of investment in rigorous evaluation

limited evidence of effectiveness
 Non-standardized GBV definitions and indicators
 Enhancing comparability of data
 Climate of urgency around results but behavior
change is long-term
M&E Challenges of GBV Programs (2)
 Sensitive nature of the information and hidden
forms of violence (homicide, trafficking, female
genital mutilation)
 Changing violence levels hard to measure and
interpret
 Ethical and methodological issues
Monitoring versus Evaluation
MONITORING =
• Tracking changes in program performance over
time
EVALUATION =
• Assessing whether objectives have been met
• Assessing extent to which program is
responsible for observed changes
PROGRAM INDICATOR
Illustration of Program Monitoring
Program
start
TIME 
Program
end
CHANGE IN
PROGRAM OUTCOME
Illustration of Program Impact
Program
start
With program
Without program
Program
impact
TIME 
Program
end
Activity 1: Is It Monitoring or
Evaluation?
 The Ministry of Women’s Affairs wants to know if
programs carried out in Province A are reducing the
prevalence of intimate partner violence (IPV).
 USAID wants to know how many villages in Region B
have been reached with anti-GBV messages by your
program this year.
 A country director is interested in finding out if the
care provided to rape victims in public clinics meets
national standards of quality.
Purpose of Monitoring and Evaluation
The purpose of monitoring and
evaluation is to measure program
effectiveness.
Monitoring and Evaluation Questions
 Were resources made available to the program in the quantities
and at the times specified by the program plan?
 Were the program activities carried out as planned?
 Which program activities were more effective and which were
less effective?
 Did the expected changes occur? How much change occurred?
 Can improved health outcomes be attributed to program
efforts?
 Did the target population benefit from the program and at what
cost?
M&E Across Program Life Cycle
ASSESSMENT
What is the nature of the
(health) problem?
1
EVALUATION
How Do I know that the strategy is working?
How do I judge if the intervention is making a
difference?
5
4
IMPLEMENTATION/MONITORING
How do I know the activities are being implemented as
designed? How much does implementation vary from site
to site? How can be program become more efficient or
effective?
STRATEGIC PLANNING
2
What primary objectives should my
program pursue to address this problem?
3
DESIGN
What strategy, interventions and
approaches should my program
use to achieve these priorities?
Activity 1 Discussion: Key Elements of
M&E Plan
 Do participants’ programs have M&E plans?
 What are the components of the plans? What
sections do the plans include?
Distribute Handout 1: Template for M&E Plan
 What is missing from their programs’ M&E plans?
 What additional components do their programs’
M&E plans include?
Activity 2: Identifying the Problem
Organize into 4 groups representing the different
categories of GBV intervention:
 community mobilization
 behavior change
 service delivery
 laws and public policy
Review Handout 2: Gender-Based Violence Case Study
 What is the problem from your group’s perspective?
 What are 2 or 3 barriers your GBV intervention group
might face when trying to help women like Marie?
Program Goals
A goal is a broad statement of a desired, longterm outcome of a program. Goals represent
general big picture statements of desired results.
Examples:
 To improve the lives of women who experience
gender-based violence
 To decrease injury and mortality due to genderbased violence
 To eliminate gender-based violence
Program Objectives
Objectives are statements of desired, specific, realistic
and measurable program results
 “SMART” acronym:
 Specific: identifies concrete events or actions that will take place
 Measurable: quantifies the amount of resources, activity, or
change to be expended and achieved
 Appropriate: logically relates to the overall problem statement and
desired effects of the program
 Realistic: provides a realistic dimension that can be achieved
with the available resources and plans for implementation
 Time-bound: specifies a time within which the objective
will be achieved
Source: GAP 2003
How to Write Program Objectives
To
[action]
The
[specify what knowledge, attitudes, skills, behaviors]
Among
[specific population or segment]
From-to
[from baseline to desired level]
By
or by
[ x percent ]
or to
[specific level]
[time frame]
Activity 3: Is it a Goal or an Objective?
 To reduce GBV
 To increase the % of men and women in
beneficiary population who believe that violence
is not an acceptable way of dealing with conflict
from 40% in 2002 to 80% by 2007
 To increase the number of domestic violence
clients recommended for counseling who utilize
counseling services by 30%
Activity 4: Defining Program Goals and
Objectives
 Return to small groups from previous activity
 Develop one goal and two objectives for a
program that will address problems identified
in Handout 2 (GBV Case Study)
 After 15 minutes, a member of each group
will share the program goal and objectives
Developing Program Strategies
Role of a Conceptual
Framework
Conceptual Frameworks
Diagram that identifies and illustrates the relationships between all relevant
systemic, organizational, individual, or other salient factors that may influence
program/project operation and the successful achievement of program or project
goals.
M&E purpose:





To show where program fits into wider context
To clarify assumptions about causal relationships
To show how program components will operate to influence outcomes
To guide identification of indicators
To guide impact analysis (causal pathways)
Illustrative Conceptual Model for
Intimate Partner Violence (IPV)
Society
Community
Relationship
Individual perpetrator
Society
• norms
• laws
• notions of
masculinity
• gender roles
• poverty
• unemployment
• family isolation
• community
acceptance of
violence
• marital conflict
• male control of
wealth
• male control of
decision making
in family
Source: Heise 1998
• being male
• witnessing
marital violence
as child
• being abused as
child
• absent or
rejecting father
• alcohol use
Activity 5: Developing Program
Activities
 Return to small groups from previous activity
 Develop 3 program activities for the goal and
objectives that the group had formulated
earlier to address the problems identified in
the GBV case study
 In 15 minutes, a member of each group will
share the program strategies and activities
Logic Models
What is a Logic Model?
Logic models are diagrams connecting
program inputs to processes, outputs,
outcome, and impact as they relate to a
specific problem or situation. Logic models
show what resources your program will need
to accomplish its goals, what your program will
do, and what it hopes to achieve, emphasizing
the links between all these things.
Logic Model Components
 Input – Resources used in a program (e.g., money, staff,
curricula, and materials)
 Process – Activities conducted by program to accomplish its
objectives (e.g., outreach, advocacy, training, BCC, etc.)
 Output – Direct results of program activities (e.g., people
trained, materials distributed, couples counseled, etc.)
 Outcome – Program results at population level (e.g.,
knowledge, attitudes, behavior, coverage)
 Impact – Long-term change in morbidity, mortality, and fertility
Activity 6: Identifying Logic Model
Components
 Distribute Handout 5
 Each of the scenarios in the handout
corresponds to one of the five components in the
logic model: Input, process, output, outcome, and
impact
 As a group, decide which component the
scenario illustrates
Handout 6: Illustrative Logic Model for
Provider Training Program
INPUT
PROCESS
OUTPUT
OUTCOME
IMPACT
Human and
financial
resources to
develop
training
materials &
implement
training
program
 Develop
GBV clinical
training
curriculum
 Providers
trained in
GBV
 Increased
awareness of
GBV as a health
issue
Improved
health and
safety of GBV
victims
 Conduct
training-oftrainers
workshops
 Conduct
GBV training
for providers
 Improved
provider
attitudes
toward GBV
 Increased
provider
ability to
identify,
counsel, care
for, and refer
GBV victims
 Increased
disclosure of GBV
 Increased
knowledge and
utilization of GBV
services
Role of Logic Model
Program planning
 Helps one think through program strategy – where
you are and where you want to be
Program management
 Helps one track and monitor operations to better
manage results
Communication
 Shows stakeholders at a glance what the program
is doing and what it expects to achieve
Role of Logic Model
Consensus building
 Promotes common understanding about what program
is, how it works, and what it is trying to achieve
Fundraising
 Can structure and streamline grant writing
Monitoring and evaluation
 Provides thorough understanding of what resources a
program has to work with, what it is doing, and what it
hopes to achieve.
Activity 7: Developing a Logic Model
 Return to small groups from previous
activities
 Develop a logic model for two of the
activities of your program
 After 20 minutes, a member of each group
will share the logic model
Results Frameworks
Results Framework
Diagram that identifies steps or levels of results and illustrates the
causal relationships linking all levels of a program’s objectives
Purposes
 Provides a clarified focus on the causal relationships that connect
incremental achievement of results to the comprehensive program
impact
 Clarifies project/program mechanics and factors’ relationships that
suggest ways and means of objectively measuring the achievement of
desired ends
Other terms used
 Strategic framework
Illustrative Results Frameworks
SO1: Increased Utilization of Reproductive Health Services
IR1 Strengthened sustainability
of GBV programs
IR1.1 Improved policy
environment for GBV prevention
and response
IR2 Expansion of high-quality health
services for GBV victims in the public and
private sectors
IR2.1 Increased availability of
GBV screening and referral
IR2.2 Improved provider attitude
towards GBV
IR1.2 Strengthened NGO
advocacy for GBV prevention
IR2.3 Improved provider competence in GBV
screening, referral, care, & counseling
Activity 8: Develop Monitoring and
Evaluation Questions
 Return to your previous group activities where
you developed your program goals and
objectives and logic model
 Come up with at least 1 monitoring question and
1 evaluation question
 After 10 minutes, a member of the group will
share the M&E questions
Selecting Indicators
What is an Indicator?
An indicator is a specific, observable, and
measurable characteristic that can be used to
show changes or progress a program is making
toward achieving a specific outcome.
Characteristics of Good Indicators
 Valid: accurate measure of a behavior, practice or task
 Reliable: consistently measurable in the same way by different
observers
 Precise: operationally defined in clear terms
 Measurable: quantifiable using available tools
and methods
 Timely: provides a measurement at time intervals relevant and
appropriate in terms of program goals and activities
 Programmatically important: linked to the program or to
achieving the program objectives that are needed for public
health impact
Characteristics of Good Indicators (cont.)
Indicators do not specify a particular level of
achievement — the words “improved,”
“increased,” or “decreased” do not belong in
an indicator.
Common Indicator Metrics
Counts
 no. of legal and service organizations for violence against women
and girls (VAW/G) survivors
 no. women and children using VAW/G social welfare services
Calculations: percentages, rates, ratios
 % of health units with at least one service provider trained to care for
and refer survivors
 % of women aged 15-49 who have ever experienced physical
violence from an intimate partner
Index, composite measures
 gender-equitable men (GEM) scale
 sexual relationship power scale
Thresholds
presence, absence; pre-determined level or standard
Common Challenges in Indicator
Selection
 Choosing an indicator that program activities
cannot affect
 Choosing an indicator that is too vague
 Indicators that do not currently exist and cannot
realistically be collected
 Selecting an indicator that does not accurately
represent the desired outcome
 Too many indicators
How Many Indicators Are Enough?
 At least one or two indicators per key activity or result
(ideally, from different data sources)
 At least one indicator for every core activity (e.g., training,
BCC)
 No more than 8-10 indicators per area of significant
program focus
 Use a mix of data collection strategies/source
Factors to Consider When Selecting
Indicators
 Logic/link to framework
 Programmatic needs/information for
decision making
 Data availability
 Resources
 External requirements (government, donor,
headquarters)
 Standardized indicators (if available)
Activity 9: Selecting Indicators
 Return to small groups from previous activities
where you developed goals, objectives, activities,
and M&E questions
 Select 3 indicators that your group might use to
measure progress towards your program goals and
objectives
 Into which logic model component does each
indicator fall?
 After 15 minutes, a member of each group will
share the indicators
Activity 10: Selecting Indicators Cont’d
As a class, look at the indicators that the groups selected in
the previous activity
Discuss the following:
 To what extent are the indicators logically linked to the group’s
M&E framework?
 Do the proposed indicators share the characteristics of good
indicators?
 How will the indicators be used for program decision-making and
for what decisions?
 Are data available to measure the indicators?
 Can the indicators be realistically collected given available
resources?
 Are there government or donor requirements for measuring the
proposed indicators?
Where to Find Standardized Indicators
for GBV Programs
Bloom, Shelah S. Violence Against Women and Girls: A Compendium of
Monitoring and Evaluation Indicators (MS-08-30). Chapel Hill, NC:
MEASURE Evaluation, Carolina Population Center, University of North
Carolina and Chapel Hill; 2008.
http://www.cpc.unc.edu/measure/publications
United Nations Division for the Advancement of Women, United Nations
Economic Commission for Europe, United Nations Statistical Division.
Indicators to Measure Violence against Women. Report of the Expert Group
Meeting, 8 to 10 October 2007. Geneva: United Nations Human Rights
Council; 2008.
Report of the special rapporteur on violence against women, its causes and
consequences, by Yakin Ertürk. Indicators on violence against women and
state response. Advance edited version 28 January 2008.
Illustrative Indicators: Community
Mobilization
 Proportion of people who would assist a woman
being beaten by her husband or partner
 Proportion of people who say that wife beating is an
acceptable way for husbands to discipline their wives
 Proportion of people who agree that rape can take
place between a man and woman who are married
 Proportion of youth-serving organizations that include
training for beneficiaries on sexual and physical
VAW/G
Illustrative Indicators: Behavior
Change Communication
 Proportion of people who have been exposed to
VAW/G prevention messages
 Proportion of girls who say that they would be willing
to report any experience of unwanted sexual activity
 Proportion of girls that feel able to say no to sexual
activity
 Proportion of individuals who know any of the legal
sanctions for VAW/G
Illustrative Indicators: Service Delivery
 Proportion of health units that have documented and
adopted a protocol for the clinical management of
VAW/G survivors
 Proportion of health units with at least one service
provider trained to care for and refer VAW/G
survivors
 Proportion of women who were asked about physical
and sexual violence during a visit to the health unit
 Proportion of rape survivors who received
comprehensive care
Illustrative Indicators: Law and Public
Policy
 Proportion of law enforcement units following a
nationally established protocol for VAW/G
complaints
 No. of law enforcement professionals trained to
respond to incidents of VAW/G according to an
established protocol
 No. of VAW/G complaints reported to the police
 Proportion of VAW/G cases that were
investigated by the police
Measuring Indicators
Indicator 1: No. of service providers trained (in the past year
or other period) to identify, refer, and care for VAW/G survivors
 “providers” include any clinician providing direct clinical
services to clients at public health facilities
 “trained” refers to attending every day of a five-day training
course on VAW/G
Measuring Indicators
Indicator 2: % of women who were asked about physical
and sexual violence during a visit to a health unit
Numerator: Number of women who were asked, during the
course of their service provision at the health unit, about
any violence that had ever occurred, either physical or
sexual, in the geographic area of study (nation, province,
state, community)
Denominator:
If record review, total number of women’s records that
were reviewed at the health unit
If exit interview, total number of women interviewed
Activity 11: Operational Definition of
Indicators
Distribute Handout 8
Organize into 4 groups
Write an operational definition for one of the following
indicators:
 intimate partner physical violence
 appropriate care for rape survivors
 support-seeking behavior of GBV victims
 gender-norm related attitudes that reinforce VAW/G
After 15 minutes, a member of each group will share the
operational definitions
Measuring Composite Indicators
 Example: How to measure “attitudes toward GBV”?
 What question do you ask?
 How do you know any one question is a good representation of
general attitude?
 How do you scale each question (agree/disagree; strongly disagree to
strongly agree; yes/no)?
 How do you combine answers to the questions to come up with one
indicator?
 What value or weight do you assign to any one question on GBV
attitude in the survey?
Example of Measuring Attitudes
Toward Wife Beating
Sometimes a man is annoyed or angered by things that his
wife does. In your opinion, is a man justified in hitting or
beating his wife in the following situations:
 she goes out without telling him
 she neglects the children
 she argues with him
 she refuses to have sex with him
 she burns the food
Activity 12: Specifying Indicator
Metrics
 Return to small groups from previous
activities
 Look at the indicators that your group has
selected to measure progress towards your
goals and objectives
 Define the metrics for each indicator
Indicator Reference Sheets
Distribute Handout 9
Setting Indicator Targets: Useful
Information Sources
 past trends
 client expectations
 donor expectations
 expert opinion
 research findings
 what has been accomplished elsewhere
Activity 13: Setting Indicator Targets
Based on Past Trends
One of the objectives of the Ministry of Women’s Affairs in
Country X is to reduce the prevalence of intimate partner
violence. The 2000 DHS showed that 30% of evermarried women aged 15-49 years in the country had ever
experienced intimate partner emotional, sexual, or
physical violence. A comparable DHS conducted in 20052006 showed that the prevalence of intimate partner
violence (emotional, physical, or sexual) was 25% among
ever-married women aged 15-49. Based on these trends,
what would be a realistic national target for this indicator in
2010?
Types of Information Sources
Quantitative data
 useful for tracking trends accurately and
highlighting differences
Qualitative data
 useful for understanding the context in which the
trends/differences occur and to interpret
quantitative data accurately (e.g., focus groups, indepth interviews)
Quantitative Information Sources
Population-based surveys
 DHS (domestic violence module), VAW/G survey,
facility surveys
 Client exit interviews; provider surveys; clinic
observations
Health service statistics
Program statistics
Crime statistics (of poor quality in many LDCs)
Behavioral surveillance systems
Triangulating Data Sources: Soul City
Community
6. Cost – Effectiveness Study
Individual
1. National Survey
2. Sentinel Site Studies
3. (National) Qualitative
Impact Assessment
4. NNVAW Partnership
Study
5. Media Monitoring
and Analysis
Society
Activity 14: Identifying Information
Sources for Indicators
 Return to small groups from previous activities
 Look at the indicators that your group has selected to
measure progress towards your program goals and
objectives
 For each indicator, identify the potential source of
data needed for its calculation
 What challenges might you face in using these
potential information sources?
 After 10 minutes, a member of the group will present
Guiding Questions for Choosing an
Evaluation Design
 What question does your program need to answer?
 What do you want to measure (indicators)?
 How sure to you want to be? (VERY SURE = GOOD )
 What is the cost of making a mistake (low,
medium, high)?
 When do you need the results? (FAST)
 How much are you willing to pay? (CHEAP)
 Has the program already started?
Choosing an Evaluation Design
GOOD, FAST, CHEAP:
PICK 2!
HIRE SOMEONE TO HELP YOU
OUT!
NO EVALUATION DESIGN IS
IMMUNE TO PROBLEMS!
Impact Evaluation Needs Specific Design
Evaluating program
impact; interested in
attribution/causality
Experimental (with
randomization) or quasiexperimental design
Program already
implemented; no
baseline
Post-test only design (then
multivariate analysis)
Program not yet
implemented
Pretest-post-test design;
time series design
When monitoring and evaluating
GBV programs, the
confidentiality, privacy, safety,
and well-being of GBV survivors
must be top priority at all times.
References
 Bloom, SS. Violence Against Women and Girls: A Compendium
of Monitoring and Evaluation Indicators (MS-08-30). Chapel Hill,
NC: MEASURE Evaluation, Carolina Population Center, the
University of North Carolina and Chapel Hill; 2008.
 Bott S, Guedes A, Claramunt MC, Guezmes A. Improving the
Health Sector Response to Gender-Based Violence: A
Resource Manual for Health Care Professionals in Developing
Countries. New York: International Planned Parenthood
Federation, Western Hemisphere Region; 2004.
 Interagency Gender Working Group. Addressing Gender-Based
Violence from the Reproductive Health/HIV Sector: A Literature
Review. Washington: The Population Technical Assistance
Project (POPTECH); 2004.
References (continued)
 Soul City 4. Impact Evaluation; Violence Against Women, Vols.
I and II. 2001.
 World Health Organization (WHO). WHO Ethical and Safety
Recommendations for Researching, Documenting and
Monitoring Sexual Violence in Emergencies. Geneva: World
Health Organization; 2007.
http://www.who.int/hac/network/interagency/news/ethical_and_safety_recommendat
ions/en/index.html