Transcript Oslo_Eval_WB

Evaluation at the World Bank
“...we will not measure our performance by dollars lent
or projects approved, but by our development impact -results on the ground. I cannot overstate the importance
of this change.”
James D. Wolfensohn
Annual Meeting Speech
October 1996
Oslo 10/14/99
1
The World Bank and Development
• A global development finance institution
– 179 country members
– $25-30 billion annual borrowings
• The largest single provider of development
assistance
–
–
–
–
over 1800 projects under implementation
(worth more than $140 billion)
250-300 new loans/credits a year
(worth $20-25 billion)*
* Excluding financial crisis loans (about $8–9 billion)
2
The World Bank and Development
• A far-flung development network
– 83 field offices
– 72 aid groups
– 800 trust funds
• A Bank of ideas and skills
– A significant research and knowledge management
budget ($99 million)
– $126 million worth of advisory and aid coordination
services
– $34 million of development training (EDI)
3
Evaluation in The World Bank
• OED is independent: it reports to the Board of
Executive Directors
• 5,000 projects have been evaluated since OED’s
creation in 1970
(325 operations, worth $26 billion,
in FY96 -FY97 alone)
• The function spans a variety of sectors
(agriculture, energy, finance, transport,
water supply, urban development)
• It evaluates projects, programs, processes, and
policies
4
The Development Environment
Has Evolved
Before
Now
Ascent of Global
Capital Market
Money Bank
Full Service Bank
Greater Pluralism
Planned
Development
Participatory
Development
Conditionality
Knowledge
Management
and Partnerships
Resource Transfer
Development
Effectiveness
End of
Ideological Divide
Advent of
Civil Society
5
The Very Concept of Development
Has Changed
Physical
Social
Human
Development
Decades
•
•
•
•
•
Fifties
Sixties
Seventies
Eighties
Nineties
Major
Objective
•
•
•
•
•
Reconstruction
Growth
Basic Needs
Adjustment
Capacity Building
Natural
Main
Instruments
Technical Assistance
Projects
Sector Investment
Adjustment Loans
Country Assistance
Strategy
Dominant
Discipline
Engineering
Finance
Planning
Neoclassical
Multi-disciplinary
6
The Meaning of Development
Effectiveness Has Changed Too
Development Effectiveness connotes a
demonstrable and efficient contribution to
economically sound, socially responsible and
environmentally sustainable growth
7
Development Effectiveness Means:
DOING THE RIGHT THINGS
DOING THINGS RIGHT
Relevance: Are the objectives
we seek responsive to country
needs and Bank priorities?
Efficacy: Are we achieving
objectives shared by the Bank
and its partners?
Institutional development: Are
we helping to build the capacity
of our partners?
Efficiency: Are we using Bank
and partners’ resources
economically?
Sustainability: Will the benefits of Bank
intervention be sustained?
8
World Bank Performance Criteria Reflect the
New Development Agenda:
Socially
and
Environmentally
Sustainable
Growth
Governance: social development, economic management, public sector reform, etc.
Poverty/gender
population,
health,
nutrition
Environment
Private sector
development
Rural
development
Finance
Social
protection
Infrastructure
Energy/mining
Human economy
Physical economy
Market economy
Education
9
Performance characteristics
• ECONOMIC
Macro policy
Sector policy
• FINANCIAL
Corporate finance
Public finance
Commercial aspects
• SOCIAL
Income distribution
Social impact
Resettlement
Indigenous groups
• INSTITUTIONAL
Organization and
management
Legal
Governance aspects
• ENVIRONMENTAL
Technical options
Engineering options
Choice of technology
Development themes
• ECONOMIC MGMT. • PSD/PSM
Adjustment
Debt
Trade policy
Fiscal policy
Public expenditures
Management
• POVERTY
REDUCTION AND
HUMAN RESOURCES
Education/training
Population
Health and nutrition
Gender aspects
Financial
intermediation
Enabling policies
Civil service reform
• ENVIRONMENTAL
Natural resource
management
Pollution
Infrastructure
Regional planning
As vehicles for policy and institutional reform, projects are evaluated in terms of the special
emphases of the development agenda.
10
The Evaluation System is Being Adjusted
Characteristics
of evaluation system
Dominant concepts
THEN
NOW
•
•
•
•
Project focus
Investment-driven growth
Import substitution
Central planning
•
•
•
•
Project evaluation
Cost/benefit analysis
Shadow pricing
Self-evaluation
•
•
•
•
Country focus
Structural Adjustment
Outward-oriented policies
Decentralized decision
making
•
•
•
•
Portfolio evaluation
Policy evaluation
Risk analysis
Participatory evaluation
Evaluation methodologies reflect the dominant paradigms of development policy.
11
Measuring Performance Matters
• For accountability: Increasingly, shareholders
want to know whether they are getting “value
for money.”
• For Bank management and staff: To enhance
professional excellence and get results on the
ground through organizational learning.
12
Project Performance
Measurement System
OED
consistent
• objective
• continuous
• professional
• external credibility
• work in process--e.g.,
NLS
•
Quality at exit
Quality at entry
QAG
Monitoring during
implementation
QAG
13
Real Time Evaluation Means Quality
Assurance
Two Complementary Components
• Portfolio Improvement Program
• Quality Assessment Program
– Quality at Entry
– Supervision
– Economic and Sector Work
14
It also Means Quality Management
• Target managerial attention on “high risk” clusters
• Regular monitoring and feedback to staff and
managers
• Partnering to disseminate lessons of experience
• Linking portfolio management indicators to
staff/managerial evaluations
15
The Other Main Challenge Is the
“Scaling-up” of Evaluation
• Country, sector, and thematic evaluations
• Adding up project-level results and linking them
to network and strategic goals.
• Non-lending services, partnerships, and strategic
alliances.
• Moving towards a results-oriented framework for
performance measurement.
16
Evaluation Gaps Must Be Filled
• Today’s evaluation system concentrates on lending
• Activity completion reporting is also needed for
research, grants, economic and sector work, trust
funds, training, etc…
• Evaluation must move to a “higher plane”
17
Country as Unit of Account
Country
Assistance
Agenda
Country
Assistance
Strategy
Performance
Factors
Development
Effectiveness
Results
Social
Bank
•
•
•
•
•
•
•
Thematic
Priorities
Country’s
Policies/Capacities
Assessment of
Past Projects
Development
Priorities
Development
Objectives
Partners’
Plans/Programs
Country Dialogue
ESW Quality
Quality at Entry
Supervision Quality
Portfolio Management
Cost Effectiveness
Partnerships
•
•
•
•
Poverty head count, change
Private consumption p.c. growth
Child mortality, change
School enrollment, by gender,
change
•
•
•
•
GDP p.c. growth
Export growth
Debt/GDP
Inflation
Economic
Country
Assistance
Strategy
Achievements
Borrower
•
•
•
•
•
Fiscal
Monetary
Trade
Size of Government
Quality of Public
Expenditure
• Commitment
Operational
Programs
Exogenous
•
•
•
•
Terms of Trade
Interest Rates
Droughts/Floods
War
Environmental
• Country-specific measures
(e.g. salinity, forest cover),
change
• Environmental Index (e.g.
enforcement of environmental
regulations), change
•
•
•
•
•
Relevance
Efficacy
Efficiency
Sustainability
Institutional
Development
Private Sector
Development
• Private Investment/GDP
• Foreign Investment/GDP
• Privatizations
Capacity Building
• Public Sector Management
• Governance
18
Sectors/Themes as Objects
of Evaluation
Thematic
and country
programs
Environ.
Private sector sustainable
development development
Economic/social Human/
governance social capital
Economic
mgmt.
PSM
Education
PHN
Soc.
policy/
social
security
3 year
sector/thematic
evaluation
PSD
Finance
Industry
Telecom.
Operating
services
programs
Urban dev.
Rural dev.
EnvironEnergy
ment
Water
Transport
Annual
business plans &
retrospectives
19
The Ultimate Objective Is Transparent Evaluation of
Development Effectiveness at Corporate Level…
ARDE
Regions
AFR
EAP
SAS
ECA
MNA
LAC
Country Assistance
Strategy and Compacts
Nonlending
Services
Lending Services
Supervision
Reports
Regions/
OPR
Mid-term
Reviews
Regions/
OPR
Performance
Portfolio
Imple.
Improvement Audit Reports
Completion
and Impact
Program
Reports
Evaluations
Reviews
Regions/
OPR
QAG
OED
Networks
HDD
ESSD
FPSI
PREM
Sector Assistance Strategies
and Thematic Progress Reports
Lending Services
Nonlending
Services
Sector
Impact
Eval.
Activity
Completion
Reports on
Nonlending
Services
Aid
Coordination
and
Cofinancing
Reviews
Networks/
OED
Regions/
QAG/OED
Regions/
OED
Research
Activity
Trust Funds
Completion Completion
Reports
Memos
DEC/
Networks
Regions/Networks
OED
Proposed
20
…Combined with Regular Monitoring of
Bank Performance (Scorecard)
Borrowing
spreads
Reserves/
loan ratio
Interest
coverage
ratio
Investment
return
Admin.
expenses as
% of assets
FPI
TRE, FPI
TRE, FPI
TRE, RM, FPI
PBD
Client feedback
on staff
performance
OED
Regions
Financial
strength
Processing
times
Quality at entry:
• 10 ECON commandments
OED/QAG
PBD
Disbursement
rates
Quality at exit:
• Outcomes
• Sustainability
• Institutional development
• Bank performance
Client
responsiveness
Score-card on
Bank
performance
Development
effectiveness
PBD
Quality of non lending serv.:
• Relevance
• Efficacy
• Efficiency
Values and
skills
Front-line services
as % of total
admin. cost
Portfolio management quality:
• Projects at risk
• Realism index
• Proactivity index
QAG
OED/QAG
PBD
Staff feedback
from surveys
ICD
Skills
assessment
Networks
Effectiveness of
staff training
LLC
Cost effec. indicators:
• Completion coefficients
• Cost/SY
PBD
21
This Vision Implies
Partnerships in Evaluation
Development effectiveness requires participatory
evaluation
– Locally: to support management conditions
– Globally: to support strategic alliances
Thus ensuring proactivity of the overall
development system
– Global networking among evaluators
– Evaluation capacity development
– beneficiary involvement in evaluation
22
NGOs
Strategic
alliances
Multilateral
Bilateral
Global
and
Regional
PERFORMANCE
PARTNERSHIPS
Civil
society
Management
coalitions
Country
Government
Donors
23
Operations Evaluation Department
DGO (171/99)
Director OED (OEDDR 175/05)
Operations Advisor
Resource Management
Manager
Country Evaluations
& Regional Relations
Manager
Sector & Thematic
Evaluations
(OEDCR - 175/20)
Manager
Partnership & Knowledge
Programs
(OEDST - 175/10)
Poverty, Gender, Social
Development & Social
Protection
Rural
Development
(OEDPK - 175/40)
Knowledge
Management
Finance and
PSD
ECDP, Partnerships &
Training
Manager
Corporate Evaluations
and Methods
PSM
Environment
and Water
Energy,Mining &
Telecommunication
Education
Country
Economics
HNP
Urban &
Infrastructure
Transport
Evaluation Process and Methods
(OEDCM - 175/50)
= Thematic & Regional Coordinators
Clusters are Interactive and Flexible
ACS Staff assigned to managerial group and cluster
24
Trends in OED Work Program
FY96, FY99 - FY00–02
Comparison of OED Outputs and Resource Utilization FY96-FY00
Outputs
Resources
($ million)
Activities
FY96
FY00
FY96
FY00
Country Evaluations
2
10
0.9
3.5
Sector & Thematic
4
7
2.2
4.1
Corporate & Process
5
4
2.5
1.7
Total Country Evaluations & Studies
11
21
5.6
9.3
Project Evaluations
372
358
8.1
4.4
Evaluation Development
2.4
4.8
Total (including Trust Funds)
16.1
18.5
25
Trends in OED Work Program
FY96, FY99 - FY00–02
Planned Country Evaluations, by Region FY00-02
FY00
FY01
FY02
Africa
Burkina Faso, Ghana,
Tanzania, Uganda
Cameroon, Madagascar,
Cðte D’Ivoire,
Eritrea, Senegal
East Asia
Papua New Guinea
Vietnam
China
South Asia
India
Eastern Europe and
Central Asia
Kyrgyz Republic
Bosnia-Herzegovina,
Kazakstan, Macedonia
Latvia, Romania,
Russia,
Latin America and the
Caribbean
Costa Rica, Paraguay
Argentina, Mexico, Peru
Chile, Dominican
Republic
Middle East and North
Africa
West Bank and Gaza
Morocco
Jordan
26
Trends in OED Work Program
FY96, FY99 - FY00–02
Major Evaluations FY00-02
FY00
FY01
I. Thematic and Sector Policy Studies
Poverty
Social Funds
Environment
Energy
Water
Transport Partnerships
Forestry
Telecommunications
Rural Development
Nutrition
Culture
Gender
Anti-Corruption
Urban
FY02
Education
Rural Poverty
Water and Sanitation
HNP
Social Development
Social Dimensions of Crisis
Lending
Transport
II. Corporate and Process Evaluations
AROE
AROE
ARDE
ARDE
AROE
ARDE
AID Coordination
Participation
New Lending Products
CDF Evaluation
Safeguard Policies
Guarantees
27
Trends in OED Work Program
FY96, FY99 - FY00–02
Sector Strategy Paper Schedule and OED Products
Network
OED Product
Sector Strategy
Paper
FY00
FY01
FY02
Rural Development
Phase II
Rural Development
Environment
ESD Strategy
Social Funds and
Service Delivery
ESD Strategy
X
Forestry Review
Forestry
X
Water Resources
Water Resources
X
Culture and
Sustainable
Development
Culture and Sustainable
Development
X
HNP
Health
X
Energy (CSAR)
Energy
X
Telecomm.
Telecommunications
X
Transport
Transport
X
Water and Sanitation
Water and Sanitation
X
Poverty
WDR 2000/2001
X
Gender
Gender Strategy
X
ESD
X
X
HD
FPSI
PREM
28