Transcript impact evaluation of performance based contracting for

IMPACT EVALUATION OF
PERFORMANCE BASED
CONTRACTING FOR
GENERAL HEALTH AND
HIV/AIDS SERVICES IN RWANDA
A collaboration between the Ministry of Health, CNLS, SPH,
INSP-Mexico and World Bank
1
Presentation plan
2
1.
2.
3.
4.
5.
Country Profile
Impact evaluation design
Results from the general health baseline and
Follow up
Results from HIV baseline and Follow up
studies
Discussion
4/1/2016
2
SECTION 2 :
Impact Evaluation Design
7
Study Rationale
8



No examples of rigorously evaluated bonus
payment schemes to public sector health care
providers in developing countries
No distinction between the incentive effect and the
effect of an increase in resources for the health
facilities
Link between worker motivation programs and
quality of care
4/1/2016
8
TRADITIONAL FINANCING VS
PBF

Traditional Way of Health financing in Rwanda
was based on 3 sources of finance:



GoR / Founders: Investment cost (Infrastructures &
Equipment)
GoR / Private / Population: Recurrent cost (salaries, drugs,
materials, trainings,..)
PBF: funds received for remuneration can be
derived for enhancing human capacity building,
HF materials, or any new strategy aiming at
increasing quality.
Hypotheses
10
For both general health services and HIV/AIDS
services, we will test whether PBC:
 Increases the quantity of contracted health services delivered
 Improves the quality of contracted health services provided
 Does not decrease the quantity or quality of non-contracted
services provided,
 Decreases average household out-of-pocket expenditures per
service delivered
 Improves the health status of the population
4/1/2016
10
Evaluation Design
11



Make use of expansion of PBC schemes over time
The rollout takes place at the District level
Treatment and control facilities were allocated as follows:
 Identify districts without PBC in health centers in 2005
 Group the districts in “similar sets” based on
characteristics:
 rainfall
 population density
 livelihoods
 Flip a coin to assign districts within each “similar” to
treatment and control groups.
4/1/2016
11
12
Roll-out plan
13

Phase 0 districts (white) are those districts in which PBF was
piloted



Cyangugu = Nyamasheke + Rusizi districts
Butare = Huye + Gisagara districts
BTC = Rulindo + Muhanga + Ruhango + Bugesera + Kigali ville

Phase 1 districts (yellow) are districts in which PBF is being
implemented in 2006, following the ‘roll-out plan’

Phase 2 districts (green) are districts in which PBF is not yet
phased in; these are the so-called ‘Phase 2’ or ‘control districts’
following the roll-out plan. According to plan, PBF will be
introduced in these districts by 2008.
4/1/2016
13
Baseline Health Facility Sample
Phase I
(Intervention district)
District
Total Facilities
Phase II
(Control districts)
District
Total Facilities
Kibungo
Nyanza
Gakenke
Rwamagana
Gatsibo
Kayonza
Nyamasheke
Ngororero
Rutsiro
Nyaruguru
Burera
Total
Kirehe
Kamonyi
10
9
Nyagatare
13
Karongi
17
Nyabihu
Nyamagabe
Musanze
Total
10
13
11
83
11
2
11
3
8
11
2
6
10
9
10
83
TOTAL 166 HEALTH FACILITY
Program Implementation Timeline
16
Jan-06
Program
Implementation
Mar 06
Jun-06 – Sept 06
Start of intervention
2007
Feb-08-Apr 08
Phase I
Start of
intervention
Phase II
Impact
Evaluation
Apr-08
SURVEYS
Baseline
(General Health)
Baseline
(HIV/AIDS)
Follow-up
4/1/2016
16
More money vs. More incentives


Incentive based payments increase the total amount
of money available for health center, which can also
affect services
Phase II area receive equivalent amounts of
transfers




average of what Phase I receives
Not linked to production of services
Money to be allocated by the health center
Preliminary finding: most of it goes to salaries
18
The baseline has 4 surveys

General Health facility survey (166 centers)

General Health household survey (2,016 HH)


HIV/AIDS facility survey (64 centers)
HIV/AIDS household survey (1994 HH)
19
Baseline Field Sampling: GH HH

Randomly selected FOUR CELLS per HF


Randomly selected THREE ZONES per cell


Done by GH HH team leader in field
Obtained household lists for each of the zones
and randomly selected ONE HOUSEHOLD per
zone


Done in SPH using Epi Info and given to team
leaders
Done by GH HH team leader in field with zone/village
leader
Random sample of 12-13 households with
children < 6 years old per HF

2,159 households.
Baseline Field Sampling: GH HH
HEALTH FACILITY
CELL 1
HH 1
HH 2
CELL 2
HH 3
HH 4
HH 5
CELL 3
HH 6
HH 7
HH 8
CELL 4
HH 9
HH 10
HH 11
HH 12
Validity of Sample

Require two different validations:

Validate the sampling for the evaluation design


Diff in means tests between Phase I and Phase II to
determine if intervention and comparison groups
balanced at baseline
Validate the quality of data

Compare descriptive stats to other sources of national
data (i.e.: 2005 & 2007 DHS, MOH data)
Analysis Plan
23
All analyses will be clustered at the district level
 Compare the average outcomes of facilities and
individuals in the treatment group to those in the
control group 24 months after the intervention began.
 Use of multivariate regression (or non-parametric
matching) : control confounding factors
 Test for differential individual impacts by:


Gender, poverty level
Parental background (If infant : maternal education, HH
wealth)
4/1/2016
23
Difference in differences models
24



To test the robustness of the analysis
Control sample (both observed and unobserved) heterogeneity
A two-way fixed effect linear regression:
y it   PBC it   Bk X itk   i   t   it
k
Where:
y it is an outcome variable for facilitiy i in period t
or for individual i who lives in the catchment area of facility i,
PBC it is an indicator of whether facility i is being paid by PBC in period t,
X itk are time varying control variables,
i is a facility fixed effect,
 t is a year fixed effect,
 it is an error term.
4/1/2016
24
SECTION 3 :
Descriptive Results from
General Health Baseline
25
GENERAL HEALTH FACILITIES
BASELINE SURVEY
26
General Health Centers Survey:
Content
27







All 166 centers in General Health
General characteristics
Human resources module: Skills, experience and
motivations of the staff
Services and pricing
Equipment and resources
Vignettes: Pre-natal care, child care, adult care VCT,
PMTCT, AIDS detection services
Exit interviews: Pre-natal care, child care, adult care,
VCT, PMTCT
Classification of facility
28
Faith Based facility
Public facility
34%
66%
Baseline Health Facility:
Main sources of revenue
60.00%
50.00%
40.00%
Primary Source
Secondary Source
30.00%
Tertiary Source
20.00%
10.00%
th
er
O
at
e
Pr
iv
el
ig
io
us
R
t
ov
er
nm
en
G
on
or
th
er
D
O
Fe
es
se
r
U
D
ru
g
Sa
le
s
0.00%
30
Baseline Health Facility:
Human Resources
On average:
1 doctor for
every 31,190
individuals,
1 nurse for
every 4,835
individuals
Obs
Nurse A1
165
Nurse A2
166
Nurse A3
164
Auxilliary Staff >1yr
166
Auxilliary Staff <1 yr
161
Laboratory Technician 165
Full-time Support Staff 166
Mean Std. Dev
0.18 0.45
6.31 8.32
0.29 0.65
1.14 1.61
0.78 1.27
0.89 1.65
3.87 3.07
Min
0
0
0
0
0
0
0
Max
3
70
4
12
6
20
28
Baseline Health Facility: Services
Summary of Services Provided
Category
Im m unizations
Curative Care
Delivery Services
Fam ily Planning
TB
% Facilities
Service
Providing Service
BCG
98.81%
DPT
98.81%
Polio
98.81%
Measles
98.81%
Tetanus
98.21%
Children <5 yrs
100%
Adults
100%
Child Grow th Monitoring
86.31%
Nutritional Information
92.81%
Prenatal Care
98.21%
Normal Delivery
89.22%
Ceasarian Delivery
1.95%
Assisted Delivery
9.03%
Home Delivery
14.19%
Blood Transfusion
2.58%
Pill
84.05%
Injection
82.82%
Implant
18.47%
Male condoms
71.78%
IUD insertion
12.74%
Female sterilization
0.64%
Male sterilization
0.64%
Diagnosis
36.36%
Treatment
49.70%
Malaria Treatment
97.56%
Avg. Years
Providing
Service
16.78
11.91
16.51
16.4
16.48
20
19.76
18.03
16.83
15.51
18.6
30.75
17
11
36.5
10.96
10.72
12.2
9.05
17.17
.
.
10.17
10.51
17.01
Avg. Charge for Service
(excluding registration fee)
RWF 158.99
RWF 164.48
RWF 16.46
RWF 35.31
RWF 717.76
RWF 892.86
RWF 513.64
RWF 41.67
RWF 10.23
RWF 12.59
RWF 85.29
RWF 2.78
RWF 13.04
RWF 59.09
RWF 6.63
RWF 332.91
Baseline Health Facility: Lab Tests
Lab Tests
Phase I (Intervention)
Phase II (Control)
Num ber
Num ber
of Obs.
cstd of Obs.
cstd T-Stat
Anemia
80
37.50% 0.487
84
29.76% 0.460 -1.046
Pregnancy
80
56.25% 0.499
84
34.52% 0.478 -2.846
Blood Count
80
1.25% 0.112
84
3.57% 0.187 0.960
Urine dip
80
50.00% 0.503
84
60.71% 0.491 1.380
Stool Sample
80
8.75% 0.284
84
3.57% 0.187 -1.385
Parasites
77
89.61% 0.307
84
94.05% 0.238 1.029
Malaria
80
50.00% 0.503
84
53.57% 0.502 0.455
Malaria dip
79
55.70% 0.500
84
72.62% 0.449 2.277
TB
80
36.25% 0.484
84
40.48% 0.494 0.553
HIV
80
62.50% 0.487
84
55.95% 0.499 -0.849
Gonorrhea
80
18.75% 0.393
84
21.43% 0.413 0.425
Syphilis
80
10.00% 0.302
84
8.33% 0.278 -0.368
Pap smear
80
38.75% 0.490
84
36.90% 0.485 -0.242
Baseline Health Facility:
Overall Patient Satisfaction
Satisfied
76.44%
Very Satisfied
Not Satisfied
Very Unsatisfied
14.66%
4.83%
4.08%
0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00%
Baseline Household Data: Education
37
Sample: ages 6 and up
Variable
PHASE I (Intervention)
PHASE II (Control)
Nr. Obs Mean Std. Error Nr. Obs Mean Std. Error T-stat
Ever attended
3104 79.68%
0.011
4042 80.88%
0.010
-0.825
No schooling
2458 10.26%
0.007
3252 10.88%
0.007
-0.606
At least some Primary
2458 82.93%
0.010
3252 79.97%
0.010
2.195 **
At least some Secondary
2458
6.46%
0.008
3252
8.42%
0.007
-1.841 *
Attended school in last12 months
Able to read Kinyarwanda
2391 41.13%
2532 63.59%
0.015
0.015
3181 41.34%
3359 66.37%
0.012
0.014
-0.112
-1.375
Baseline Household Data: Assets
Variable
PHASE I
PHASE II
Nr. Obs Mean Std. Error Nr. Obs Mean Std. Error T-stat
Complete sofa set
921
2.18%
0.005
1147
3.08%
0.007
-1.073
Radio
921
49.63%
0.023
1147 50.27%
0.017
-0.221
Radio-cassette or music system
921
4.29%
0.009
1147
5.89%
0.008
-1.274
Telephone Mobile
921
1.07%
0.005
1147
3.50%
0.007
-2.778 ***
Mosquito nets
921
22.53%
0.023
1147 28.00%
0.027
-1.542
Sewing machine
921
0.76%
0.003
1147
1.18%
0.004
-0.883
A bed
921
62.82%
0.023
1147 58.71%
0.022
1.290
Wardrobe
921
4.53%
0.009
1147
6.07%
0.010
-1.155
Metalic library
921
0.57%
0.002
1147
1.28%
0.004
-1.645
Table
921
63.47%
0.022
1147 63.40%
0.021
0.021
Chair
921
921
85.48%
16.37%
0.018
0.020
1147 84.82%
1147 17.26%
0.019
0.027
0.255
-0.262
A bicycle
Baseline Household Data: Prenatal Care
40
Phase I
Variable
Nr.Obs.
Mean
Phase II
Std.Error Nr.Obs
Mean
Std.Error T -stat
Of all births since Jan. 2005
In facility birth
Of most recent pregnancy
Times received PNC
Injection to prevent tetanus
1238 32.05%
0.026
1462 34.87%
0.025
-0.777
723
2.781
0.046
900
2.687
0.052
1.368
728
74.52%
0.023
900
72.56%
0.019
0.654
42
Baseline Household Data:
Child Immunization
Phase I
Variable
12-23 months old
fully_immunized
12-71 months old
fully_immunized
Phase II
Nr.Obs.
Mean
Std.Error
Nr.Obs
Mean
Std.Error
T -stat
262
75.36%
0.03
295
77.04%
0.04
-0.33
1154
63.77%
0.03
1387
65.28%
0.02
-0.43
Baseline Household Data:
Child Recent Illness and Symptoms (<6 years)
35.00%
30.79%
30.00%
25.00%
20.65%
20.00%
16.37%
15.85%
15.00%
11.71%
10.00%
7.16%
5.00%
2.77%
2.20%
Ear Infection
Accident
0.00%
Fever
Respiratory
Diarrhea
Headache
Malaria
Eye Infection
Baseline Household Data:
Child Biomarkers (<6 years)
27.82%
30.00%
25.00%
22.16%
18.67%
20.00%
19.53%
Phase I
15.00%
Phase II
10.00%
7.51%
7.33%
5.00%
0.00%
Anemic
Malaria
Fever and Malaria
Validity of Sample and Data
46

Evaluation design

Of 110 key characteristics and output variables of HF,
the sample is balanced on 104 of the indicators.

Of 80 key HH output variables, the sample is balanced
on 73 of the variables.

Quality of data

HH Results comparable to the 2005 DHS, MOH data
Follow-up Field Sampling: GH HH

Result:





In total 2159 were suppose to be surveyed in the
catchment area of 167 facilities.
1888 (87%) were interviewed in 2006 and 2008
267 (12%) were replaced
4 (0.2%) were not found and not replaced
% of replacement by region:

South (24%), North (14%), East (4%) and West
(13 %)
HIV HEALTH FACILITIES
BASELINE SURVEY
53
Baseline Health Facility Sample
Assessm ent of the quality of ARV and HIV/AIDS service delivery:
Health Facility Sam ple
Phase I Phase II MAP TOTAL SAMPLE
Facility Level Data
HIV/AIDS Services and Human Resources
18
18
28
64
Provider Level Data
General Providers
18
18
28
64
PMTCT Providers
17
17
27
61
VCT Providers
17
18
27
62
TOTAL
52
53
82
187
Patient Level Data
PMTCT Patients
108
108
107
323
VCT Patients
116
87
209
412
TOTAL
224
195
316
735
Baseline Household Sample

38 ARV facilities for household sampling



13 MAP; 10 Phase I; 15 Phase II ARV sites for HH
sampling
Original sample of 1,487 patients from health
facilities and associations
Final sample consists of 1,961 households and
7,494 individuals
Patient Treatment: Distribution
Number
%
No Treatment
Cotrimoxazole
ARV
196
439
852
13.18%
29.52%
57.30%
TOTAL
1487
100.00%
Household Level: Sample Size of Patients


We find there is an additional sample of
individuals in our household sample who selfreport HIV+ status
Based on this, final distribution of patients:
Patient Sample: Distribution
Number
%
Phase I
Phase II
MAP
701
694
381
39.47%
39.08%
21.45%
TOTAL
1776
100.00%
Baseline Health Facility:
Main sources of revenue
General Facility Inform ation: Finances
Phase I
Obs
Phase II
Std
Obs
Std
tstat
funding_centralgovt
17
11.76%
0.33
18
33.33%
0.49
1.53
ratio_centralgovt
16
6.25%
0.25
17
21.76%
0.35
1.45
funding_province
17
23.53%
0.44
17
5.88%
0.24
-1.46
ratio_province
17
10.67%
0.25
16
1.42%
0.06
-1.42
funding_healthdistri
17
0.00%
0.00
17
5.88%
0.24
1.00
ratio_healthdist
17
0.00%
0.00
16
1.67%
0.07
1.03
funding_admindistric
17
11.76%
0.33
17
0.00%
0.00
-1.46
ratio_admindist
17
2.15%
0.06
16
0.00%
0.00
-1.38
funding_userfees
17
82.35%
0.39
17
70.59%
0.47
-0.79
ratio_userfees
16
28.07%
0.37
16
27.11%
0.32
-0.08
funding_mutuelles_cu
17
52.94%
0.51
18
61.11%
0.50
0.48
funding_mutuelles_me
17
64.71%
0.49
18
61.11%
0.50
-0.21
ratio_mutuelles
17
50.48%
0.41
17
42.38%
0.40
-0.59
donsgov
17
4586525 3057811.00
16 1.55E+07 35600000.00 1.25
donsnongov_dum
6
100.00%
12
donsnongov
12
7974796 14100000.00 14
8591750 7649821.00
0.14
comite_gestion
18
33.33%
0.49
18
44.44%
0.51
0.67
comite_sante
18
55.56%
0.51
18
44.44%
0.51
-0.65
other_management
18
11.11%
0.32
18
11.11%
0.32
0.00
0.00
75.00%
0.45
-1.33
Baseline Health Facility: Services
HIV/AIDS Services Offered
% Offer Service
HIV counseling
96.83%
Rapid HIV test
91.8%
Elisa HIV test
7.32%
CD4 Count
12.73%
Viral Load
2.5%
Systematic Testing of Blood Donations
19.51%
VCT
88.14%
Distribute Condoms
93.22%
PMTCT
93.10%
Preventative Cotrimoxazole Treatment
100%
Treatment of OIs
91.8%
ARV
80.7%
TB Diagnosis
84.75%
TB Treatment
92.98%
Nutritional Support
80.85%
Baseline Health Facility:
Overall Patient Satisfaction
70.00%
60.00%
58.60%
54.88%
49.77% 49.77%
50.00%
48.86%
50.00%
43.72%
40.00%
Not satisfied
35.35%
Satisfied
Very Satisfied
30.00%
20.00%
10.00%
6.05%
0.47%
1.40%
1.14%
0.00%
Time Attended
Confidentiality
Staff Attitude
Cost medications
HIV Patients vs. Non-Patients:
Mortality in the Household
Death in the last 5 years
18.00%
16.84%
16.00%
14.00%
12.00%
10.00%
7.61%
8.00%
6.00%
4.00%
2.00%
0.00%
Patients
Non-Patients
Patients vs. Non-Patients:
Mortality in the Household in the last 5 years
100.00%
94.09%
90.00%
80.00%
71.88%
70.00%
60.00%
Patients
50.00%
Non-Patients
40.00%
35.08%
30.00%
15.79%
20.00%
10.00%
0.00%
Illness related Death
Sold Assets
HIV Patients vs. Non-Patients:
Health Care Utilization
Illness in last 6 months
80.00%
69.67%
70.00%
60.00%
50.00%
40.00%
30.00%
17.32%
20.00%
10.00%
0.00%
Patients
Non-Patients
Patients vs. Non-Patients:
Sexual Behavior
90.00%
76.74%
80.00%
70.00%
56.14%
60.00%
50.00%
39.63%
40.00%
Patients
37.42%
30.00%
20.00%
10.00%
0.00%
Sex in the last 12 months
Condom during last encounter
Non-Patients
Follow-up Field Sampling: HIV HH

Objective:



Return to the same households to create panel
data set (2006-2008)
Print baseline roster (names, codes) and keep
consistent across waves
Account for household members who left and new
arrivals from 2006-2008
Follow-up Field Sampling: HIV HH

Result:




In total 1996 were to be surveyed in the
catchment area of 38 ARV facilities.
1,716 (87.06%) were interviewed in 2006 and
2008
280 (12.94%) were not found and not replaced
% of missing by region:

South (12.6%), North (12.7%), East (7%) and
West (16.4%)