Transcript The Quality of Medical Advice: Vignettes (and more)

The Quality of Medical
Advice: Vignettes (and more)
Jishnu Das
World Bank
Saving Meena: Story
of a death foretold
From the local to the global
Poor Health
 Life Expectancies (Ethiopia 53; Kenya 54;
Zambia 42 in 2007)
 U5 Mortality (per 1000 born): Ethiopia 127 ;
Kenya 114; Zambia 174)
Mali 2001
Chad 2004
Mozambique 2003
Nigeria 2003
Zambia 2001/02
Lesotho 2004
Guinea 2005
Benin 2001
Rwanda 2005
Burkina Faso 2003
Kenya 2003
Ethiopia 2005
Congo (Brazzaville) 2005
Malawi 2004
Cameroon 2004
Bangladesh 2004
Tanzania 2004
Nepal 2001
Ghana 2003
Senegal 2005
Madagascar 2003/2004
India 2005
Eritrea 2002
Bolivia 2003
Indonesia 2002/2003
Morocco 2003-2004
Paraguay 1990
Egypt 2005
Dominican Republic 2002
Nicaragua 2001
Philippines 2003
Armenia 2005
Jordan 2002
Vietnam 2002
Moldova Republic of 2005
US 2000 (All)
US 2000 (AfricanAmericans)
Infant Mortality Rate per 1000 births
0
50
100
Many countries
Figure 2: Infant Mortality Rates in Selected Countries
IMR: Selected Countries
Data from DHS Surveys
Note: Data are based on the Demographic and Health Surveys (ORC Macro, 2007. Measure DHS STATcompiler) for all
countries except India, which is based on the National Family Health Survey (2005). Countries that are cross-shaded form
the basis of this paper. No data are available for Paraguay after 1990.
Why?
 Infectious diseases: HIV/AIDS, Malaria,
Diarrhea, TB

Kenya: TB incidence rate is 62/10,000 (one of the
highest in the world)
 Partly to do with low incomes

But see Riley on countries that improved
health outcomes at low-income levels
 Partly to do with access to care

Particularly a problem in rural areas
Some quotes
 There is an obvious difference between rural and urban
postings. Working in rural areas involves helping the poor… in
urban areas one can learn, have more income, have a good
school for one’s children.
 It is Siberia!.
Doctor in Ethiopia
Doctor in Addis Ababa
 There is no plan of development for a doctor in the rural
areas; it is as if you are lost. The lack of career development
means that it is as if you are punished.
Doctor in Rwanda
 Promotion is as important as remuneration because you
cannot stay in the same place forever.
Doctor in Ghana
Sourced from Serneels and others
And yet…
Figure 1: The Use of Health Facilities for ARI
Data from DHS Surveys
Madagascar 2003/2004
Philippines 2003
Mozambique 2003
Tanzania 2004
Lesotho 2004
Nicaragua 2001
Indonesia 2002/2003
Egypt 2005
India 2005
Moldova Republic of 2005
Dominican Republic 2002
Vietnam 2002
Zambia 2001/02
Jordan 2002
U.S. (1988)
Chad 2004
Ethiopia 2005
Bangladesh 2004
Nepal 2001
Rwanda 2005
Nigeria 2003
Armenia 2005
Malawi 2004
Mali 2001
Morocco 2003-2004
Burkina Faso 2003
Cameroon 2004
Benin 2001
Guinea 2005
Eritrea 2002
Ghana 2003
Senegal 2005
Congo (Brazzaville) 2005
Kenya 2003
Paraguay 1990
Bolivia 2003
Proportion Using a Health Facility
0
20
40
60
80
Use of Health Facilities for ARI
Note: The graph shows the % of respondents who said that they had taken their child to a health facility for the treatment of
an Acute Respiratory Infection. Data are based on the Demographic and Health Surveys (ORC Macro, 2007. Measure DHS
STATcompiler) for all countries except India, which is based on the National Family Health Survey (2005). Countries that
are cross-shaded form the basis of this paper. No data are available for Paraguay after 1990. The equivalent number for the
US is not easily available. We use instead health seeking behavior among children for pharyngitis as reported in Stoddard,
Jeffrey J., Robert F. St. Peter, and Paul W. Newacheck. 1994.
In many countries around the world, people use facilities a lot more than they do in the US
And yet…
How Many Medical Care Providers
0
10
20
30
Deep Rural Madhya Pradesh
19 24 21 36 6 14 26 29 1 12 25 28 30 4 17 32 34 11 15 7 10 13 27 20 23 31 33 38 8 18
This is ongoing work
In deep rural Madhya Pradesh, one of the states in India with the worst HD outcomes, there
are 11.3 medical care providers accessible for representative rural households who do not
live close to national highways or close to urban centers
And yet…
How Many Medical Care Providers
80
60
40
0
20
Households per Provider
100
Deep Rural Madhya Pradesh
28 33 4 38 13 8 18 26 31 6 27 15 11 10 23 17 20 36 29 1 25 12 32 7 30 34 19 21 24 14
This is ongoing work
This is not because these villages are huge: for these villages, there are around 20
households per provider
New Studies on Doctor Visits
 In 5 countries studied, the poor visit doctors
almost as much as the rich

Kenya: 60% of poorest quintile seek care
when sick relative to 78% of richest quintile
 In urban India, new survey methods show
that the poor visit doctors more than the rich

The difference arises because of shorter recall
periods
 In rural India, households visit doctors twice
as much as in the United States
Do the poor really go to doctors
less than the rich?
Experimental data from Delhi show that when households are asked on the basis of monthly
recall, self-reported doctor visits fall by 65 percent for the poor compared to weekly recall.
In fact, the poor go to doctors more than the rich. We don’t know what happens in other
countries, because weekly recall questionnaires are very rare!
One alternative
 Health outcomes may be related to the quality of
health care that people receive
Structural Quality
 Quality traditionally defined as structural quality—
state of infrastructure, availability of medicines.
This is clearly informative BUT



Its not correct when demand is a factor (medicines)
The quality of medical advice may be equally (or
even more) important
The relationship between quality of advice and
structural quality is weak (India, Indonesia,
Tanzania)
The Quality of Medical Advice:
New evidence
 Since 2002, team working on the quality of
medical advice at The World Bank and the
University of Maryland
 Basic Idea



What can be measured, and how?
How do these measurements help us
understand the quality of medical advice?
How can the quality of medical advice be
improved?
Where we are
 Quality of medical advice can be decomposed
into


Competence: What does a doctor or medical care
provider know about how to treat an illness
Practice: What does a doctor or medical care
provider do when faced with an illness
 We systematically find that the two are very
different. This has important policy implications


Improving competence is about training
Improving practice (given a competence) is about
getting doctors to exert greater effort
 Countries: Tanzania, Urban India, Indonesia,
Paraguay

Ongoing: Rwanda, Rural India, Argentina
Remainder of Presentation
 Basic facts about competence
 Basic facts about practice
 More interesting facts
 Lessons learnt (thus far!)
What are Vignettes
 Standardized mix of (in our case) 5 cases
 One interviewer is `patient’; other is `recorder and observer’
 Child with Diarrhea
“My child has been suffering from diarrhea for the last two
days, and I do not know what to do”
 History
 Doc gets 1 point if he asks about last urination (for
instance)
 Examination
 Doc gets 1 point if he asks about temperature
 (IF asked, recorded responds `98.8 F’)
 Treatment
 Doc prescribes treatment: treatment is graded by
independent raters in South Asia and the US
Scoring Vignettes
 The questions that doctors ask are compared to
a checklist of essential procedures
 An aggregate “index” is compiled that accounts
for the different “difficulties” of different checklist
items
 We call this index “competence”
 We normalize mean = 0 and standarddeviation=1

Moving from 0 to 2 moves you from 50th percentile to
95th percentile
Vignettes: Advantages
 Advantages
 Standardized case-load (same 5 cases to all docs)
 Standardized patient (specify that patient will
`comply fully with all medications and tests’
om
as
tC
Av
pe
2
te
er
nt
ag nd
Q
e
u
C
i
n
om
ti
pe l e
te
n
4
M th Q ce
os
t C ui n
om tile
pe
te
Le
nt
as
tC
om
p
Av
2n ete
er
nt
ag
d
Q
e
u
C
om inti
pe l e
t
4t enc
h
e
M
Q
os
t C ui n
t
om ile
pe
te
Le
nt
as
tC
om
Av
p
2n ete
er
nt
ag
d
Q
e
ui
C
om nti
pe l e
te
n
4
M th Q ce
os
u
in
tC
om tile
pe
te
Le
nt
as
tC
om
Av
p
2n ete
er
nt
ag
d
Q
e
ui
C
om nti
pe l e
te
n
4
M th Q ce
os
u
in
tC
om tile
pe
te
nt
Le
0
20
40
60
80
100
What they know
What doctors know
Delhi Doctors
Diarrhea
PreEclampsia
TB
ViralPharyngitis
Public or Private?
Distribution of Competence by Qua lification
Private--M BBS
.4
.2
.3
Dens ity
.3
.2
0
0
.1
.1
Density
.4
.5
.5
Public--All M BBS
-2
-1
0
C om petence
His togram
1
2
-2
-1
Kernel Dens ity
0
C om petence
His togram
Kernel Dens ity
.5
.1
.2
.3
.4
Dens ity /Percent
.4
.3
.2
0
0
.1
Density
2
All Providers
.5
Private--Non-M BBS
1
-2
-2
-1
0
C om petence
His togram
1
2
-1
0
C om petence
Public Providers
Kernel Dens ity
1
2
Private--MBBS
Private--Non-MBBS
An MBBS is the formal Indian medical degree—roughly the equivalent of an MD
in the US
Distribution of Competence
Competence Across Neighborhood Incomes
Overall Competence
Public PHCs
.5
0
-.5
Low
Middle
High
Regression Models
Table 5: What Explains Quality?
(1)
Income and
Institution
-0.013
(0.002)***
(2)
Income, Institution
and Qualification
-0.007
(0.002)***
(3)
All variables
excluding price
-0.009
(0.002)***
(4)
All variables
including price
-0.009
(0.002)***
% middle income
households in
community
-0.011
(0.003)***
-0.006
(0.003)*
-0.004
(0.003)
-0.003
(0.003)
Public Doctor
0.451
(0.172)***
-0.185
(0.175)
-0.174
(0.171)
0.365
(0.348)
1.132
(0.153)***
0.957
(0.156)***
0.848
(0.171)***
-0.020
(0.007)***
-0.019
(0.007)***
% of poor households
in community
MBBS Degree Holder
Tenure in Locality
Usual Price Charged
(from census)
0.144
(0.080)*
Controls for Origin of
Provider?
Constant
NO
NO
YES
YES
0.616
(0.154)***
-0.194
(0.175)
-0.399
(0.255)
-0.804
(0.347)**
Observations
204
204
194
187
R-squared
0.19
0.36
0.44
0.45
R-squared corrected for
measurement error
0.28
0.53
0.64
0.65
Results very similar in other
countries
•More competent doctors in
urban areas
•More competent doctors in
richer areas (within urban and
within rural areas)
•More training increases
competence
Competence across countries
Measuring practice
 Sit in the doctors
office
 Record details
about every
interaction
between the
doctor and patient




Photo Credit: Ken Leonard
Time
History taking
Physical exams
Drugs prescribed
Practice: Some numbers
Table 3: International Comparisons of Effort
Sample
Delhi
Effort Categories or Country
Time Spent
Questions
asked of Patient
% Who do
Physical
Exams
Poly-pharmacy
(Total number of
medicines given)
Doctors who exert low effort
1.9
1.36
14
2.13
Doctors who exert medium
effort
3.36
2.94
78
2.72
Doctors who exert high effort
6.15
5.32
98
3.05
All Doctors
3.80
3.20
63
2.63
5.79
5.33
1.38
1.36
7.90
7.50
2.93
1.55
11.34
11.91
3.64
1.65
8.33
8.23
2.65
1.52
Doctors who exert low effort
(25th Percentile)
3
2
0
N/A
All Doctors
6.32
3.96
1.51
N/A
Tanzaniac (1991)
3.0
N/A
N/A
2.2
Nigeriac
6.3
N/A
N/A
2.8
Malawic
2.3
N/A
N/A
1.8
UKd
9.4
N/A
N/A
N/A
Doctors who exert low effort
Paraguay
Doctors who exert medium
effort
Doctors who exert high effort
All Doctors
Tanzania
International
Comparisons
25 th
Notes: We divide doctors by terciles of effort in India and Paraguay, and the
percentile versus all doctors for Tanzania. The
data are based on the following sources
India: Das and Hammer (2007); Paraguay: Das and Sohnesen (2007); Tanzania: Leonard (Mimeo); International Comparisons:
Hogelzeir and others (1993) and Deveugele and others (2003).
Another look at practice
7
6
5
4
low effort
medium
high
3
2
1
0
time
Less than 2 minutes
questions
exams
Just one question
Almost none!
Public-Private Again
Got Time?
5.89
6
5.39
4.76
Lowest
Second
Median
Fourth
ic
bl
Pu
iv
at
e
2
Pr
at
e
iv
Pr
ic
bl
Pu
at
e
iv
Pr
ic
bl
Pu
at
e
iv
Pr
ic
bl
Pu
Pr
iv
at
e
1.56
3.26
3.1
ic
2.19
4
bl
2.28
Time in Minutes
4.06
Pu
4.22
Top
0
Public-Private Again
Table 4: Practice and Competence across Sectors
Quintiles of ability
Lowest
2nd Group
Median
4th Group
Highest
Type of provider
Private
Total history questions
2.93
Probability of. examination
(percent)
70
Time spent (minutes)
4.22
Fees charged (Rs.)
21.5
Public
1.72
28
1.56
0
Private
3.37
71
4.76
34.6
Public
1.88
41
2.28
0
Private
3.55
75
4.06
32.8
Public
2.17
42
2.19
0
Private
3.67
81
5.39
44.0
Public
3.55
41
3.10
0
Private
4.71
81
5.89
57.3
Public
4.04
72
3.26
.01
Note: This table disaggregates provider practice (observed in the clinic) by competence measured in the vignettes and sector (public/private). History questions refer to
the number of questions regarding the illness that the provider asked the patient. An examination consists of any physical contact between the provider and the patient or
the use of measuring instruments, such as a thermometer, sphygmomanometer, or stethoscope. Note that an examination only implies that the device was used, not that is
was used correctly. Fees charged refers to the total payment at the end of the interaction. Finally, public providers are those who were observed in their public practice
and need not be providers who work only in the public sector.
Percentage of Essential Tasks Completed
What they know, what they do
40% of
essential
questions
asked
Private MBBS
Public MBBS
Private, No MBBS
0
10
20
30
40
50
60
70
Lost Training: Tanzania
0
10
20
30
40
50
60
Competence (% of required items)
70
80
90
Individual clinician's competence and performance
Predicted quadratic relationship of competence to performance (Public)
Predicted quadratic relationship of competence to performance (Non-Public)
Performance = Competence
Sourced from Ken Leonard
Lost Training: India
1
Rotating The Curve
.6
.2
.4
Lost Training:
Private
Additional Lost Training:
Public
0
What they Do
.8
These doctors are
operating at the frontier
of their knowledge
0
.2
.4
.6
What they said they would do
What they know
What they do: Public
.8
W hat they Do: Private
1
Training or Effort?
 Because of lost training

We simulate that the impact of training is very
small relative to improvements in effort
The big caveat?
 Is it that public doctors put in a lot less effort
because they see many more patients?


We find more effort in India in hospitals, which
typically see many more patients
Two new and incredibly sad pieces of
research
Case loads and doctor shortage
 Maestad and others (2009)
went and sat in many doctors
clinics in rural Tanzania
 “The average doctor sees
18.5 patients per day and total
time use is 5.7 mins per
patient. The doctor completes
22% of essential tasks”
 That’s less than 2 hours a day
in an 8 hour day
 There is no relationship
between caseload and doctor
effort!
Effort jumps when
doctors are
observed
35
40
45
50
55
60
65
70
75
The Hawthorne Effect
-10
-5
0
5
10
Number of Previous Consultations Under Observation
doctor observed from t = 1
15
doctor never observed
Doctors significantly increase their effort in Tanzania when they know they are
being observed with no detrimental effect on patient (Leonard, 2007) Based on Ken
Leonard (2007) in Journal of Health Economics
Improving health outcomes
 New research suggests that improving effort
could also improve outcomes


Bjorkman and Svensson, 2008: Uganda
Rwanda Pay-for-Performance experiment:
Ongoing, Rwanda
Summary
 The quality of medical advice is key and
understanding the levels and correlates of quality
is an urgent priority
 Measuring either competence or practice is a
good start

But measuring both vastly increases our
understanding of what is going on and where the
policy levers may be
 Resources exist to help do this
 A new website ready soon with all studies and
resources in one site
 Support from the Chief Economists office and DEC
Lessons Learnt: What worked (1)
 The overall methodology is sound and
important


Sound: Correlates with various characteristics
as predicted by common sense
Important: Highlights potential and limitations
of different policy measures



The distribution of quality across public/private or
rich/poor
The distribution of effort
The know-do gap
Lessons Learnt: What worked (2)
 Initial worry that the
Practitioner Qualifications and Drug Use
variation in doctors is too
large in India (Allopathic,
Ayurvedas, Unani,
Homeopaths) for a single
instrument to capture quality
2.7
Tr
ai
ni
ng
0.7
R
M
P/
No
0.1
2.5
BH
M
S
0.5
2.6

S
0.5
M
BB
BI
M
S/
BA
M
S/
BU
M
S/
0.5
0.0
0
1
Medicines per patient
Alternative Medicines per patient
2
3
Antibiotics per patient
Regardless of the style of the provider’s
training, the type of medications
dispensed were very similar

Turned out to not be a
bit worry because
doctors were all
treating patients using
the same (allopathic)
medicines
Therefore, they could
be graded on the
same scale
Lessons Learnt: What did not work
Vignettes
 We chose the simplest
0
-2
-4
0
-6
5
Percent
10
95% Confidence Intervals
2
15
Classification Errors and Sample Density
-2
-1
0
Competence
Sample Density
Competence
1
2
Upper Confidence Interval
Lower Confidence Interval
cases possible with no
complications
 Even then, our vignettes are
not good at distinguishing
bad from very bad doctors
(high standard-errors for
less than average providers)
 Perhaps adding in a simple
set of written questions
would help

Source: Author’s calculations based on World Bank-ISERDD (2003). The horizontal axis in the graph is
competence, the left vertical axis is the density (in percentages) for the histogram of competence and the
right vertical axis shows confidence intervals of competence. The solid line is estimated competence, which
is plotted against itself (this would be the 45o line if the scales were the same). The two dashed curves
represent the upper and lower confidence intervals at the 95% level of confidence. The histogram
underlying the confidence interval curves shows how competence is distributed at values of the index with
large and small standard errors.
For instance: “If a child is
suffering from diarrhea, what
should you give the child?”
Lessons Learnt: What did not work
Vignettes
 All the 5 cases we chose did not require any
treatment at the primary level.

Therefore, the “mistakes” we usually pick up are
errors of commission—doctors doing things that
they should not have
 Definitely consider including cases that require
treatment at the primary level (pneumonia?)

This allows us to pick up errors of omission
Lessons Learnt: What did not work
Practice
 The vignettes standardize the case-mix and the patient-mix
 Observing real patients does not. This leads to problems


Unobserved patient characteristics could affect inference
(for instance sorting)
 Use an exit-survey to pick some of this up, if
possible
The cases that overlap with the vignettes are limited (rare
cases almost never seen)
 And may not be perfect overlaps
 Possibility of using “simulated standardized patients”

Pilot currently underway: if this works out, it vastly
improves our diagnostic abilities
Lessons Learnt: What did not work
Overall
 The studies thus far are mostly “boutique” studies
 We are working on how to mainstream them
 We are not sure what the cost of deviation from the “boutique”
approach would be
What I would do (again)
 Make sure that you keep lots of time for case
development
 Pilot the vignettes until all (>95%) questions that
providers ask have a predefined answer
 Train enumerators until they have memorized the
entire vignettes module
 NEW: use video-recordings of doctor-patient
interactions as training material for direct observation
(these are being developed)
Things to do differently
 Post-code treatments (the ultimate nightmare)
 Make some changes to the direct observation form
 Patient order
 Interviewer assessments of patients (to be piloted)
 Number of questions that patients ask
 Have a clear idea of the timeline and the work
program (but that’s for any of this work!)
Papers on which this presentation is based