Transcript An integrated mixed-methods approach

NIHR CLAHRC
Collaboration
for Leadership in Applied Health Research and Care
Northwest London
Northwest London
Evaluation of Improvement
Initiatives
AN INTEGRATED MIXED-METHODS
APPROACH
Dr Tom Woodcock @DrTomWoodcock
25th January 2016
@DrTomWoodcock
The National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care (NIHR CLAHRC) Northwest London
is hosted by Chelsea and Westminster Hospital NHS Foundation Trust and academically led by Imperial College London, in partnership with Northwest London
NIHR CLAHRC
Northwest London
Outline
• Introduction
• Measurement for improvement and
statistical process control (SPC)
• SPC and theory-driven evaluation: could
the whole be greater than the sum of its
parts?
@DrTomWoodcock
NIHR CLAHRC
Northwest London
EvidenceBased
Medicine
Quality
Improvement
Methods
@DrTomWoodcock
Healthcare
Practice
Quality
Improvement
in practice
NIHR CLAHRC
Northwest London
Evaluation in, and of,
Improvement
• Are (particular) changes happening in the way
health care is delivered?
• Are these associated with (particular)
improvements for patients, carers & the public?
• Are these changes and improvements causally
linked, and how?
• How can we reproduce this improvement
elsewhere, avoiding excessive “cost” in doing so?
@DrTomWoodcock
NIHR CLAHRC
Northwest London
“Designs that are better suited to the
evaluation of clearly defined and static
interventions may be adopted without giving
sufficient attention to the challenges
associated with the dynamic nature of
improvement interventions and their
interactions with contextual factors.”
How to study improvement interventions: a brief overview of possible study
types. Portela et al. BMJ Qual Saf doi:10.1136/bmjqs-2014-003620
@DrTomWoodcock
NIHR CLAHRC
Northwest London
Consolidated Framework For
Implementation Research
“Adaptability relies on
a definition of the
'core components' […]
versus the 'adaptable
periphery' […] often
the distinction […] can
only be discerned
through trial and error
over time as the
intervention is
disseminated more
widely and adapted
for a variety of
contexts”
@DrTomWoodcock
Damschroder et al. Implementation Science
2009, 4:50 doi:10.1186/1748-5908-4-50
NIHR CLAHRC
Northwest London
Action-Effect Diagrams
M
Measures
E
Patient-Centred Aim
Evidence
Care
coordination
Care in hospital
M
E
E
Selfmanagement
M
M M
@DrTomWoodcock
E
Inhaler
technique
Pulmonary
Rehab.
E Staff training
M
Patient info
materials
M
M
Follow-up
health care
Referral
process
M
NIHR CLAHRC
Northwest London
COPD Care Bundle - Hospital X
Overall Compliance
100%
90%
80%
70%
60%
Percentage Compliance
50%
Average
Lower Control Limit
40%
30%
20%
10%
0%
2-Jan
2-Feb
@DrTomWoodcock
2-Mar
2-Apr
2-May
2-Jun
2-Jul
2-Aug
2-Sep
2-Oct
2-Nov
2-Dec
NIHR CLAHRC
Northwest London
Section 2
Measurement for Improvement
@DrTomWoodcock
NIHR CLAHRC
Northwest London
The 3 reasons for measurement
Characteristic
Judgement
Research
Improvement
Aim
Achievement of
target
New knowledge
Improvement of
service
Testing Strategy
No tests
One large test
Sequential tests
Sample Size
Obtain 100% of
available,
relevant data
“Just in case”
data
“Just enough” data,
small sequential
samples
Type of
hypothesis
No hypothesis
Fixed hypothesis
Hypothesis is
flexible, changes as
learning takes place
Variation (Bias)
Adjust measures
to reduce
variation
Design to
eliminate
unwanted
variation
Accept consistent
variation
Determining if a
change is an
improvement
No change
focus
Statistical tests
(t-test, chi
square), pvalues
Run charts,
Shewhart control
charts
@DrTomWoodcock
Source: Solberg et al 1997
NIHR CLAHRC
Northwest London
An approach to measurement
for improvement
• Action-effect method – to inform measure
selection
• Operational definition of measures
• Rigorous approach to data quality
• Statistical process control – formative
evaluation
@DrTomWoodcock
NIHR CLAHRC
Northwest London
The measurement cycle
1 Decide aim
2 Choose measures
3 Confirm collection
7 Review measures
6 Take appropriate
action
8 Repeat
steps
4-6
4 Collect data
5 Analyse & present
@DrTomWoodcock
Adapted from a slide by Mike Davidge
NIHR CLAHRC
Northwest London
@DrTomWoodcock
Action Effect Diagram
NIHR CLAHRC
Northwest London
http://bit.ly/1rtdb6L
@DrTomWoodcock
NIHR CLAHRC
Northwest London
What to measure?
Aim
Concept
Measure
Data
Collection
Data
Collection
Plan
Operational
Definitions
Analysis
@DrTomWoodcock
Action
R. Lloyd. Quality Health
Care: A Guide to
Developing and Using
Indicators.
NIHR CLAHRC
End here
Northwest London
Patient’s son
was concerned,
asked doctor
questions
Start here
12.30pm
Called GP
from home
7.15pm
Clerking
includes drug
history,
previous
admissions
etc.
GP arranged
for admission
to AAU
5.00pm
Patient
arrives at
AAU
5.30pm
Patient clerked by
junior doctor (takes
approx. 1 hr)
Elderly patient (86yo
female) presents to AAU
with fever, rigors,
breathlessness and a cough
Patient has had 3 previous admissions
to hospital with similar symptoms.
Has concurrent COPD.
@DrTomWoodcock
Blood taken and IV
cannula inserted by
junior doctor
Patient unsure
as to what was
happening as
previous similar
admissions were
via A+E
Paracetamol and
saline given via drip
Seen by doctors and
told of need to stay
longer for
observation
Bedpan used to
urinate due to
unsteadiness on feet
8.00pm
More tablets given
orally
Chest x-ray taken
8.50pm
Tablets given orally
Patient had no
idea what tablets
she was taking
and what they
were for, even
after asking
9.35am
After sweating
profusely during the
night, the patients’
nightgown was
changed
Patient feels she
is on too many
drugs and they
are doing more
harm than good
NIHR CLAHRC
Northwest London
Doctor
discusses with
patient about
discharge
(medicines &
expectations)
GP takes
over
care of
patient
DSUM
emailed to GP
Nurse liaises
with
community
services
Is patient
waiting for
TTO on
ward?
No
Medical
assessment
Is TTO
written?
No
1
2
No
Is patient
safe for
discharge?
Yes
Inform
medical team
and
document in
patient
record
Nurse incharge
reminds
doctors to
write DSUM
7
Finish
here
7
No
Is DSUM
written?
Is
pharmacist
informed
that TTO is
complete?
Yes
Yes
Pharmacist
screens TTO
Yes
Doctor (or
pharmacist)
prepares
prescription;
Doctor writes
DSUM
Team fit for
discharge
Is patient
safe for
discharge?
Nurse incharge/
pharmacist
reminds
doctors to
write TTO
Yes
Doctor
informs nurse
that TTO is
complete
3
Estimat
e/
review
predicte
d date
of
discharg
e
DSUM
faxed/mailed
to GP
Yes
3
Start
here
No
Yes
Yes
Does
patient
need
community
services
input?
Can DSUM
be emailed
to GP?
Doctor writes
DSUM
Nurse liaises
with patient a
suitable time
to return for
TTO
No
Yes
No
Nurse or
doctor
informs
pharmacist
that TTO is
complete
Was
patient
given TTO?
Pharmacist
discusses
with doctor
to reconcile
medicines
No
5
,
6
Patient
returns to
collect TTO
after
discharge
Patient
discharged
home
4
No
Nurse
counsels
patient about
medicines
No
Are all
medicines
reconciled
on TTO?
Yes
Pharmacist
counsels
patient about
medicines
(where
possible)
Is it out-ofhours?
Nurse
assessment
Yes
Pharmacist
assessment
Is a supply
of
medicines
needed?
TTO ready on
AAU or no
TTO needed
No
Yes
Rapid
Response
Team
assessment
Does
patient
use/need a
compliance
aid?
No
Can TTO be
dispensed
on AAU?
TTO
dispensed on
AAU
Yes
Yes
Pharmacist
liaises with
community
pharmacy
No
TTO
dispensed in
pharmacy
Can TTO be
sent via air
chute?
TTO sent via
air chute
Legend – action to be completed by:
All staff – red
Doctor – grey
Nurse – dark blue
Patient – light blue
Pharmacist – green
Rapid Response Team - purple
Ward clerk – pale purple
For information, please email:
[email protected]
No
@DrTomWoodcock
Yes
Note:
* Discharge process runs in parallel with other ward
activities
* Reference numbers highlight where there are issues
with the current process which needs to be resolved.
Please refer to interventions plan for details.
Ward phoned
to collect TTO
or TTO sent
to ward with
porter
PMNIHR
Process Map
CLAHRC
Northwest London
Measure Definition Process
•Team in place
•Overall Aim agreed
• Action-Effect
Diagram
•Process Map
CP
1
•Work in progress
on definitions.
•Data sources
CP
2
•Interventions ID, Stakeholder
agreement
•Measure concepts and names
@DrTomWoodcock
CP
3
•WISH Operational
•Outcome Measures agreed
& access to data established
CP4
CP
5
•Sign off definitions
•Initial data collection PDSAs
completed
•Baseline data
•Ongoing
evaluation
CP
6
CP
7
•Improvement
Cycles
NIHR CLAHRC
Northwest London
Measurement Planning Assessment
Framework
Collection and
Management
Design
Action
@DrTomWoodcock
Sustainability
Analysis
NIHR CLAHRC
Northwest London
Measurement Planning Assessment
Examples
Is it clear how the
measures are
linked to the aim?
Will there be
quality assurance
reviews of data
entry?
Who will receive
reports/review the
measures
regularly?
@DrTomWoodcock
Are specific
statistical methods
outlined in the
plan?
Is any aspect of the
measurement
process dependent
on an individual?
NIHR CLAHRC
Northwest London
Statistical Process Control
• Analyse variation (common/special cause)
• Take action, measure, analyse…
• Control charts and run charts
• Probability based rules for special causes
• Robust to distribution of the data
• Well suited to evaluation in and of
improvement intiatives!
@DrTomWoodcock
NIHR CLAHRC
Northwest London
Enumerative vs Analytic Study
On Probability As a Basis For Action, W E Deming,
The American Statistician, Vol. 29 No. 4 1975, pp. 146-152
(and previously in ‘42 and ‘50)
“An enumerative study has for its aim an estimate
of the number of units of a frame that belong to a
specified class.
An analytic study has for its aim a basis for action
on the cause-system or the process, in order to
improve product of the future”
@DrTomWoodcock
NIHR CLAHRC
Northwest London
Enumerative vs Analytic Study
Analytical studies: a framework for quality
improvement design and analysis, Lloyd P Provost
BMJ Qual Saf 2011; 20 (Suppl. 1) doi:10.1136/bmjqs.2011.051557
“Because of the temporal nature of improvement, the
theory and methods for analytical studies are a
critical component of the science of improvement.”
Analogy: pond vs river
@DrTomWoodcock
NIHR CLAHRC
Northwest London
Enumerative or Analytic?
1. Establish new patient wait times for appointment
for each GP in CCG X
2. Do practices with a full time nurse practitioner
have shorter waits than those without?
3. Will introduction of nurse practitioners in practices
without one decrease their wait times?
@DrTomWoodcock
NIHR CLAHRC
Northwest London
SPC: Origins
• Dr. Walter Shewhart: physicist & engineer,
Western Electric and Bell Laboratories, 1920s.
Quality in manufacturing.
• Dr. W. Edwards Deming extended Shewhart’s
work, developing and explaining applications (U.S.
Then Japan after WWII, worldwide in the 80s and
90s).
• Increasingly seen in healthcare – but applied with
varying degrees of rigor and success.
@DrTomWoodcock
NIHR CLAHRC
Northwest London
Understanding variation
• SPC tools help us to understand and learn
from variation
• Everything we can measure will vary!
• SPC separates out Special Cause from
Common cause (routine variation)
– Common: those inherent to the process
– Special: Arising because of specific reason
@DrTomWoodcock
NIHR CLAHRC
Northwest London
Control charts
characterise variation
SOMEWHERE HOSPITAL
X Chart of Time in Department of patients attending A&E
Flow 1 attendances | 26 January 2007 | 09:00 to 21:00
Time in department (mins)
90
75
60
45
30
15
0
9:10
9:19
9:55
10:25
11:01
11:07
11:15
12:05
12:25
12:52
13:08
13:22
14:43
15:01
15:30
15:54
16:55
17:14
18:33
18:41
Patient attendances from 09:00 to 21:00
@DrTomWoodcock
19:27
20:12
20:56
NIHR CLAHRC
Northwest London
1. A single point outside the control limits
3. Trend - Six or more consecutive points all
increasing or decreasing
2. A run of Eight or more points in a row all above
or all below the centre line
4. Two of three consecutive points near a control
limit (in the outer third)
5. Fifteen consecutive points close to centre line
(inner third)
HC Data Guide p116.
@DrTomWoodcock
NIHR CLAHRC
Northwest London
Diagnosing your process 1
• A process can be in or out of statistical control
• “In statistical control” = no rule breaks
– many causes of variation – chance, none dominant over the others
– most data fall within process limits
– this is routine variation or “common cause” variation
– process will continue to deliver same results (predictable variation)
• In statistical control:
–only way to improve is to change the process
@DrTomWoodcock
NIHR CLAHRC
Northwest London
Diagnosing your process 2
• “Out of statistical control” = rule breaks
– Still contains routine variation
– also exceptional variation – e.g. outside limits
– causes that dominate (assignable/special)
– BUT process is in flux (for better or for worse)
– variation is not predictable
• Out of statistical control
–Improve by removing assignable causes of variation (or
changing the process).
@DrTomWoodcock
NIHR CLAHRC
Northwest London
Making better decisions with data
Improvement needed
Aim setting
Learn from and act on special causes
Yes
Measure
definition
process
Data
collection
Create /
update
control
chart
Special
cause
variation?
No
Change the system
Monitor to ensure
sustained
@DrTomWoodcock
Yes
Improvement
needed?
No
NIHR CLAHRC
Northwest London
Recalculating limits
A break of rule 2 constitutes evidence of a medium
term change in the process.
If this occurs, it may make sense to recalculate the
limits starting from the rule break (two separate
processes)
@DrTomWoodcock
NIHR CLAHRC
Northwest London
100%
Weekly percentage of patients on Ward X clerked
within 4 hours of arrival (weekdays in-hours)
95%
90%
85%
80%
75%
Percentage clerked in
4h
Average for period
70%
65%
Upper Control Limit
60%
Lower Control Limit
55%
50%
@DrTomWoodcock
Week commencing
NIHR CLAHRC
Northwest London
Monthly % ED Attendances Admitted; England 2011-2014
30%
25%
20%
Average and Range Chart for Monthly
Seasonal Factors
5%
1.06
1.04
1.02
1
0.98
0.96
0.94
0.92
0.9
0.5
Average
0.4
Grand Average
0.3
0.2
0.1
0
1 2 3 4 5 6 7 8 9 101112
Month
0%
@DrTomWoodcock
Range Chart
10%
Average Chart
15%
Lower Average
Limit
Upper Average
Limit
Percentage ED
attendances admitted
Seasonalised
Average
Seasonalised Lower
Process Limit
Seasonalised Upper
Process Limit
NIHR CLAHRC
Northwest London
Web Improvement Support for
Healthcare
Plan Do Study Act
cycles
Comments context
@DrTomWoodcock
NIHR CLAHRC
Northwest London
eLearning Module
@DrTomWoodcock
NIHR CLAHRC
Northwest London
Section 3
SPC and theory-driven evaluation
@DrTomWoodcock
NIHR CLAHRC
Northwest London
Theory in improvement work
“…the explicit application of theory could
shorten the time needed to develop
improvement interventions, optimise their
design, identify conditions of context
necessary for their success, and enhance
learning from those efforts.”
Davidoff et al. BMJ Quality and Safety; Jan 2015 10.1136/bmjqs-2014-003627
@DrTomWoodcock
NIHR CLAHRC
Northwest London
Theory driven evaluation
•
•
•
•
Map out the programme theory
Research evaluation to test out that theory
When, how, why intervention works?
Unpick the complex relationship between
context, content, application and outcomes
• Develop a necessarily contingent and situational
understanding of effectiveness
• Seek theoretical generalisability
Walshe Int. J. for Quality in Health Care; 2007 Vol. 19, No. 2: pp. 57 – 59
doi 10.1093/intqhc/mzm004
@DrTomWoodcock
NIHR CLAHRC
Northwest London
Safer Clinical Systems II
“The evaluation team sought to identify the theory (concepts, rationale
and assumptions) behind the Safer Clinical Systems approach, to
determine how far the approach helped the sites to make their systems
more reliable, and to explain how the approach might work (the
mechanisms of change), while also considering contextual factors.”
• Mixed-method longitudinal study design
• SPC – did the sites make systems more reliable?
• Combination of qualitative and quantitative research
findings gives a richer picture
This evaluation identified the need to improve skills and
processes relating to measurement of quality and safety.
@DrTomWoodcock
NIHR CLAHRC
Northwest London
COPD Bundle: process and
outcome
COPD Care Bundle - Hospital X
Overall Compliance
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
@DrTomWoodcock
Percentage Compliance
Average
Lower Control Limit
Context, adaptability and
reproducibility
Theory-driven SPC evaluation
Theory-driven SPC evaluation
Theory-driven SPC evaluation
NIHR CLAHRC
Northwest London
Testing a theory
Assume each measure either: improves, remains
unchanged, deteriorates (SPC)
With 2 factors there are 9 possible scenarios.
E.g.
This result would not support the hypothesis represented
by the diagram – investigating why should promote learning
@DrTomWoodcock
NIHR CLAHRC
Northwest London
@DrTomWoodcock
NIHR CLAHRC
Northwest London
With one outcome measure, and n process measures hypothesized to
influence it, and two time periods only; 3n+1 possible “results”.
@DrTomWoodcock
NIHR CLAHRC
Northwest London
PDSA, SPC and Programme Theory
Development
Patient-Centred Aim
Care
coordination
Care in hospital
M
E
E
Selfmanagement
M
M M
M
Measures
@DrTomWoodcock
E
Evidence
E
Inhaler
technique
Pulmonary
Rehab.
E Staff training
M
Patient info
materials
M
M
Follow-up
health care
Referral
process
M
NIHR CLAHRC
Northwest London
Conclusions
• Imperative to learn from improvement efforts
• Statistical evaluation of improvement efforts
should be “analytic” in nature
• Publish process and outcome measures along
with programme theory
• “Negative” results don’t necessarily mean the
intervention (defined by a core) doesn’t work
• To understand this we need mixed methods – but
there is a methodological gap
• Theory-driven SPC may provide useful insights in
this direction
@DrTomWoodcock
NIHR CLAHRC
Northwest London
@DrTomWoodcock
NIHR CLAHRC
Northwest London
@DrTomWoodcock
NIHR CLAHRC
Northwest London
The Chebychev Inequality
Tells us how much data MUST lie with a certain distance of the mean
AT LEAST
• 75% of the data must lie within 2 standard deviations of the mean
• 89% of the data must lie within 3 standard deviations of the mean
• 94% of the data must lie within 4 standard deviations of the mean
... and so on...
So, suppose you know that the mean waiting time for a patient in a
particular GP waiting room is 20 mins, and the standard deviation is
5 mins, you know that (unless something changes) 75% of patients
will wait between 10 and 30 mins.
@DrTomWoodcock
NIHR CLAHRC
Northwest London
The Chebychev Inequality
(1867)
Let X be a random variable with expected value μ and
finite variance σ 2. Then, for any real number k > 0,
1
Pr  X    k   2
k
@DrTomWoodcock
NIHR CLAHRC
Northwest London
Trends in resting pulse rates in 9–
11-year-old children in the UK
1980–2008
Age (y)
Study
year
N†
Boys
Mean
Range
Pulse
rate‡
BMI
Overweig
ht/ obese
(%)
Height
Girls
Pulse
rate‡
BMI
Overweig
ht/ obese
(%)
Height
1980
12 164
10.3
9.8–10.7
78.4
(10.5)
16.9 (1.9) 5.7
139.0
(6.3)
81.9
(11.0)
17.1 (2.2) 9.7
138.6 (6.5)
1984–
1985
774
9.7
9.0–10.7
79.2
(10.1)
17.2 ( 2.2) 6.9
141.0
(5.7)
82.1
(10.4)
16.9 (1.9) 8.3
140.5 (6.3)
1991
1293
10.3
9.0–11.7
78.2
(12.6)
17.5 (2.4) 10.0
140.3
(6.0)
82.4
(13.8)
17.7 (2.7) 18.1
140.0 (6.5)
1994
3363
10.5
9.0–11.8
79.3
(12.2)
17.5 (2.7) 13.0
141.0
(5.7)
82.8
(12.1)
17.8 (2.6) 16.5
140.2 (5.9)
1995–
1998
3083
10.5
9.0–11.9
78.6
(11.3)
18.3 (3.3) 16.0
140.2
(6.2)
82.2
(12.0)
18.3 (3.1) 21.6
140.5 (6.4)
2002
1181
10.5
9.0–11.9
80.9
(11.8)
18.8 (3.3) 21.9
140.1
(6.4)
84.0
(11.6)
19.2 (3.7) 30.4
140.8 (6.7)
2006–
2008
985
10.5
9.0–11.9
80.6
(11.6)
18.3 (3.3) 21.1
141.0
(6.8)
82.0
(12.4)
18.8 (3.6) 24.2
141.1 (7.3)
Total
22 843
10.3
9.0–11.9
78.8
(11.1)
17.3 (2.5) 12.0
139.8
(6.4)
82.2
(11.6)
17.7 (2.9) 16.7
139.6 (6.7)
Source: Arch Dis Child doi:10.1136/archdischild-2013-304699
@DrTomWoodcock
NIHR CLAHRC
Northwest London
Example (slightly modified from
table)
In the 2006-2008 sample, there were 985 children.
Suppose we know the following statistics calculated
on the BMI measurements for these 985 children:
Mean: 18.3 kg/m2
Standard deviation: 3.3 kg/m2.
Given this, what is the greatest possible number of
children in the sample with BMI at least 30?
@DrTomWoodcock
NIHR CLAHRC
Northwest London
Unimodal distributions

@DrTomWoodcock

NIHR CLAHRC
Northwest London
Vysochanskiï–Petunin inequality
(1980)
Let X be a random variable with unimodal distribution,
mean μ and finite, non-zero variance σ2. Then, for any
k > √(8/3) = 1.63...
4
Pr  X    k   2
9k
@DrTomWoodcock
NIHR CLAHRC
Northwest London
Vysochanskiï–Petunin inequality
So, if we have unimodal data, we can get more precise, tighter
bounds on how much data must lie within a certain distance of
the mean.
AT LEAST
• 89% of the data must lie within 2 standard deviations of the
mean
• 95% of the data must lie within 3 standard deviations of the
mean
• 97% of the data must lie within 4 standard deviations of the
mean
... and so on...
@DrTomWoodcock
NIHR CLAHRC
Northwest London
What distribution?
@DrTomWoodcock
NIHR CLAHRC
Northwest London
Do you know what distribution?
SDs from
mean
ANY
data
Unimodal
data
Normal data
2
75%
89%
95.450%
3
89%
95%
99.730%
4
94%
97%
99.994%
@DrTomWoodcock
NIHR CLAHRC
Northwest London
@DrTomWoodcock
NIHR CLAHRC
Northwest London
@DrTomWoodcock