Transcript UMMC Template - Institute for Health Technology Transformation

Predicting Readmissions
(and other outcomes)
Doesn’t Take a PhD
John Showalter, MD MSIS
Chief Health Information Officer
University of Mississippi Medical Center
November 12, 2013
Section name here
Objectives
 Discuss why advanced mathematical modeling is not
always superior to straight-forward calculations.
 Describe how readily available administrative data can
be used to predict risk for readmission at your
institution.
 Explain certainty factor analysis and how it can be
applied to healthcare analytics
Section name here
Punch-line
 You can predict readmissions:
– At the time of admission
– With data you already have
– Without advanced analytics software
Section name here
MYCIN
 Computer program that used a rule based
system to suggest treatment for serious
infections
 Developed in the early 1970s
 Outperformed specialists in treatment selection
 Based on a novel method of handling
uncertainty in decision making (Certainty
Factors)
Section name here
Certainty Factors
 Designed to handle dependency between
variables
 Based on individual estimates of certainty
 Scale: -100 (absolute certainty of no event) to
100 (absolute certainty of event)
 Calculates the strength of a belief not a
probability
 Widely used in rule-based computer programs
for a short period
Section name here
Fall of Certainty Factors
 Mathematically proven to be inferior to
advanced conditional probabilistic models
– Except for simple belief calculations
 Development of Belief Networks for both simple
and advanced belief calculations
 Only allows forward reasoning
 Infrequently used and even more infrequently
mentioned in the last 20 years
Why use Certainty Factors in
healthcare?
Section name here
 Almost all variables are dependent
– Weight effects diabetes risk
– Age effects heart attack risk
– Treatments effect outcomes
 It is designed for rule based logic systems
– Almost all/if not all clinical decision support systems
are rule-based systems
 The math is straight-forward and can be
handled in the vast majority of EHRs
Section name here
Necessary Modifications to Certainty
Factors
 Only explore the certainty of the event
occurring (i.e. 0 – 100)
 Calculate Certainty Factors based on rates
since data is readily available
 Correlate strength of belief (Certainty
Factor)with risk stratification of potential event
Section name here
Predicting Readmission Study
 Create a method of predicting readmissions at
the time of admission
 Use readily available administrative data
 Compare modified certainty factor analysis to
advanced machine learning algorithms
 6,448 discharges for the Internal Medicine
Service
 30 day readmissions
Section name here
Predicting Readmission Study
 Used four administrative variables
– Number of diagnoses bill in 1 year prior to admission
– Boolean (Y/N)
• Hospital admission within 1 year prior to current admission
• ED visit within 1 year prior to current admission
• Outpatient clinic visit within 1 year prior to current admission
 Compared Several Predictive Model
–
–
–
–
Certainty Factors
Bayesian Network
2 Artificial Neural Networks
Support Vector Machine
Section name here
Study Results – All Readmissions
Certainty
Factors
Bayesian
Network
ANN Multilayer
Perception
ANN Radial
Basis Function
Number of
Discharges
Number of
Readmissions
Rate
Number of
Discharges
Number of
Readmissions
Rate
Number of
Discharges
Number of
Readmissions
Rate
Number
of
Discharges
Number of
Readmissions
Rate
Low
Risk
1,032
(37.6)
108
(26.5%)
10.5
%
1,754
(63.9)
217
(53.3)
12.3%
1,453
(53.0)
173
(42.3)
11.8%
1,720
(62.7)
202
(49.6)
11.7%
Moderate
Risk
1,441
(52.5)
212
(52.1)
14.7
%
741
(27.0)
115
(28.3)
15.5%
999
(36.4)
140
(34.4)
14.0%
741
(27.0)
115
(28.3)
15.5%
High
Risk
270
(9.8)
87
(21.4)
32.3
%
248
(9.0)
75
(18.4)
30.2%
291
(10.6)
95
(23.3)
32.6%
282
(10.3)
90
(22.1)
31.9%
AUC
0.596
0.587
0.599
0.615
Section name here
Study Results – All Readmissions
Certainty
Factors
Bayesian
Network
ANN Multilayer
Perception
ANN Radial
Basis Function
Number of
Discharges
Number of
Readmissions
Rate
Number of
Discharges
Number of
Readmissions
Rate
Number of
Discharges
Number of
Readmissions
Rate
Number
of
Discharges
Number of
Readmissions
Rate
Low
Risk
1,032
(37.6)
108
(26.5%)
10.5
%
1,754
(63.9)
217
(53.3)
12.3%
1,453
(53.0)
173
(42.3)
11.8%
1,720
(62.7)
202
(49.6)
11.7%
Moderate
Risk
1,441
(52.5)
212
(52.1)
14.7
%
741
(27.0)
115
(28.3)
15.5%
999
(36.4)
140
(34.4)
14.0%
741
(27.0)
115
(28.3)
15.5%
High
Risk
270
(9.8)
87
(21.4)
32.3
%
248
(9.0)
75
(18.4)
30.2%
291
(10.6)
95
(23.3)
32.6%
282
(10.3)
90
(22.1)
31.9%
AUC
0.596
0.587
0.599
0.615
Section name here
Study Results – All Readmissions
Certainty
Factors
Bayesian
Network
ANN Multilayer
Perception
ANN Radial
Basis Function
Number of
Discharges
Number of
Readmissions
Rate
Number of
Discharges
Number of
Readmissions
Rate
Number of
Discharges
Number of
Readmissions
Rate
Number
of
Discharges
Number of
Readmissions
Rate
Low
Risk
1,032
(37.6)
108
(26.5%)
10.5
%
1,754
(63.9)
217
(53.3)
12.3%
1,453
(53.0)
173
(42.3)
11.8%
1,720
(62.7)
202
(49.6)
11.7%
Moderate
Risk
1,441
(52.5)
212
(52.1)
14.7
%
741
(27.0)
115
(28.3)
15.5%
999
(36.4)
140
(34.4)
14.0%
741
(27.0)
115
(28.3)
15.5%
High
Risk
270
(9.8)
87
(21.4)
32.3
%
248
(9.0)
75
(18.4)
30.2%
291
(10.6)
95
(23.3)
32.6%
282
(10.3)
90
(22.1)
31.9%
AUC
0.596
0.587
0.599
0.615
Section name here
Study Results –
Unplanned Readmissions*
Certainty
Factors
Bayesian
Network
ANN Multilayer
Perception
ANN Radial
Basis Function
Number of
Discharges
Number of
Readmissions
Rate
Number of
Discharges
Number of
Readmissions
Rate
Number of
Discharges
Number of
Readmissions
Rate
Number
of
Discharges
Number of
Readmissions
Rate
Low
Risk
1,032
(37.6)
56
(18.5)
5.4%
2,335
(85.1)
216
(71.3)
9.3%
2,055
(74.9)
163
(53.8)
7.9%
2,173
(79.2)
183
(60.4)
8.4%
Moderate
Risk
1,441
(52.5)
165
(54.4)
11.5
%
160
(5.8)
21
(6.9)
13.1%
274
(10.0)
38
(12.5)
13.9%
279
(10.2)
34
(11.2)
12.2%
High
Risk
270
(9.8)
82
(27.1)
27.1
%
248
(9.0)
66
(21.8)
26.6%
415
(15.1)
102
(33.7)
24.6%
291
(10.6)
86
(28.4)
29.6%
AUC
0.648
0.620
0.647
0.686
* Defined by readmission to the Internal Medicine Service
Section name here
Study Results –
Unplanned Readmissions*
Certainty
Factors
Bayesian
Network
ANN Multilayer
Perception
ANN Radial
Basis Function
Number of
Discharges
Number of
Readmissions
Rate
Number of
Discharges
Number of
Readmissions
Rate
Number of
Discharges
Number of
Readmissions
Rate
Number
of
Discharges
Number of
Readmissions
Rate
Low
Risk
1,032
(37.6)
56
(18.5)
5.4%
2,335
(85.1)
216
(71.3)
9.3%
2,055
(74.9)
163
(53.8)
7.9%
2,173
(79.2)
183
(60.4)
8.4%
Moderate
Risk
1,441
(52.5)
165
(54.4)
11.5
%
160
(5.8)
21
(6.9)
13.1%
274
(10.0)
38
(12.5)
13.9%
279
(10.2)
34
(11.2)
12.2%
High
Risk
270
(9.8)
82
(27.1)
27.1
%
248
(9.0)
66
(21.8)
26.6%
415
(15.1)
102
(33.7)
24.6%
291
(10.6)
86
(28.4)
29.6%
AUC
0.648
0.620
0.647
0.686
* Defined by readmission to the Internal Medicine Service
Section name here
Study Results –
Unplanned Readmissions*
Certainty
Factors
Bayesian
Network
ANN Multilayer
Perception
ANN Radial
Basis Function
Number of
Discharges
Number of
Readmissions
Rate
Number of
Discharges
Number of
Readmissions
Rate
Number of
Discharges
Number of
Readmissions
Rate
Number
of
Discharges
Number of
Readmissions
Rate
Low
Risk
1,032
(37.6)
56
(18.5)
5.4%
2,335
(85.1)
216
(71.3)
9.3%
2,055
(74.9)
163
(53.8)
7.9%
2,173
(79.2)
183
(60.4)
8.4%
Moderate
Risk
1,441
(52.5)
165
(54.4)
11.5
%
160
(5.8)
21
(6.9)
13.1%
274
(10.0)
38
(12.5)
13.9%
279
(10.2)
34
(11.2)
12.2%
High
Risk
270
(9.8)
82
(27.1)
27.1
%
248
(9.0)
66
(21.8)
26.6%
415
(15.1)
102
(33.7)
24.6%
291
(10.6)
86
(28.4)
29.6%
AUC
0.648
0.620
0.647
0.686
* Defined by readmission to the Internal Medicine Service
UMMC Preliminary Results –
All Readmissions
Certainty
Factors
Number of
Discharges
Number of
Readmissions
Rate
Low
Risk
2,566
(59.7)
84
(21.9)
3.3%
Moderate
Risk
1,045
(24.4)
135
(35.2)
12.9
%
High
Risk
675
(15.7)
165
(43.0)
24.4
%
AUC
0.744
Section name here
Section name here
Create Certainty Factor Model
General Equation
CFT = CF1 + CF2 * (1 – CF1 ) + (CF3*(1-(CF1 + CF2 * (1 – CF1 ))))...
Certainty Factor of Usage (CFU)
Certainty Factor of Diagnosis (CFD)
Prior IP
Prior OP
Prior ED
Number of Diagnoses
in Prior Year
Yes
Yes
Yes
0
Yes
Yes
No
1-10
Yes
No
Yes
Yes
No
No
No
Yes
Yes
No
Yes
No
No
No
Yes
No
No
No
Readmission
Rate (CFU)
Readmission Rate (CFD)
Greater than 10
Calculate CFR and then select risk level cut-offs
Equation for readmssion model
CFR = CFU + CFD * (1 – CFU)
Study CFR Cut-offs
Low Risk 0–0.199
Moderate Risk 0.2–0.352
High Risk 0.353–0.6
Certainty Factors –
Potential Applications
 Risk stratification based on historic data
– Incidental finding on chest x-ray
 Risk assessment based on current data
– Mortality from infection/sepsis
 Real-time alerts about changes in risk
– Failure to rescue
Section name here
Section name here
Recap
 You don’t need “Big Data” to make predictions
 You don’t need a PhD to do the math
 Timely, actionable knowledge is possible with
Certainty Factors
Section name here
Questions and Answers
 Email: [email protected]