Transcript (2014) Improved Medical Risk Assessment and - Bayes

Improved Medical
Risk Assessment and Decision-making
with Bayesian Networks
Department of Surgery and Cancer
Imperial College London
20 May 2014
Norman Fenton
Queen Mary University of London
and
Agena Ltd
Overview
•
•
•
•
•
Why Bayes?
Why Bayesian networks?
Why NOT learn the models from data only?
Case study
Challenges and conclusions
1. WHY BAYES?
The Harvard Problem
One in a thousand people has a prevalence for a
particular heart disease.
A test to detect this disease has:
• 100% sensitivity
• 95% specificity
If a randomly selected person tests positive
what is the probability that the person
actually has the disease?
Bayes Theorem
We have a hypothesis H with prior probability P(H)
We now get some evidence E.
Waste of time
H
E
showing
this
to
(hypothesis)
(evidence)
most people!!!
We know P(E|H) but we want the posterior P(H|E)
P(E|H)*P(H)
P(H|E) = P(E|H)*P(H) =
P(E)
P(E|H)*P(H) + P(E|not H)*P(not H)
P(H|E)
=
1*1/1000
1*1/1000+ 5/100*999/1000
=
0.001
0.001 + 0.04995
 0.0196
Imagine 100,000
people
Slide 6
Out of whom
100 has the
disease
Slide 7
But about 5%
of the
remaining
99900 people
without the
disease test
positive.
That is 4995
people
Slide 8
So 100 out of
5095 who test
positive
actually have
the disease
That’s just
under 2%
That’s very
different from
the 95%
assumed by
most medics
Slide 9
100%
Have the disease
100
Test positive
100
0%
Test negative
0
5%
Test positive
4,995
1/1000
Total people
100,000
999/1000
Don’t have
the disease
99,900
95%
Test negative
94,905
So 100 out of 5,095
who test positive
actually
have the disease, i.e.
under 2%
2. WHY BAYESIAN NETWORKS?
A Simple Bayesian Network
..but here is
a typical
causal
model
Calculations from
first principles are
infeasible and
incomprehensible
Actual model in medical negligence case
•
•
•
•
•
•
MRA
CA
Ischaemic
Small aneurysm
Large aneurysm
CSP
This model already reaches
limit of comprehensibility for
manual calculations and event
trees
CA Test Pathway
Cause of Palsy
Test Result
99%
99%
1%
Large
9,900
Aneurysm
10,000
1%
Total people
1,000,000
Detected by Test
9,900
1%
Undetected by Test
100
90%
Detected by Test
90
10%
Undetected by Test
10
0%
CSP
10,000
98%
1%
98%
Stroke
99
Strokes
99
2
Stroke
1
1
Stroke
1
1
Die from burst/bleeding
0
Don’t die
1%
10
0
Stroke
0
0
Detected by Test
0
50%
100%
Deaths
Die from burst/bleeding
2
Don’t die
1%
98
2%
1%
Others (ischaemic)
980,000
1%
2%
Small
100
98%
Outcome
Undetected by Test
10,000
Die from CSP
5000
5000
50%
Don’t die
5000
1%
1%
Stroke
50
50
Stroke
9799
TOTAL
14,952 out of 1,000,000 give risk
9799
5002
9950
= 1.495%
MRA Test Pathway
Cause of Palsy
Test Result
95%
99%
Large
9,900
5%
Outcome
Deaths
Detected by Test
9,405
Undetected by Test
495
0
Die from burst/bleeding
10
2%
Aneurysm
10,000
1%
1%
50%
Detected by Test
50
50%
Undetected by Test
50
2%
Detected by Test
9,000
20%
Undetected by Test
1000
50%
Small
100
Total people
1,000,000
1%
90%
CSP
10,000
98%
10%
0
Die from burst/bleeding
1
Die from CSP
1800
Die from CSP
500
Others (ischaemic)
980,000
0
TOTAL
2311 out of 1,000,000 give risk
2311
= 0.2311%
Much better solution
…use a Bayesian Network tool
Computation for Catheter Angiogram
Mean:
9950
Mean:
5002
Computation for MRA Scan
Mean:
0
Mean:
2311
The Calculator Analogy
No need for p-tests or classical
confidence intervals
• Drug “Precision” weight loss: Everyone in trial
lost between 4.5 and 5.5 pounds
• Drug “Oomph” weight loss: Everyone in trial
lost between 10 and 30 pounds
• Which drug can we ‘accept’, i.e. reject null
hypothesis of ‘no weight loss’?
• Classical stats provides nonsensical answers
No need for p-tests or classical
confidence intervals
3. WHY NOT LEARN THE MODELS
FROM DATA ONLY?
A typical data-driven study
Age
Delay in
arrival
Injury
type
Brain
scan
result
Arterial
pressure
Pupil
dilation
Outcome
(death
y/n)
17
25
A
N
L
Y
N
39
20
B
N
M
Y
N
23
65
A
N
L
N
Y
21
80
C
Y
H
Y
N
68
20
B
Y
M
Y
N
22
30
A
N
M
N
Y
…
…
…
..
…
…
A typical data-driven study
Injury
type
Brain scan
result
Arterial
pressure
Delay in
arrival
Pupil
dilation
Age
Outcome
Purely data driven
machine learning
algorithms will be
inaccurate and produce
counterintuitive results
e.g. outcome more likely
to be OK in the worst
scenarios
Causal model with intervention
Injury
type
Brain scan
result
Arterial
pressure
Delay in
arrival
Pupil
dilation
Age
Danger
level
TREATMENT
Outcome
..crucial variables missing
from the data
Determining drug effectiveness
Basic results for drug effectiveness
Drug A
The mean financial
benefit is $4156
Drug B
The mean financial
benefit is $2777
Ban drug B?
Model with latent variable (same data)
Note that most patients
in the sample had minor
case of the condition
…and most
patients were
given drug A
Results with 'Patient condition' major
Drug A
Only 10% positive outcome.
The mean financial benefit is
$400
Drug B
30% positive outcome.
The mean financial benefit is
$1000
OK, so we might need expert judgment
when we have missing data, but with
good experimental design and lots of
good quality data we can surely
remove dependency on experts ……
A machine learning fable
A and B are two medical conditions very well known to
doctors Bill and Ludmila.
These conditions are pretty rare (both have an
incidence of about one in 1,000 people).
There is a third medical condition C (whose name is
“FiroziliRalitNoNeOba”) that Bill has heard the name of,
but knows nothing about.
But Bill has heard that patients with either A or B
usually also have C.
Bill has a massive database of 600,000 people with the
details of which conditions they have.
Bill’s data
Patient number
1
2
3
4
5
6
7
8
9
10
11
12
13
14
600,000
A
No
No
Yes
No
No
No
Yes
No
No
No
No
No
No
No
….
….
No
B
No
No
No
No
No
No
No
Yes
No
No
No
Yes
No
No
…
…
No
C
No
No
Yes
No
No
No
Yes
Yes
No
No
No
Yes
No
No
…
…
No
Bill’s machine learning mate Fred
Can use this database to ‘discover’ the
underlying causal model (Bayesian Network)
relating A, B, and C.
But Ludmila says she knows the correct model
without data:
She also “knows”
the probability
tables
Fred warns against this
Fred’s learnt model
600 out of 600,000 have
condition A
600 out of 600,000 have
condition B
Every single person with
condition A also has C and
every single person with B
also has C.
• Ludmilla disagrees with the last column of table C
• Fred: “Not enough data for that”
• Bill: “…why can’t we simply conclude that C must be
true when both A and B are?”
Ludmilla’s knowledge
• The name of Condition C FiroziliRalitNoNeOba - is actually a Russian
word.
• Its literal translation is:
– ‘A person suffering from either Firoz or Ralit but
not both’.
– ‘Firoz’ is the Russian word for condition A and
‘Ralit’ is the Russian word for condition B.”
Moral of the story
• Sometimes you have to trust experts to
provide more informed quantitative
judgement than you will get from data alone.
• Even really big datasets will be insufficient for
some very small problems.
• Trusting the expert can save you a whole load
of unnecessary data-collection and machine
learning effort.
4. CASE STUDY
Trauma Care Case Study
• QM RIM Group
– William Marsh
– Barbaros Yet
• The Royal London Hospital
– Mr Zane Perkins
– Mr Nigel Tai
– ACIT Data
• US Army Institute of
Surgical Research
– Lower Extremity Injury
Data
Yet, B., Perkins Z., Fenton, N.E., Tai, N., Marsh, W., "Not Just
Data: A Method for Improving Prediction with Knowledge",
Journal of Biomedical Informatics, 2014 Apr;48:28-37
BN v MESS Score
How the BN Model Differs
• Prediction: coagulopathy, death (c.f. GCS, TRISS)
• Flexible inputs
• Patient’s physiological state
– Causal modelling: informed by knowledge
Life Saving: Prediction of Physiological
Disorders
Limb Saving: Prediction of Limb
Viability
www.traumamodels.com
5. CHALLENGES AND CONCLUSIONS
Challenges
•
•
•
•
Apparent paradox on using experts
Expert systems have a bad reputation
Resistance to subjective priors
Building new large-scale BN models, especially
with minimal data
• Interacting with large-scale BN models
• Explaining the results of BN models
BAYES-KNOWLEDGE (Effective Bayesian
Modelling with Knowledge Before Data)
www.eecs.qmul.ac.uk/~norman/projects/B_Knowledge.html
Conclusions (1)
• Purely data driven approaches using Machine
learning and statistics DO NOT WORK
• At best captures what did happen Vs what
would have happened
• Need to move to data + knowledge approach
• BNs provide the key
Conclusions (2): BN Benefits
•
•
•
•
Data + knowledge
Models uncertainty and causality
Predictions and diagnosis
Avoid medical statistics fixation on p-values
and confidence intervals
• Incorporate qualitative and quantitative
variables
• Identify causal effects without RCTs
• New generation expert systems
Blatant Plug for Book
CRC Press, ISBN: 9781439809105 , ISBN 10: 1439809100