Transcript Document

Using Weather Data to Improve
Detection of Aerosol Releases
William Hogan, MD, MS
Wind
direction
2 days
ago
Bad influenza
outbreak or
anthrax?
Need to look at areas outside
the “linear” pattern to
determine whether they also
have increased level of
disease
Scan Statistics
• Dr. Moore already described
• Rectangular regions
• Compare alignment to recent wind
directions
Sverdlovsk
Goal: Automate this type
of outbreak detection
6
Meselson M et al. The Sverdlovsk Anthrax Outbreak of 1979. Science, 1994;266(5188):1202-1208.
How to Get There
Input data
Model
Output of
model
Detection Algorithm
Weather data
Surveillance data
There are
two pieces
to build!
“Inverse”
Dispersion
Model
Model of
disease
Was there a
release?
Location of
release? (in 3
dimensions)
Amount of
substance
released?
Traditional Dispersion Problem
Meteorological
data
Amount of
substance
released
Location of
release (in 3
dimensions)
Physical
characteristics of
substance
Dispersion
Model
Atmospheric
concentration
of substance at
any given
downwind
location
Detection Problem
Meteorological
data
Distribution
of cases
Physical
characteristics of
substance
“Inverse”
Dispersion
Model
Model of
disease
Was there a
release?
Location of
release? (in 3
dimensions)
Amount of
substance
released?
“Inverse” Gaussian Plume
Atmospheric
concentration
at n points
(usually n=4)
“Inverse”
Gaussian
Plume Model
“Best” set of
release
parameters
Heuristic search
over large space of
possible release
parameters
Inverse Gaussian Plume - Performance
• Differences between actual and found values on
randomly selected release scenarios:
x (m)
Mean
Median
Maximum
y (m)
h (m)
Q (kg)
144.53
420.44
48.02
1.92
15.48
11.65
4
1
1860.8
8539.4
823
29
Disease Model
Aerosol
Background
level of disease
Anthrax Disease Model
Anthrax aerosol
# spores
inhaled
Background respiratory disease
t
P(resp CC due to anthrax)
Month
of year
Day of
week
Zip
code
P(resp CC due to background)
Today’s count of resp CCs in zip code
Preliminary Results
• False positives
– Ran on 7 months of data not used for training
– No false positives with threshold (likelihood ratio) of 1
• Simulated aerosol anthrax release:
Days Ln(LR)
2
3
4
79
203
285
x*
y*
4301 1825
2354 1495
9685 2821
*Values
Q*
h*
11
709
4
285
1 1700
are difference from actual
Algorithm Evaluation
William Hogan, MD, MS
Algorithm Evaluation
•
•
•
•
Performance measurements
Datasets for evaluation
Additional Considerations
Examples
Performance Measurements
• Timeliness
• False alarm rate
• Sensitivity
Timeliness
• Relatively new metric for evaluating performance
• If two algorithms have identical sensitivity and
false alarm rates, then:
The algorithm that detects outbreaks earlier is better
• Outbreak detection falls into a class of problems
known as “activity monitoring”:
Activity Monitoring Operating Characteristic (AMOC)
analysis*
*See
Fawcett T, Provost F. Activity monitoring: Noticing interesting changes
in behavior. In Proc. Fifth International Conference on Knowledge Discovery
and Data Mining, pages 53--62, 1999.
Characteristics of Activity Monitoring
Domains
• Goal is to identify in a timely fashion that
positive activity (e.g. outbreak) has begun
• Goal is NOT to classify each individual
observation or data point as representing
positive or negative activity
• Alarms after the first alarm may add no
value
• No fixed notion of true negative
False Alarm Rate
• Do not need datasets that include outbreaks
of interest
• Procedure:
– Run algorithm on time series with no outbreaks
– Count false alarms
– Divide by length of time series
Sensitivity
• Fraction (or percentage) of outbreaks
detected
• NOT the fraction (or percentage) of data
points correctly classified as occurring
during the outbreak
Standard
WSARE2.0
WSARE2.5
WSARE3.0
Typical AMOC Plot
Evaluation Datasets
• Real outbreaks
– Hard to get enough to achieve statistical significance
– Getting datasets from real outbreaks is difficult
– Seasonally recurring respiratory & gastrointestinal illness
• Not as specific as single disease
• May not be a good proxy for outbreaks we want to detect
• Simulated outbreaks added to real baseline data
(semisynthetic datasets)
– Use epidemic curves from real outbreaks of interest to create shape
of outbreak
– Simulate entire outbreak (e.g., aerosol anthrax release)
Determining Shape From Real Outbreak
Fit curve
using
expectationmaximization
algorithm
4.6 std dev
increase
Simulate Entire Outbreak
Aerosol
Disease/
Population
models
Models of
atmospheric
dispersion
Models of
illness
behavior
Additional Considerations
• Vary size and duration of epidemic curve to
understand limits of detectability
• Include effects of percent coverage
• Sensitivity analysis over assumptions of
simulation models is crucial
Taking Into Account Coverage
• North Battleford curve derived from one
pharmacy
• Naïve approach – add 4.6 standard deviation
increase to baseline
• Assuming 10 stores and all have same mean and
standard deviation:
– Mean and variance increase 10 fold
– Standard deviation increases sqrt(10) times
– Peak of curve is thus 14.5 standard deviations
Examples of Simulation Model
Assumptions
• Infectious dose of anthrax spores
• Incubation period
• Illness behavior - probability that affected
individual, at a given time after symptom
onset, will:
– Purchase OTC
– Go to ED
– Call primary care MD
Example 1: DARPA ’03 Algorithm
Challenge
• All real data
• Three data sets from 5 cities over 4 years
– Data sets
• military outpatient visits to MD
• civilian outpatient visits to MD
• military prescription drugs
– Cities: Charleston, Louisville, Norfolk, Pensacola, Seattle
– Years
• Training - 1/1 or 6/1/1999 through 8/31/2002
• Test – 9/1/02 through 5/31/03
DARPA ’03 Challenge (cont)
• Method:
– Outbreak detection group
• Public health individuals
• From all but one contractor (us)
• Determined gold-standard outbreaks
– Training data to contractors for ~6 weeks
– Test data for 2 weeks, then submit results
– Java application to score results
Pitt/CMU Algorithm Results
Syndrome False Alarm Best Algorithm
Median
Sensitivity
Rate
Timeliness
1 per 2 weeks wav8ssm_max
GI
1
7/7
1 per 4 weeks PANDA1 (change point statistic)
1
6/7
1 per 6 weeks PANDA1 (change
- point statistic)
1
6/7
RESP
1 per 2 weeks
1 per 4 weeks
1 per 6 weeks
Took into
account
holidays but
gold standard
may not have
wav8ssmtwrf_sum
wav8ssmtwrf_sum
Sick -avail (best sensitivity)
wav8ssm_max (best timeliness,
tie)
wav8ssmtwrf_max (best
timeliness, tie)
1
1
6
1
8/8
8/8
8/8
7/8
1
7/8
Limitations of DARPA Eval
• Small n (only 15 outbreaks)
• Natural “outbreaks” studied were:
– not homogeneous (seasonal respiratory and GI)
– how similar to outbreaks of interest?
– subjectively determined, albeit by experts
Example 2: Injected North Battleford Outbreak
False Alarm Rate=25%
10%
5%
2%
Detection Delay (Days from Start of Inject)