Transcript cimms.ou.edu

The 2008 Artificial Intelligence
Competition
Valliappa Lakshmanan
National Severe Storms Laboratory & University of Oklahoma
Elizabeth E. Ebert
Bureau of Meteorology Research Center, Australia
Sue Ellen Haupt
Penn State University, State College, PA
Sponsored by Weather Decision Technologies
7/28/2015
[email protected]
1
Why a competition?



AI committee organizes:
 Conference with papers
 Tutorial session before conference (every 2 years)
The tutorial sessions are very popular, but:
 Gets repetitive
 Same set of techniques presented too often
 Often by same speakers!
 Not clear what the differences are
 Different datasets, etc.
 Can I not just use a machine intelligence or neural network toolbox?
Purpose of competition is to replace tutorial but provide learning experience
 Same dataset, different techniques
 Competitive aspect is just a sideshow – don’t put too much stock into it!
7/28/2015
[email protected]
2
The 2008 Artificial Intelligence Competition
Dataset
Results
7/28/2015
[email protected]
3
Project 1: Skill Score By Storm Type

Try to answer this question (posed by Travis Smith)
 Very critical, but hard to answer based on current knowledge
Does the skill score of a forecast office as evaluated by the NWS depend on
the type of storms that the NWS office faced that year?
Is it the type of weather or is it the forecaster skill?
 Initially, concentrate on tornadoes
 Based on radar imagery, classify the type of storms at every time step
 Take NWS warnings and ground truth information for a lot of cases
 Compute skill scores by type of storm
Summer REU project
 Eric Guillot, Lyndon State
 Mentors: Travis Smith, Don Burgess, Greg Stumpf, V Lakshmanan


7/28/2015
[email protected]
4
Project 2: National Storm Events Database

Build a national storm events database
 With high-resolution radar data combined from multiple radars
 Derived products
 Support spatiotemporal queries
 Collaboration between NSSL, NCDC and OU (CAPS, CSA)
7/28/2015
[email protected]
5
Approach


Project 1: How to get classify lots and lots of radar imagery?
 Need automated way to identify storm type
 Technique:
 Cluster radar fields
 Extract storm characteristics for each cluster
 Associate storm characteristics to human-identified storm type
 Train learning technique (NN/decision tree) to do this automatically
 Let it loose on entire dataset
Project 2: How to support spatiotemporal queries on radar data?
 Can create polygons based on thresholding data
 But need to tie together different data sources
 Need automated way to extract storm characteristics for querying
7/28/2015
[email protected]
6
WDSS-II CONUS Grids


In real-time, combine data from 130+ WSR-88Ds
 Reflectivity and azimuthal shear fields
 Use these to derive products:
 Reflectivity Composite
 VIL
 Echo top heights
 Hail probability (POSH), Hail size estimates (MESH), etc.
 Low-level, mid-level shear
 Many others (90+)
Have the 3D reflectivity and shear products archived
 Can use these to recreate derived products
7/28/2015
[email protected]
7
Cluster Identification Using Kmeans

Hierarchical clustering using
texture segmentation and Kmeans clustering


Technique yields 3 different
scales of clustering


7/28/2015
[email protected]
Lakshmanan, V.,
R. Rabin, and
V. DeBrunner, 2003:
Multiscale storm
identification and
forecast. J. Atm. Res.,
67, 367-380
Chose D to train the
decision tree
Cluster attributes at 420
km^2 (scale D) used for
our study
8
Manual Storm Classification
• Manually classified over
1,000 storms over three days
worth of data (March 28th,
May 5th, and May 28th of
2007).
•Used all the fields
ultimately available to
automated algorithm
•VIL, POSH, MESH,
Rotation Tracks, etc.
•Available in real-time
at
http://wdssii.nssl.noaa.
gov/ over entire
CONUS
7/28/2015
[email protected]
9
Hail Case (Apr. 19, 2003; Kansas)
Reflectivity
Composite from
KDDC, KICT,
KVNX and KTWX
7/28/2015
[email protected]
10
Echo Top
Height of echo
above 18 dBZ
7/28/2015
[email protected]
11
MESH
Maximum
expected size of
hail
7/28/2015
[email protected]
12
VIL
Vertical
Integrated Liquid
7/28/2015
[email protected]
13
Cluster Table

Each identified cluster has these
properties:








ConvectiveArea in km^2
MaxEchoTop and LifetimeEchoTop
MESH and LifetimeMESH
MaxVIL, IncreaseInVIL and
LifetimeMaxVIL
Centroid, LatRadius, LonRadius,
Orientation of ellipse fitted to
cluster
MotionEast, MotionSouth in m/s
Size in km^2
One set of clusters per scale

7/28/2015
We used only the 420km^2 cluster
[email protected]
14
Controlling the Cluster Table


Can choose any gridded field for output
From gridded field, can compute the following statistics within cluster





7/28/2015
Minimum value, Maximum value
Average, Standard deviation
Area within interval (Useful to create histograms)
Increase in value temporally
 Does not depend on cluster association being correct
 Computed image-to-image
Lifetime maximum/minimum
 Depends on cluster association being correct, so better on larger clusters
[email protected]
15
Input Parameters
AspectRatio
dimensionless
An ellipse is fitted to the
storm. This is the ratio of the
length of the major axis to
the length of the minor axis
of the fitted ellipse.
ConvectiveArea
km^2
Area of the storm that is
convective
LatRadius
km
Extent of the storm in the
north-south direction
LatitudeOfCentroid
Degrees
Location of storm's centroid
LifetimeMESH
mm
Maximum expected hail size
of the storm over its entire
past history
LifetimePOSH
dimensionless
Peak probability of severe
hail of the storm over its
entire past history
LonRadius
km
Extent of the storm in the
east-west direction
Continued on next slide
7/28/2015
[email protected]
16
Input Parameters (contd.)
LonRadius
km
Extent of the storm in the
east-west direction
LongitudeOfCentroid
Degrees
Location of the storm's
centroid
LowLvlShear
s^-1
Shear closest to the ground
as measured by radar
MESH
mm
Maximum expected hail size
from storm
MaxRef
dBZ
Maximum reflectivity
observed in storm
MaxVIL
kg/m^2
Maximum vertical integrated
liquid in storm
MeanRef
dBZ
Mean reflectivity within
storm
MotionEast
MetersPerSecond
Speed of storm in easterly
direction
MotionSouth
MetersPerSecond
Speed of storm in southerly
direction
7/28/2015
[email protected]
Continued on next slide
17
Input Parameters (contd.)
OrientationToDueNorth
degrees
Orientation of major axis of
ellipse to due north. A value
of 90 indicates a storm that
is oriented east-west. The
more circular a storm is (see
aspect ratio), the less
reliable this measure is.
POSH
dimensionless
Peak probability of severe
hail in storm
Rot120
s^-1
Peak probability of severe
hail in storm
Rot30
s^-1
Maximum azimuthal shear
observed in storm over the
past 30 minutes
RowName
dimensionless
Storm id
Size
km^2
Storm size
Speed
MetersPerSecond
Speed of storm
7/28/2015
[email protected]
18
Types of Storms

Four categories:
 Not organized
 Isolated supercell
 Convective lines
 Includes lines with embedded supercells
 Pulse storms
7/28/2015
[email protected]
19
Decision Tree Training
• Trained decision tree
using manually classified
storms in order to develop
a logical process for
automatically classifying
them
• Tested this decision tree
on three additional cases
(April 21st of 2007, and
May 10th and 14th of 2006)
• TSS=0.58; good
enough for NWS
study to continue
7/28/2015
[email protected]
20
Decision Tree

Why decision tree?



7/28/2015
Didn’t know whether the dataset
was tractable
Wanted to be able to analyze
resulting “machine”
Make sure extracted rules were
reasonable
[email protected]
21
The 2008 Artificial Intelligence Competition
Dataset
Results
7/28/2015
[email protected]
22
Entries


Received 6 official, and one unofficial, entry by competition deadline
 Unofficial entry not accompanied by abstract or AMS manuscript
 Neil Gordon (Met Service, New Zealand): random forest
 Not eligible for prize, but included in comparisons
Official Entries:
 John K. Williams and Jenny Abernathy: random forests and fuzzy logic
 Ron Holmes: neural network
 David Gagne and Amy McGovern: boosted decision tree
 Jenny Abernathy and John Williams: support vector machines
 Luna Rodriguez: genetic algorithms
 Kimberly Elmore: discriminant analysis and support vector machines
7/28/2015
[email protected]
23
Distribution of storm categories
Category 1 - Isolated supercell
1200
1200
1000
1000
800
800
Count
Count
Category 0 - Not severe
600
600
400
400
200
200
0
0
Entry
Entry
Category 4 - Pulse storm
1200
1200
1000
1000
800
800
Count
Count
Category 2 - Convective line
600
600
400
400
200
200
0
0
Entry
Truth
Baseline
Abernethy & Williams
7/28/2015
Entry
Elmore & Richman
Gagne & McGovern
Gordon
[email protected]
Holmes
Rodriguez
Williams & Abernethy
24
Classifications for observed class 0
(Not severe)
Baseline
Gordon
Not severe
7/28/2015
Abernethy & Williams
Holmes
Elmore & Richman
Rodriguez
Isolated supercell
Gagne & McGovern
Williams & Abernethy
Convective line
[email protected]
Pulse storm
25
Classifications for observed class 1
(Isolated supercell)
Baseline
Gordon
Not severe
7/28/2015
Abernethy & Williams
Holmes
Elmore & Richman
Rodriguez
Isolated supercell
Gagne & McGovern
Williams & Abernethy
Convective line
[email protected]
Pulse storm
26
Classifications for observed class 2
(Convective line)
Baseline
Gordon
Not severe
7/28/2015
Abernethy & Williams
Holmes
Elmore & Richman
Rodriguez
Isolated supercell
Gagne & McGovern
Williams & Abernethy
Convective line
[email protected]
Pulse storm
27
Classifications for observed class 4
(Pulse storm)
Baseline
Gordon
Not severe
7/28/2015
Abernethy & Williams
Holmes
Elmore & Richman
Rodriguez
Isolated supercell
Gagne & McGovern
Williams & Abernethy
Convective line
[email protected]
Pulse storm
28
Similarity matrix - % of identical
classifications among entries
Truth
Abernethy
&
Elmore & Gagne &
Baseline Williams Richman McGovern Gordon
Williams
&
Aberneth
Holmes Rodriguez
y
100
74
72
67
77
76
62
53
77
Baseline
74
100
77
69
84
84
62
52
84
Abernethy & Williams
72
77
100
70
83
80
61
52
83
Elmore & Richman
67
69
70
100
75
76
54
55
73
Gagne & McGovern
77
84
83
75
100
93
62
57
93
Gordon
76
84
80
76
93
100
61
58
91
Holmes
62
62
61
54
62
61
100
32
62
Rodriguez
53
52
52
55
57
58
32
100
55
Williams & Abernethy
77
84
83
73
93
91
62
55
100
Truth
7/28/2015
[email protected]
29
Statistical results – True Skill Statistic
True Skill Statistic
Joint First
1
Third
0.8
0.6
Baseline
Abernethy & Williams
Elmore & Richman
0.4
Gagne & McGovern
Gordon
Holmes
0.2
Rodriguez
Williams & Abernethy
0
7/28/2015
Entry
[email protected]
30
Statistical results – Accuracy and Heidke
Skill Score
1
Accuracy (fraction correct)
0.8
0.6
0.4
0.2
0
Baseline
Entry
1
Heidke Skill Score
Abernethy & Williams
Elmore & Richman
0.8
Gagne & McGovern
0.6
Gordon
Holmes
0.4
Rodriguez
0.2
Williams & Abernethy
0
Entry
7/28/2015
[email protected]
31
Acknowledgements

Thanks to:
 Weather Decision Technologies for sponsoring the prizes
 The AMS probability and statistics committee
 For loaning us Beth Ebert’s expertise
 All the participants for entering competition and explaining methodology
 Can be hard to find time to do “extra-curricular” work
 Very grateful that you could enter this competition
7/28/2015
[email protected]
32
Where to go from here?


Please share with us your thoughts and suggestions
 Is such a competition worth doing?
 Was this session a learning experience?
 How can it be improved in the future?
 Is there something that you would have done differently? Why?
Our thoughts:
 Classification is not the only aspect of machine intelligence
 Estimation, association finding, knowledge capture, clustering, …
 Perhaps a future competition could address one of these areas
 Address another aspect of AMS besides short-term severe weather
7/28/2015
[email protected]
33