Transcript Linked Environments for Atmospheric Discovery (LEAD)

Linked Environments for
Atmospheric Discovery
(LEAD)
Kelvin K. Droegemeier
School of Meteorology and
Center for Analysis and Prediction of Storms
University of Oklahoma
Jay Alameda
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
Linked Environments for Atmospheric Discovery
Geosciences CI Challenges
• Enormously complex human-natural system
– Vast temporal (sec to B yrs) and spatial (microns to
1000s of km) scales
– Highly nonlinear behavior
• Massive data sets
–
–
–
–
–
physical and digital
static/legacy and dynamic/streaming
geospatially referenced
multidisciplinary and heterogeneous
open access
Linked Environments for Atmospheric Discovery
Geosciences CI Challenges
• Massive computation
– weather, space weather,
climate, hydrologic modeling
– seismic inversion
– coupled physical system models
• Inherently field-based, visual disciplines with the
need to manage information for long periods of
time
• Bringing advanced CI capabilities to education at all
levels
• Connecting the last mile to operational practitioners
Linked Environments for Atmospheric Discovery
Where ALL These Elements
Converge: Mesoscale Weather
• Each year, mesoscale weather – floods, tornadoes,
hail, strong winds, lightning, and winter storms –
causes hundreds of deaths, routinely disrupts
transportation and commerce, and results in annual
economic losses > $13B.
Linked Environments for Atmospheric Discovery
Linked Environments for Atmospheric Discovery
What Would You Do???
Linked Environments for Atmospheric Discovery
What Weather Technology Does…
NEXRAD Radar
Forecast Models
Decision Support Systems
Linked Environments for Atmospheric Discovery
What Weather Technology Does…
NEXRAD Radar
Forecast Models
Absolutely Nothing!
Decision Support Systems
Linked Environments for Atmospheric Discovery
The LEAD Goal
Provide the IT necessary to allow
People (scientists, students, operational
practitioners)
and
Technologies (models, sensors, data
mining)
TO INTERACT WITH WEATHER
Linked Environments for Atmospheric Discovery
The Roadblock
• The study of mesoscale weather is stifled by rigid
IT frameworks that cannot accommodate the
– real time, on-demand, and dynamically-adaptive needs of
mesoscale weather research;
– its disparate, high volume data sets and streams; and
– its tremendous computational demands, which are
among the greatest in all areas of science and
engineering
• Some illustrative examples…
Linked Environments for Atmospheric Discovery
Traditional Methodology
STATIC OBSERVATIONS
Analysis/Assimilation
Prediction/Detection
Radar Data
Mobile Mesonets
Surface Observations
Upper-Air Balloons
Commercial Aircraft
Geostationary and Polar
Orbiting Satellite
Wind Profilers
GPS Satellites
Quality Control
Retrieval of Unobserved
Quantities
Creation of Gridded Fields
PCs to Teraflop Systems
Product Generation,
Display,
Dissemination
End Users
NWS
Private Companies
Students
Linked Environments for Atmospheric Discovery
Traditional Methodology
STATIC OBSERVATIONS
Analysis/Assimilation
Prediction/Detection
Radar Data
Mobile Mesonets
Surface Observations
Upper-Air Balloons
Commercial Aircraft
Geostationary and Polar
Orbiting Satellite
Wind Profilers
GPS Satellites
Quality Control
Retrieval of Unobserved
Quantities
Creation of Gridded Fields
PCs to Teraflop Systems
Product Generation,
Display,
Dissemination
The Process is Entirely Serial
and Static (Pre-Scheduled):
No Response to the Weather!
End Users
NWS
Private Companies
Students
Linked Environments for Atmospheric Discovery
The Consequence: Model Grids
Fixed in Time – No Adaptivity
Linked Environments for Atmospheric Discovery
The LEAD Vision: No Longer Serial or Static
STATIC OBSERVATIONS
Analysis/Assimilation
Prediction/Detection
Radar Data
Mobile Mesonets
Surface Observations
Upper-Air Balloons
Commercial Aircraft
Geostationary and Polar
Orbiting Satellite
Wind Profilers
GPS Satellites
Quality Control
Retrieval of Unobserved
Quantities
Creation of Gridded Fields
PCs to Teraflop Systems
Product Generation,
Display,
Dissemination
Models Responding to Observations
End Users
Linked Environments for Atmospheric Discovery
NWS
Private Companies
Students
Model Dynamic Adaptivity
t = to
20 km
10 km
3 km
1 km
Linked Environments for Atmospheric Discovery
t = to + 6 Hours
20 km
10 km
3 km
10 km
3 km
3 km
Linked Environments for Atmospheric Discovery
3 km
Today’s Standard Computer Forecast
Radar
12-hour National
Forecast (coarse grid)
Radar
(Tornadoes
in Arkansas)
Linked Environments for Atmospheric Discovery
Today’s Standard Computer Forecast
Radar
12-hour National
Forecast (coarse grid)
Radar
(Tornadoes
in Arkansas)
Linked Environments for Atmospheric Discovery
Experimental Mesoscale Window
Radar
Radar
6-hour Mesoscale
Forecast
(medium grid)
Radar
(Tornadoes
in Arkansas)
Linked Environments for Atmospheric Discovery
Experimental Mesoscale Window
Radar
Radar
6-hour Mesoscale
Forecast
(medium grid)
Radar
(Tornadoes
in Arkansas)
Linked Environments for Atmospheric Discovery
Experimental Storm-Scale Window
Radar
6-hour Local
Forecast (fine grid)
Linked Environments for AXue
tmospheric
Discovery
et al. (2003)
Dynamic Adaptivity in Action
Linked Environments for Atmospheric Discovery
11 h Forecast
20 June 2001
(6 km)
Courtesy Weather Decision Technologies, Inc.
Linked Environments for Atmospheric Discovery
9 h Forecast
20 June 2001
(6 km)
Courtesy Weather Decision Technologies, Inc.
Linked Environments for Atmospheric Discovery
5 h Forecast
20 June 2001
(6 km)
Courtesy Weather Decision Technologies, Inc.
Linked Environments for Atmospheric Discovery
3 h Forecast
20 June 2001
(6 km)
Courtesy Weather Decision Technologies, Inc.
Linked Environments for Atmospheric Discovery
LEAD: Users INTERACTING with Weather
Mesoscale
Weather
Linked Environments for Atmospheric Discovery
LEAD: Users INTERACTING with Weather
NWS National Static
Observations & Grids
Mesoscale
Weather
Linked Environments for Atmospheric Discovery
LEAD: Users INTERACTING with Weather
NWS National Static
Observations & Grids
Mesoscale
Weather
Local Observations
Linked Environments for Atmospheric Discovery
LEAD: Users INTERACTING with Weather
NWS National Static
Observations & Grids
ADaM ADAS
Mesoscale
Weather
Users
Tools
Local Observations
Linked Environments for Atmospheric Discovery
LEAD: Users INTERACTING with Weather
NWS National Static
Observations & Grids
Virtual/Digital Resources
and Services
ADaM ADAS
Mesoscale
Weather
Users
Tools
MyLEAD
Portal
Remote Physical
(Grid) Resources
Local Physical Resources
Local Observations
Linked Environments for Atmospheric Discovery
LEAD: Users INTERACTING with Weather
Interaction Level I
NWS National Static
Observations & Grids
Virtual/Digital Resources
and Services
ADaM ADAS
Mesoscale
Weather
Users
Tools
MyLEAD
Portal
Remote Physical
(Grid) Resources
Local Physical Resources
Local Observations
Linked Environments for Atmospheric Discovery
Traditional Methodology
STATIC OBSERVATIONS
Analysis/Assimilation
Prediction/Detection
Radar Data
Mobile Mesonets
Surface Observations
Upper-Air Balloons
Commercial Aircraft
Geostationary and Polar
Orbiting Satellite
Wind Profilers
GPS Satellites
Quality Control
Retrieval of Unobserved
Quantities
Creation of Gridded Fields
PCs to Teraflop Systems
Product Generation,
Display,
Dissemination
Observing Systems Operate
Largely Independent of the
Weather – Little Adaptivity
End Users
Linked Environments for Atmospheric Discovery
NWS
Private Companies
Students
NEXRAD Doppler Weather
Radar Network
Linked Environments for Atmospheric Discovery
The Limitations of NEXRAD
Linked Environments for Atmospheric Discovery
The Limitations of NEXRAD
#1. Operates largely independent
of the prevailing weather conditions
Linked Environments for Atmospheric Discovery
The Limitations of NEXRAD
#1. Operates largely independent
of the prevailing weather conditions
#2. Earth’s curvature prevents 72% of the
atmosphere below 1 km from being observed
Linked Environments for Atmospheric Discovery
The Limitations of NEXRAD
#1. Operates largely independent
of the prevailing weather conditions
#3. Operates entirely independent from
the models and algorithms that use its data
#2. Earth’s curvature prevents 72% of the
atmosphere below 1 km from being observed
Linked Environments for Atmospheric Discovery
The Consequence: 3 of Every 4
Tornado Warnings is a False Alarm
NWS
of Science and Technology
LSource:
inked
EOffice
nvironments
for Atmospheric Discovery
The LEAD Vision: No Longer Serial or Static
DYNAMIC OBSERVATIONS
Analysis/Assimilation
Prediction/Detection
Quality Control
Retrieval of Unobserved
Quantities
Creation of Gridded Fields
PCs to Teraflop Systems
Product Generation,
Display,
Dissemination
Models and Algorithms Driving Sensors
End Users
Linked Environments for Atmospheric Discovery
NWS
Private Companies
Students
New NSF Engineering Research
Center for Adaptive Sensing of the
Atmosphere (CASA)
• UMass/Amherst is lead institution
• Concept: inexpensive, dual-polarization phased array Doppler radars on
cell towers – existing IT and power infrastructures!
• Adaptive dynamic sensing of multiple targets (“DCAS”)
Linked Environments for Atmospheric Discovery
New NSF Engineering Research
Center for Adaptive Sensing of the
Atmosphere (CASA)
• UMass/Amherst is lead institution
• Concept: inexpensive, dual-polarization phased array Doppler radars on
cell towers – existing IT and power infrastructures!
• Adaptive dynamic sensing of multiple targets (“DCAS”)
Linked Environments for Atmospheric Discovery
New NSF Engineering Research
Center for Adaptive Sensing of the
Atmosphere (CASA)
• UMass/Amherst is lead institution
• Concept: inexpensive, dual-polarization phased array Doppler radars on
cell towers – existing IT and power infrastructures!
• Adaptive dynamic sensing of multiple targets (“DCAS”)
Linked Environments for Atmospheric Discovery
LEAD: Users INTERACTING with Weather
NWS National Static
Observations & Grids
Virtual/Digital Resources
and Services
ADaM ADAS
Mesoscale
Weather
Users
Tools
MyLEAD
Portal
Remote Physical
(Grid) Resources
Local Physical Resources
Local Observations
Linked Environments for Atmospheric Discovery
LEAD: Users INTERACTING with Weather
NWS National Static
Observations & Grids
Virtual/Digital Resources
and Services
Mesoscale
Weather
Experimental Dynamic
Observations
ADaM ADAS
Users
Tools
MyLEAD
Portal
Remote Physical
(Grid) Resources
Local Physical Resources
Local Observations
Linked Environments for Atmospheric Discovery
LEAD: Users INTERACTING with Weather
Interaction Level II
NWS National Static
Observations & Grids
Virtual/Digital Resources
and Services
Mesoscale
Weather
Experimental Dynamic
Observations
ADaM ADAS
Users
Tools
MyLEAD
Portal
Remote Physical
(Grid) Resources
Local Physical Resources
Local Observations
Linked Environments for Atmospheric Discovery
The LEAD Goal Restated
• To create an integrated, scalable framework that
allows analysis tools, forecast models, and data
repositories to be used as dynamically adaptive,
on-demand systems that can
– change configuration rapidly and automatically in
response to weather;
– continually be steered by new data (i.e., the weather);
– respond to decision-driven inputs from users;
– initiate other processes automatically; and
– steer remote observing technologies to optimize data
collection for the problem at hand;
– operate independent of data formats and the physical
location of data or computing resources
Linked Environments for Atmospheric Discovery
CS Challenges/Barriers
• Workflow
– Dynamic/agile/reentrant
• Data
– Synchronization, fault-tolerance, metadata, cataloging,
interchange, ontologies
• Monitoring and performance estimation
– Detection of vulnerabilities, recovery, autonomy
• Mining
– Grid functionality, scheduling, fault tolerance
Linked Environments for Atmospheric Discovery
Meteorology Challenges/Barriers
• “Packaging” of complex systems (WRF, ADAS)
• Fault tolerance
• Continuous model updating for effective use of truly
streaming observations
• Storm-scale ensemble methodologies
• Hazardous weather detections based upon gridded
analyses versus use of “raw” sensor data alone
• Dynamically adaptive forecasting (models and
observations) – how good compared to current static
methodologies?
Linked Environments for Atmospheric Discovery
LEAD Architecture
User
Interface
Crosscutting
Services
LEAD Portal
Desktop Applications
Portlets
Resource
Access
Services
Distributed
Resources
Linked Environments for Atmospheric Discovery
Data Services
Application & Configuration Services
Workflow
Services
Application Resource
Broker (Scheduler)
Catalog
Services
Configuration and
Execution Services
Client Interface
LEAD Architecture
User
Interface
Crosscutting
Services
LEAD Portal
Portlets
Education
Workflow
Visualization
MyLEAD
Desktop Applications
• IDV
• WRF Configuration GUI
Query
Ontology
Control
Browse
Monitor
Control
Monitoring
Notification
Application & Configuration Services
Workflow
Engine/Factories
Host Environment
Execution Description
Application Host
Application Description
VO Catalog
GPIR
Applications (WRF, ADaM,
IDV, ADAS)
THREDDS
Resource
Access
Services
Distributed
Resources
Globus
GRAM
Scheduler
SSH
Computation
OPenDAP
LDM
Observations
• Streams
• Static
• Archived
Workflow
Services
Workflow
Service
Generic
Ingest Service
Specialized
Applications
Linked Environments for Atmospheric Discovery
Stream
Service
Control
Service
Query
Service
Ontology
Service
Data Services
Authentication
Application Resource
Broker (Scheduler)
Catalog
Services
Authorization
Configuration and
Execution Services
Client Interface
Decoder/Resolver
Service
RLS
Steerable
Instruments
OGSADAI
Data Bases
Storage
Key System Components and
Technologies
Capability/Resource
Principal Technologies
Atmospheric, Oceanographic, LandSurface Observations
CONDUIT, CRAFT, MADIS, IDD,
NOAAPort, GCMD, SSEC, ESDIS,
NVODS, NCDC
Operational Model Grids
CONDUIT, NOMADS
Data Assimilation Systems
ADAS, WRF 3DVAR
Atmospheric Prediction Systems
WRF, ARPS
Visualization
IDV
Data Mining
ADaM
NSF NMI Project
Globus Tool Kit
Semantic Interchange and Formatting
ESML, NetCDF, HDF5
Adaptive Observing Systems (Radars)
CASA OK Test Bed, V-CHILL
LEAD Portal
NSF NMI Project (OGCE)
Workflow Orchestration
BPEL4WS
Monitoring
Autopilot
Data Cataloging/Management
THREDDS, MCS, SRB
Linked Environments for Atmospheric Discovery
The LEAD Research Process
The End Game: Canonical Research & Education Problems
End User Focus Group Testing and Deployment
Technology Generations
Building Blocks
Basic Research
Prototypes
Test Beds
System Architecture and Definition of Services
Fundamental Scientific and Technological Barriers
System Functional Requirements and Capabilities
Linked
nvironments
for Atmospheric
The E
Driver:
Canonical
ResearchD&iscovery
Education Problems
LEAD Technology Generations
Generation 3
Adaptive
Sensing
Generation 3
Adaptive
Sensing
Generation 2
Dynamic
Workflow
Generation 2
Dynamic
Workflow
Generation 2
Dynamic
Workflow
Generation 1
Static
Workflow
Generation 1
Static
Workflow
Generation 1
Static
Workflow
Generation 1
Static
Workflow
Year 2
Year 3
Technology & Capability
Look-Ahead Research
Look-Ahead Research
Generation 1
Static
Workflow
Year 1
Year 4
Linked Environments for Atmospheric Discovery
Year 5
In LEAD, Everything is a Service
• Finite number of services – they’re the “low-level” elements but
consist of lots of hidden pieces…services within services.
Service A
(ADAS)
Service B
(WRF)
(NEXRAD Stream)
Service D
(MyLEAD)
Service E
(VO Catalog)
Service F
(IDV)
Service G
(Monitoring)
Service H
(Scheduling)
Service I
(ESML)
Service J
(Repository)
Service K
(Ontology)
Service L
(Decoder)
Service C
Many others…
Linked Environments for Atmospheric Discovery
Start by Building Simple Prototypes to
Establish the Services/Other Capabilities…
Service C
(NEXRAD Stream)
Service F
(IDV)
Service L
(Decoder)
Prototype X
Linked Environments for Atmospheric Discovery
Start by Building Simple Prototypes to
Establish the Services/Other Capabilities…
Service C
(NEXRAD Stream)
Service D
(MyLEAD)
Service E
(VO Catalog)
Service F
(IDV)
Service L
(Decoder)
Prototype Y
Linked Environments for Atmospheric Discovery
Start by Building Simple Prototypes to
Establish the Services/Other Capabilities…
Service A
(ADAS)
Service D
(MyLEAD)
Service C
(NEXRAD Stream)
Service E
(VO Catalog)
Service F
(IDV)
Service I
(ESML)
Service L
(Decoder)
Service J
(Repository)
Prototype Z
Linked Environments for Atmospheric Discovery
…and then Solve General Problems
by Linking them Together in Workflows
Service D
(MyLEAD)
Service C
(NEXRAD Stream)
Service L
(Decoder)
Service A
(ADAS)
Service B
(WRF)
Service L
(Mining)
Service J
(Repository)
Linked Environments for Atmospheric Discovery
…and then Solve General Problems
by Linking them Together in Workflows
Service D
(MyLEAD)
Service C
(NEXRAD Stream)
Note that these services
can be used as stand-alone
capabilities, independent of
the LEAD infrastructure
(e.g., portal)
Service L
(Decoder)
Service A
(ADAS)
Service B
(WRF)
Linked Environments for Atmospheric Discovery
Service L
(Mining)
Service J
(Repository)
Linked Environments for Atmospheric Discovery
Linked Environments for Atmospheric Discovery
Linked Environments for Atmospheric Discovery
Linked Environments for Atmospheric Discovery
Linked Environments for Atmospheric Discovery
Feedback from Application Scientists
Benefits
• Single sign-on feature is very handy
• Secured access to compute resources from a
browser, is increasing productivity
Difficulties
• Grid authentication is not trivial to use - important
feature needed by an application scientist
• Hard to keep track of continuously evolving grid
middleware
• System needs continuous development as
middleware on production machines moves forward
and is not backward compatible
Linked Environments for Atmospheric Discovery
Canonical Problem #3
Problem #3: Dynamically Adaptive, High-Resolution
Nested Ensemble Forecasts
Goal: For the continental United States (CONUS),
automatically generate a 1-km grid spacing ADAS analysis
every 30 minutes, and a 6-hour, 2-km grid spacing CONUS
forecast every 3 hours. Automatically launch finer-grid
spacing nested WRF ensemble forecasts when data mining
algorithms – applied to both the CONUS analyses and
forecasts – detect features indicative of storm potential (e.g.,
convergence lines, strong instability, incipient convection) or
actual storm development. Conduct rigorous post-mortem
assessment of statistical forecast skill and compare the highresolution nested grid forecasts with the single-grid CONUS
run at coarser resolution.
Canonical Problem #3
Define Data
Requirements
and Query for
Desired Data
START
Canonical Problem #3
Allocate Storage and
Move/Stream Data to
Appropriate Location
(e.g., PACI Center)
Define Data
Requirements
and Query for
Desired Data
START
Canonical Problem #3
Allocate Computational
Resources
Allocate Storage and
Move/Stream Data to
Appropriate Location
(e.g., PACI Center)
Define Data
Requirements
and Query for
Desired Data
START
Data
Surface Observations
Upper-Air Observations
Commercial Aircraft Data
NEXRAD Radar Data
Satellite Data
Wind Profiler Data
Land Surface Data
Terrain Data
Background Model Fields
and Previous Forecasts
Allocate Computational
Resources
Allocate Storage and
Move/Stream Data to
Appropriate Location
(e.g., PACI Center)
Define Data
Requirements
and Query for
Desired Data
START
Canonical Problem #3
Data
Surface Observations
Upper-Air Observations
Commercial Aircraft Data
NEXRAD Radar Data
Satellite Data
Wind Profiler Data
Land Surface Data
Terrain Data
Background Model Fields
and Previous Forecasts
Allocate Computational
Resources
Allocate Storage and
Move/Stream Data to
Appropriate Location
(e.g., PACI Center)
Define Data
Requirements
and Query for
Desired Data
START
Canonical Problem #3
ESML &
Decoding
Data
Surface Observations
Upper-Air Observations
Commercial Aircraft Data
NEXRAD Radar Data
Satellite Data
Wind Profiler Data
Land Surface Data
Terrain Data
Background Model Fields
and Previous Forecasts
Allocate Computational
Resources
Allocate Storage and
Move/Stream Data to
Appropriate Location
(e.g., PACI Center)
Define Data
Requirements
and Query for
Desired Data
START
Canonical Problem #3
ESML &
Decoding
Remapping,
Gridding,
Conversion
ADAS Quality
Control
Data
Surface Observations
Upper-Air Observations
Commercial Aircraft Data
NEXRAD Radar Data
Satellite Data
Wind Profiler Data
Land Surface Data
Terrain Data
Background Model Fields
and Previous Forecasts
Allocate Computational
Resources
Allocate Storage and
Move/Stream Data to
Appropriate Location
(e.g., PACI Center)
Define Data
Requirements
and Query for
Desired Data
START
Canonical Problem #3
ESML &
Decoding
Remapping,
Gridding,
Conversion
ADAS Quality
Control
ADAS Quality
Control
ADAS
Analysis
Processing
Data
Surface Observations
Upper-Air Observations
Commercial Aircraft Data
NEXRAD Radar Data
Satellite Data
Wind Profiler Data
Land Surface Data
Terrain Data
Background Model Fields
and Previous Forecasts
Allocate Computational
Resources
Allocate Storage and
Move/Stream Data to
Appropriate Location
(e.g., PACI Center)
Define Data
Requirements
and Query for
Desired Data
START
Canonical Problem #3
ESML &
Decoding
Remapping,
Gridding,
Conversion
ADAS Quality
Control
ADAS Quality
Control
ADAS
Analysis
Processing
ADAS
Analysis (3D
Gridded
Fields) +
Background
Fields
Data
Surface Observations
Upper-Air Observations
Commercial Aircraft Data
NEXRAD Radar Data
Satellite Data
Wind Profiler Data
Land Surface Data
Terrain Data
Background Model Fields
and Previous Forecasts
Allocate Computational
Resources
Allocate Storage and
Move/Stream Data to
Appropriate Location
(e.g., PACI Center)
Define Data
Requirements
and Query for
Desired Data
START
Canonical Problem #3
ESML &
Decoding
Remapping,
Gridding,
Conversion
ADAS Quality
Control
ADAS Quality
Control
ADAS-to-WRF
Converter
ADAS
Analysis
Processing
ADAS
Analysis (3D
Gridded
Fields) +
Background
Fields
Data
Surface Observations
Upper-Air Observations
Commercial Aircraft Data
NEXRAD Radar Data
Satellite Data
Wind Profiler Data
Land Surface Data
Terrain Data
Background Model Fields
and Previous Forecasts
Allocate Computational
Resources
Allocate Storage and
Move/Stream Data to
Appropriate Location
(e.g., PACI Center)
Define Data
Requirements
and Query for
Desired Data
START
Canonical Problem #3
ESML &
Decoding
Remapping,
Gridding,
Conversion
ADAS Quality
Control
ADAS Quality
Control
3D Gridded Fields in WRF
Mass Coordinate + Suite
of Ensemble Initial
Conditions
ADAS-to-WRF
Converter
ADAS
Analysis
Processing
ADAS
Analysis (3D
Gridded
Fields) +
Background
Fields
Data
Surface Observations
Upper-Air Observations
Commercial Aircraft Data
NEXRAD Radar Data
Satellite Data
Wind Profiler Data
Land Surface Data
Terrain Data
Background Model Fields
and Previous Forecasts
Allocate Computational
Resources
Allocate Storage and
Move/Stream Data to
Appropriate Location
(e.g., PACI Center)
Define Data
Requirements
and Query for
Desired Data
START
Canonical Problem #3
ESML &
Decoding
Remapping,
Gridding,
Conversion
ADAS Quality
Control
ADAS Quality
Control
3D Gridded Fields in WRF
Mass Coordinate + Suite
of Ensemble Initial
Conditions
Multiple Copies
of WRF Forecast
Model Running
Simultaneously
ADAS-to-WRF
Converter
ADAS
Analysis
Processing
ADAS
Analysis (3D
Gridded
Fields) +
Background
Fields
Data
Surface Observations
Upper-Air Observations
Commercial Aircraft Data
NEXRAD Radar Data
Satellite Data
Wind Profiler Data
Land Surface Data
Terrain Data
Background Model Fields
and Previous Forecasts
Allocate Computational
Resources
Allocate Storage and
Move/Stream Data to
Appropriate Location
(e.g., PACI Center)
Define Data
Requirements
and Query for
Desired Data
START
Canonical Problem #3
ESML &
Decoding
Remapping,
Gridding,
Conversion
ADAS Quality
Control
ADAS Quality
Control
3D Gridded Fields in WRF
Mass Coordinate + Suite
of Ensemble Initial
Conditions
Multiple Copies
of WRF Forecast
Model Running
Simultaneously
ADAS-to-WRF
Converter
WRF Gridded
Output
ADAS
Analysis
Processing
ADAS
Analysis (3D
Gridded
Fields) +
Background
Fields
Data
Surface Observations
Upper-Air Observations
Commercial Aircraft Data
NEXRAD Radar Data
Satellite Data
Wind Profiler Data
Land Surface Data
Terrain Data
Background Model Fields
and Previous Forecasts
Allocate Computational
Resources
Allocate Storage and
Move/Stream Data to
Appropriate Location
(e.g., PACI Center)
Define Data
Requirements
and Query for
Desired Data
START
Canonical Problem #3
ESML &
Decoding
Remapping,
Gridding,
Conversion
ADAS Quality
Control
ADAS Quality
Control
3D Gridded Fields in WRF
Mass Coordinate + Suite
of Ensemble Initial
Conditions
Multiple Copies
of WRF Forecast
Model Running
Simultaneously
ADAS-to-WRF
Converter
WRF Gridded
Output
ADAS
Analysis
Processing
ADAS
Analysis (3D
Gridded
Fields) +
Background
Fields
myLEAD
Storage
Data
Surface Observations
Upper-Air Observations
Commercial Aircraft Data
NEXRAD Radar Data
Satellite Data
Wind Profiler Data
Land Surface Data
Terrain Data
Background Model Fields
and Previous Forecasts
Allocate Computational
Resources
Allocate Storage and
Move/Stream Data to
Appropriate Location
(e.g., PACI Center)
Define Data
Requirements
and Query for
Desired Data
START
Canonical Problem #3
ESML &
Decoding
Remapping,
Gridding,
Conversion
ADAS Quality
Control
ADAS Quality
Control
3D Gridded Fields in WRF
Mass Coordinate + Suite
of Ensemble Initial
Conditions
Multiple Copies
of WRF Forecast
Model Running
Simultaneously
ADAS-to-WRF
Converter
WRF Gridded
Output
Meta Data
Creation and
Cataloging
ADAS
Analysis
Processing
ADAS
Analysis (3D
Gridded
Fields) +
Background
Fields
myLEAD
Storage
Data
Surface Observations
Upper-Air Observations
Commercial Aircraft Data
NEXRAD Radar Data
Satellite Data
Wind Profiler Data
Land Surface Data
Terrain Data
Background Model Fields
and Previous Forecasts
Allocate Computational
Resources
Allocate Storage and
Move/Stream Data to
Appropriate Location
(e.g., PACI Center)
Canonical Problem #3
ESML &
Decoding
Remapping,
Gridding,
Conversion
ADAS Quality
Control
ADAS Quality
Control
3D Gridded Fields in WRF
Mass Coordinate + Suite
of Ensemble Initial
Conditions
Multiple Copies
of WRF Forecast
Model Running
Simultaneously
ADAS-to-WRF
Converter
WRF Gridded
Output
Meta Data
Creation and
Cataloging
Define Data
Requirements
and Query for
Desired Data
START
Visualization &
Data Mining
ADAS
Analysis
Processing
ADAS
Analysis (3D
Gridded
Fields) +
Background
Fields
myLEAD
Storage
Data
Surface Observations
Upper-Air Observations
Commercial Aircraft Data
NEXRAD Radar Data
Satellite Data
Wind Profiler Data
Land Surface Data
Terrain Data
Background Model Fields
and Previous Forecasts
Allocate Computational
Resources
Allocate Storage and
Move/Stream Data to
Appropriate Location
(e.g., PACI Center)
Canonical Problem #3
ESML &
Decoding
Remapping,
Gridding,
Conversion
ADAS Quality
Control
ADAS Quality
Control
3D Gridded Fields in WRF
Mass Coordinate + Suite
of Ensemble Initial
Conditions
Multiple Copies
of WRF Forecast
Model Running
Simultaneously
ADAS-to-WRF
Converter
WRF Gridded
Output
START
Visualization &
Data Mining
ADAS
Analysis (3D
Gridded
Fields) +
Background
Fields
myLEAD
Storage
Meta Data
Creation and
Cataloging
Define Data
Requirements
and Query for
Desired Data
ADAS
Analysis
Processing
STOP
Data
Surface Observations
Upper-Air Observations
Commercial Aircraft Data
NEXRAD Radar Data
Satellite Data
Wind Profiler Data
Land Surface Data
Terrain Data
Background Model Fields
and Previous Forecasts
Allocate Computational
Resources
Allocate Storage and
Move/Stream Data to
Appropriate Location
(e.g., PACI Center)
Canonical Problem #3
ESML &
Decoding
Remapping,
Gridding,
Conversion
ADAS Quality
Control
ADAS Quality
Control
3D Gridded Fields in WRF
Mass Coordinate + Suite
of Ensemble Initial
Conditions
Multiple Copies
of WRF Forecast
Model Running
Simultaneously
ADAS-to-WRF
Converter
WRF Gridded
Output
START
Visualization &
Data Mining
ADAS
Analysis (3D
Gridded
Fields) +
Background
Fields
myLEAD
Storage
Meta Data
Creation and
Cataloging
Define Data
Requirements
and Query for
Desired Data
ADAS
Analysis
Processing
STOP
Data
Surface Observations
Upper-Air Observations
Commercial Aircraft Data
NEXRAD Radar Data
Satellite Data
Wind Profiler Data
Land Surface Data
Terrain Data
Background Model Fields
and Previous Forecasts
Canonical Problem #3
ESML &
Decoding
Allocate Storage and
Move/Stream Data to
Appropriate Location
(e.g., PACI Center)
START
Multiple Copies
of WRF Forecast
Model Running
Simultaneously
Adjust Forecast
Configuration and
Schedule Resources
ADAS Quality
Control
ADAS Quality
Control
3D Gridded Fields in WRF
Mass Coordinate + Suite
of Ensemble Initial
Conditions
Allocate Computational
Resources
Define Data
Requirements
and Query for
Desired Data
Remapping,
Gridding,
Conversion
ADAS-to-WRF
Converter
WRF Gridded
Output
ADAS
Analysis (3D
Gridded
Fields) +
Background
Fields
myLEAD
Storage
Meta Data
Creation and
Cataloging
Visualization &
Data Mining
ADAS
Analysis
Processing
STOP
Data
Surface Observations
Upper-Air Observations
Commercial Aircraft Data
NEXRAD Radar Data
Satellite Data
Wind Profiler Data
Land Surface Data
Terrain Data
Background Model Fields
and Previous Forecasts
Canonical Problem #3
ESML &
Decoding
Allocate Storage and
Move/Stream Data to
Appropriate Location
(e.g., PACI Center)
START
Multiple Copies
of WRF Forecast
Model Running
Simultaneously
Adjust Forecast
Configuration and
Schedule Resources
ADAS Quality
Control
ADAS Quality
Control
3D Gridded Fields in WRF
Mass Coordinate + Suite
of Ensemble Initial
Conditions
Allocate Computational
Resources
Define Data
Requirements
and Query for
Desired Data
Remapping,
Gridding,
Conversion
ADAS-to-WRF
Converter
WRF Gridded
Output
ADAS
Analysis (3D
Gridded
Fields) +
Background
Fields
myLEAD
Storage
Meta Data
Creation and
Cataloging
Visualization &
Data Mining
ADAS
Analysis
Processing
STOP
How Would One Go
About Setting This
Up in LEAD??
• The “First LEAD Commandment”
– Thou shalt not use unintelligible computer science jargon
in the portal for describing options/tasks to end users
– Foo, portlet, ontology, widget, daemon, worm, hash…
Linked Environments for Atmospheric Discovery
Linked Environments for Atmospheric Discovery
Linked Environments for Atmospheric Discovery
Data Environment
Select/Search for
Data
Select Region of Interest
Linked Environments for Atmospheric Discovery
Linked Environments for Atmospheric Discovery
Tools Environment
Select Tools
IDV Visualizer
ADAS Assimilator
WRF Predictor
ADaM Data Miner
Decoders
Linked Environments for Atmospheric Discovery
Linked Environments for Atmospheric Discovery
Experiments Environment
new
load saved
Linked Environments for Atmospheric Discovery
Grid Resources Environment
Select Resource
Linked Environments for Atmospheric Discovery
Data
Surface Observations
Upper-Air Observations
Commercial Aircraft Data
NEXRAD Radar Data
Satellite Data
Wind Profiler Data
Land Surface Data
Terrain Data
Background Model Fields
and Previous Forecasts
Canonical Problem #3
ESML &
Decoding
Allocate Storage and
Move/Stream Data to
Appropriate Location
(e.g., PACI Center)
START
Multiple Copies
of WRF Forecast
Model Running
Simultaneously
Adjust Forecast
Configuration and
Schedule Resources
ADAS Quality
Control
ADAS Quality
Control
3D Gridded Fields in WRF
Mass Coordinate + Suite
of Ensemble Initial
Conditions
Allocate Computational
Resources
Define Data
Requirements
and Query for
Desired Data
Remapping,
Gridding,
Conversion
ADAS-to-WRF
Converter
WRF Gridded
Output
ADAS
Analysis (3D
Gridded
Fields) +
Background
Fields
myLEAD
Storage
Meta Data
Creation and
Cataloging
Visualization &
Data Mining
ADAS
Analysis
Processing
STOP