Transcript WATERS Network - Department of Civil, Architectural and

The WATERS (WATer and
Environmental Research Systems
Network) Network: A Joint
CLEANER and CUAHSI Venture
Barbara Minsker, U of Illinois, Urbana, IL
David Maidment, U of Texas, Austin, TX
April 9, 2016
The Need…and Why Now?
Nothing is more fundamental to life than water. Not only is water
a basic need, but adequate safe water underpins the nation’s
health, economy, security, and ecology.
NRC (2004) Confronting the nation’s water problems: the role of research.
● Water use globally will triple in the next two decades, leading to increases in
erosion, pollution, dewatering, and salinization.
● Major U.S. aquifers (e.g., the Ogallala) are being mined and the resource
consumed.
● Only ~55% of the nation’s river and stream miles and acres of lakes and
estuaries fully meet their intended uses; ~45% are polluted, mostly from
diffuse-source runoff.
● From 1990 through 1997, floods caused more than $34 billion in damages in
the U.S.
● Of 45,000 U.S. wells tested for pesticides, 5,500 had harmful levels of at
least one.
● Fish consumption advisories are common in more than 30 states because of
elevated mercury levels (source: mostly fossil fuel combustion; mercury is a
neurotoxin).
WATERS Network Grand Challenge from the
November Joint CUAHSI/CLEANER
Workshop
How are water quantity, quality, and related earth system
processes affected by natural and human-induced
changes to the environment?
• How do we detect and predict the effects of natural phenomena
and human activities on the quantity, distribution, and quality
of water across a range of scales?
• How do we manage, engineer, and adapt to aspects of the urban
water cycle to achieve sustainable water use and availability
for humans and ecosystems?
• How do hydrologic, biologic, geomorphic and chemical
transformations in the atmosphere, surface and subsurface
affect water quality over multiple space and time scales?
CLEANER Grand Challenge Questions
(from All-Hands Meeting, Sept. 20-21, 2005)
1. How do we detect and
predict waterborne
hazards in real time?
2. How do we predict the
effects of human
activities on the
quantity, distribution,
and quality of water?
3. How do we improve
water cycle engineering
management strategies
to provide water
quantity and quality to
sustain humans and
ecosystems?
WATERS Network Infrastructure
• These questions cannot be adequately
addressed without observatory infrastructure:
– Network of field sites (15-20 nationwide)
• Large spatial scale to study complex environmental systems
• Equipped with state-of-the-art sensors and other
instrumentation
• Technical staff to help with experiments and IT
– Linked to national community via cyberinfrastructure
• Computer hardware, networks, and software
• Creates a “collaboratory” for interdisciplinary
teams in different universities to collaborate on
large-scale research
WATERS Network Infrastructure
domainDNS
Disk array
OBSERVATORIES
& Environmental Field
Facilities
Investigators
CYBERINFRASTRUCTURE
& Modeling
Education & Outreach
MEASUREMENT FACILITY
& Sensor Development
Science & Applications
Communities
SYNTHESIS
Q( x ) 
 r ( x )dx
CA
The WATERS Network will feature:
(a) sites with gradients across the range of human impacts
(b) where possible, co-location of minimally impacted sites with other EO field sites
(c) nested watersheds ranging from local catchments to major river basins to
improve understanding of environmental processes across scales
WATERS Network CI Planning
• CLEANER Project Office
(http://cleaner.ncsa.uiuc.edu; Minsker PI, Jerry
Schnoor & Chuck Haas co-PIs)
– Cyberinfrastructure Committee (Chairs Jeanne Van
Briesen & Tom Finholt) is creating a CI program plan
in collaboration with CUAHSI
• Two groups are creating CI demonstrations for
WATERS Network:
– CUAHSI Hydrologic Information System project
(Maidment, PI)
– NCSA Environmental CI Demonstration (ECID) project
(Minsker and Jim Myers co-leads)
– We have proposed a draft common environmental CI
architecture
Environmental CI Architecture:
Research Services
Integrated CI
Knowledge
Services
Data
Services
ECID Project Focus
Supporting Technology
Workflows
& Model
Services
MetaWorkflows
Collaboration
Services
Digital
Library
HIS Project Focus
Create
Hypothesis
Obtain
Data
Analyze
Data &/or
Assimilate
into
Model(s)
Link &/or
Run
Analyses
&/or
Model(s)
Discuss
Results
Publish
Research
Process
Knowledge Services
• Help users find information they need quickly
•
and effectively
Includes information from:
– Archives of community documents, data, workflows,
models, collaboration transcripts, etc.
– Metadata, including automatically generated
metadata collected from other users’ activities and
relationships among the metadata (“knowledge
networks”)
– Web crawls to create dynamic databases on web sites
& topics of interest to the community
Knowledge Services (cont’d.)
• Includes both “information push” and “pull”
– Information push – make referrals to users based on
their interests and preferences
– Information pull – standard user searches
• To provide comprehensive knowledge services,
provenance of all user activities must be stored in
metadata
– History and origin of all products (data, workflows,
documents, etc.) stored in a flexible and expandable
metadata scheme
Knowledge Services: ECID
Technology Development
• Metadata harvesting from
all CI activities enables
comprehensive knowledge
services
– Using RDF & Kowari to log
“provenance” (source & links)
of all objects in “triples”
(subject, object, property)
• E.g., graph shows that data2
came from data1 using a
meta-workflow
Knowledge Services: ECID
Technology Development
• CI-KNOW (CI Knowledge Networks on the Web)
uses metadata to make referrals to users
– Built with social networking algorithms (Contractor)
Knowledge
Network
searches for
Jose
au
t
ho
Topic C
author
s
ta
in
co
n
Robert
author
Document B
co
n
ta
in
r
s
Topic A
Sophia
contains
Document C
contains
in
ta
author
n
co
s
Topic B
Document A
author
James
Environmental CI Architecture:
Research Services
Integrated CI
Knowledge
Services
Create
Hypothesis
Data
Services
Workflows
& Model
Services
Obtain
Data
Analyze
Data &/or
Assimilate
into
Model(s)
Supporting Technology
MetaWorkflows
Link &/or
Run
Analyses
&/or
Model(s)
Collaboration
Services
Discuss
Results
Digital
Library
Publish
Research
Process
CUAHSI Hydrologic Information System
A multiscale web portal system for accessing, querying, visualizing, and publishing
water observation data and models for any location or region in the United States
Multiscale data delivery
1:1,000,000 scale
North American Scale
(e.g. North American
Regional Reanalysis of climate)
1:500,000 scale
Continental US Scale
(coast to coast data
coverage, HIS-USA)
1:100,000 scale
Regional Scale
(e.g. Neuse basin)
1:24,000 scale
Watershed Scale
(e.g. Eno watershed )
Site scale
Site Scale
(experimental site level)
Point
Point Observation Scale
(gage, sampling location)
Observatories
LTER
Ameriflux
NCAR
NCDC
Storet
NWIS
CUAHSI Web Services
Excel
Visual Basic
ArcGIS
C/C++
Matlab
Fortran
Access
SAS
Some operational services
CUAHSI Web Services
Web application: Data Portal
Your application
• Excel, ArcGIS, Matlab
• Fortran, C/C++, Visual Basic
• Hydrologic model
• …………….
Your operating system
• Windows, Unix, Linux, Mac
Internet
Web Services
Library
Simple
Object
Access
Protocol
Direct and Indirect Web Services
• Direct web service
– The data agency provides direct querying
ability into its archives through SOAP or
OpenDAP (NCDC)
• Indirect web service
– CUAHSI constructs a “web page mimic”
service, housed at SDSC, that
programmatically mimics the manual use of
an agency’s web pages (USGS, Ameriflux)
Observation Site Files
Ameriflux Towers
USGS NWIS Stations
Automated Surface Observing System
Climate Reference Network
Observation Site Map for US
USGS NWIS
ASOS
Climate Research Network
Ameriflux
+ others…….
Neuse Basin with all points
NWIS Streamflow and Water Quality
ASOS
NWIS Groundwater
NARR
Ameriflux
Filtered Site Map
NWIS Streamflow and
Water Quality
ASOS
NARR
Ameriflux
HydroObjects Library
• CUAHSI has
developed a
HydroObjects
Library with web
service wrappers
that know where to
access each web
service and how to
interpret its output
User Application
(Excel, ArcGIS, …..)
HydroObjects Library
with web service wrappers
for NWIS, Ameriflux, NCDC, …
Direct or Indirect
web services
Web data
Transfer of research results
• CUAHSI web services for NWIS were announced at a
•
cyberseminar on Friday Oct 28
On Wednesday Nov 2, Jason Love, from a private
firm, RESPEC, in Sioux Falls, South Dakota, posted on
the EPA Basins list server: “Occasionally one comes across
something that is worth sharing; the CUAHSI Hydrologic Information
Systems - Web Services Library for NWIS is a valuable tool for those
of us interested in rapidly acquiring and processing data from the
USGS, e.g., calibrating models and performing watershed
assessments.”
• He provided a tutorial on how to use the services
•
from Matlab (which CUAHSI had not developed)
Technology transfer took less than 1 week!
Environmental CI Architecture:
Research Services
Integrated CI
Knowledge
Services
Create
Hypothesis
Data
Services
Workflows
& Model
Services
Obtain
Data
Analyze
Data &/or
Assimilate
into
Model(s)
Supporting Technology
MetaWorkflows
Link &/or
Run
Analyses
&/or
Model(s)
Collaboration
Services
Discuss
Results
Digital
Library
Publish
Research
Process
Series and Fields
Features
Series – ordered sequence of numbers
Point, line, area, volume
Discrete space representation
Surfaces
Time series – indexed by time
Frequency series – indexed by frequency
Fields – multidimensional arrays
Continuous space representation
Scalar fields – single value at each location
Vector fields – magnitude and direction
Tensor fields – several vectors
Random fields – probability distribution
Digital Watershed
Hydrologic
Observation
Series
Geospatial
Data
Digital
Watershed
Remote Sensing
Fields
Weather and Climate
Fields
A digital watershed is an overlay of
observation series and fields on a geospatial framework
to form a connected database for a hydrologic region
ArcGIS ModelBuilder Application for Automated Water Balancing
Fields
Series
Geospatial
Environmental CI Architecture:
Research Services
Integrated CI
Knowledge
Services
Create
Hypothesis
Data
Services
Workflows
& Model
Services
Obtain
Data
Analyze
Data &/or
Assimilate
into
Model(s)
Supporting Technology
MetaWorkflows
Link &/or
Run
Analyses
&/or
Model(s)
Collaboration
Services
Discuss
Results
Digital
Library
Publish
Research
Process
Meta-Workflow
• Many of the observatory efforts involve:
– Studying complex environmental systems that require
coupling analyses or models of different components
of the systems, often created by different people,
• E.g., water flows drive contaminant transport in multiple
media, and both affect ecological flora and fauna
– Real-time, automated updating of analyses and
modeling that required diverse tools
• E.g., spreadsheets, scripts, GIS tools, models
• “Meta-workflow” tools enable heterogeneous
workflows to be coupled and run on the
desktop, on remote servers, or across the
Computational Grid
CyberIntegrator: ECID Project’s MetaWorkflow Tool
Environmental CI Architecture:
Research Services
Integrated CI
Knowledge
Services
Create
Hypothesis
Data
Services
Workflows
& Model
Services
Obtain
Data
Analyze
Data &/or
Assimilate
into
Model(s)
Supporting Technology
MetaWorkflows
Link &/or
Run
Analyses
&/or
Model(s)
Collaboration
Services
Discuss
Results
Digital
Library
Publish
Research
Process
Collaboration Services
• Environmental observatories will require:
– Tools for easy remote communication, such as:
• Wikis for collaborative editing (integrated with word
processors)
• Instant messenger and chat rooms
• Voice over IP and videoconference connections
• Screen sharing
• Numerous technologies exist, but are not:
– Easily integrated with data and analyses for technical
discussions
– Scalable to large groups across different platforms
– Integrated with knowledge services to support
collaborative knowledge sharing within & across
communities
CyberCollaboratory is the ECID
project’s prototype collaboration
service. To check it out, create an
account at
http://cleaner.ncsa.uiuc.edu/cyber
collab
A New Paradigm for Research
• WATERS Network will create a new paradigm for
environmental research
– Shared infrastructure at large scales
– Interdisciplinary teams collaborating remotely to
address complex environmental issues
• New paradigm will enable improved
•
understanding & management of large-scale
natural environmental systems
Cyberinfrastructure can create a nationwide
knowledge network for environmental
researchers
– Supports all research & education, regardless of their
focus
Acknowledgments
• Contributors:
– CLEANER project office & planning grant
teams
– NCSA ECID team
– CUAHSI HIS team
• Funding sources:
– NSF grants BES-0414259, BES-0533513, and
SCI-0525308, EAR-0413265
– Office of Naval Research grant N00014-04-10437