Transcript OpenWater_Data_Flow_024

What is CCAMP OpenWater?
A Prototype Internet based Open Source Software Toolkit
Focused on water quality and quantity assessment and visualization.
“Make everything as simple as possible, but not simpler” (Albert Einstein)
"If I can't picture it, I can't understand it." (Albert Einstein)
CCAMP OpenWater is a software toolkit intended to run in a multi-server cloud environment
that provides real-time and scheduled execution data mining ,data grooming, statistical analysis,
stochastic, deterministic, and hueristic reasoning, and data visualization tools.
CCAMP OpenWater is a pile of software gizmos
that might be useful beyond the Central Coast.
Why Open Source?
Cross platform Open Source Code provides numerous benefits.
• Reduces system development failure risk
• Provides access to an international community of developers and
international code standards
• Empowers agency staff, agency users, and development partners
• Avoids pre-committing the State and others to licensing agreements
with sole source commercial vendors.
Notes: “Beware of Geeks bearing gifts.” (Virgil 19 BC)
1.
Commercial software vendors work on the leading edge of technology to
provide features no other company has, in order to market their products.
2.
The ‘leading edge’ can become the ‘bleeding edge’.
CCAMP Open Source Framework
Apache:
PHP:
MySQL:
MediaWiki:
R:
PostGreSQL:
PostGIS:
FileZilla:
GeoServer:
OpenLayers:
JPGraph:
PHPExcel:
Basic web server
Automates web and database communication
Generalized database management system
Collaborative Development and Documentation
Statistical library and graphics support
Generalized database management system
Automates Geo-Spatial processing
File transfer (ftp) support
Web mapping support
Web mapping support
Web graphing support
File transfers to and from Excel
SWAMP
Data Flow Example
CEDEN
WQX
CCAMP App Staging
Data Tables
Web Views
GeoTracker
DPR PUR
And other
sources
The system is intended to use
routine automated queries to
keep data up-to-date.
Water Quality Data sources
•
•
•
•
•
•
•
•
Moss Landing Data Warehouse
SWRCB CEDEN Data Warehouse
WQX USEPA WQ Data Exchange Network
USGS National Water Information System
GeoTracker
National Hydrography Dataset Plus
Grant data via CalDucs
Numerous other sources
Data Grooming
•
•
•
•
•
•
•
Synonym Dictionaries
Analyte name standardization
Units of measurement standardization
Matrix standardization
QA data filtering
Handling of duplicates
Handling of probe data
GeoSpatial processing and linking
• Automated linking of monitoring sites to GIS layers
• Handling GIS layer idiosyncrasies
• Linking of land use and other datasets to sites
• Pesticide use characterization
• Land Cover characterization
• Flow and Load estimation
Geospatial Framework
• USGS National Hydrography Dataset Plus
• USGS National Hydrography Dataset
• National Watershed Boundary Dataset
• National Land Cover Dataset
• Public Land Survey System Boundaries
• Bulletin 118 Groundwater Basin Boundaries
• California Healthy Watersheds (CADMUS)
Statistics and other calculations
•
•
•
•
•
•
Calculating Exceedances
Calculating Magnitude Exceedance Quotients-MEQ
Summary Statistics
Change and Trend detection
Various bivariate and multivariate analysis tools
Data Scanner for 303(d) - 305(b)
CWA 303(d)-305(b) Data Processing
Water Quality Goal Scanning System
•
Compile goals using J. Marshack’s compilation as the baseline
Extend goal list to include:
USEPA OPP Benchmarks
NOAA Screening Quick Reference Tables (SQuiRTs)
Regional Basin Plan Goals
Waterbody Specific Goals
Site Specific Goals
Numeric Threshold Goals that implement Narrative Objectives
TMDL compliance point Goals
Link WQ Goals to Waterbodies
• Establish spatial areas of applicability of goals
– Provide web based table upload for updates
– Integrate Jeff K’s GIS mapping project
Scan Water Quality Data
• Monitoring site scale scanning for Data Navigators,
Healthy Watersheds, and TMDLs
• Waterbody scale scanning for CalWQA and ELEP
• Catchment scale scanning for
– USEPA ATTAINS
– Healthy Watersheds
– Recovery Potential