Transcript Class21_AnalysisWebServicesx

Using Python to Interact with
the EPA WATERS Web Services
Jon Goodall
Hydroinformatics
Fall 2014
This work was funded by National Science
Foundation Grants EPS 1135482 and EPS
1208732
EPA Waters
• “The Watershed Assessment, Tracking &
Environmental Results
System (WATERS) unites water quality
information previously available only from
several independent and unconnected
databases.”
• http://water.epa.gov/scitech/datait/tools/wat
ers/
WATERS Web Services
• Description: “WATERS provides a suite of
interoperable services that expose
components that perform complex analysis
and supporting strategic datasets, such as
NHD, NHDPlus, and WBD.”
• Source:
http://water.epa.gov/scitech/datait/tools/wat
ers/services/index.cfm
Web and Database Services
• The WATERS Web and Database services
provide open interfaces to complex analyses.
• These services make extensive use of the
National Hydrography Dataset (NHD)
• Designed as loosely coupled modular units,
the services are developed in a common
architecture.
• The HTTP Services provided REST-like access
to the WATERS database services.
Source: http://water.epa.gov/scitech/datait/tools/waters/services/index.cfm
NHD = National Hydrography Dataset
• http://nhd.usgs.gov
REST
• REST = Representational State Transfer
• You can think of this as an alternative to SOAP that we
discussed and you used previously in this course.
• Most people seem to prefer REST because you can
simply enter a URL into a web browser and see the
response.
• The response is often structured text that uses either
XML or JSON.
• Wikipedia page has further details if you are
interested:
http://en.wikipedia.org/wiki/Representational_state_tr
ansfer
What is JSON?
• JSON = JavaScript Object Notation
• A way of structuring data in a machine-readable
way.
• An alternative to XML
• Thread on Stackoverflow.com on XML vs JSON:
http://stackoverflow.com/questions/3536893/wh
at-are-the-pros-and-cons-of-xml-and-json
• You can read more details on the Wikipedia page:
http://en.wikipedia.org/wiki/JSON
Back to the EPA WATERS Web
Services…
• You can find a list of the available services
with documentation here:
http://water.epa.gov/scitech/datait/tools/wat
ers/services/index.cfm
• We will use the Name Service in today’s class
as an example:
http://water.epa.gov/scitech/datait/tools/wat
ers/services/name_service.cfm
Name Service
• Name service documentation is here:
http://water.epa.gov/scitech/datait/tools/wat
ers/services/name_service.cfm
• Brief description: “Advanced query function to
find water feature names and provide general
location information for NHDPlus features.”
How do you use these services?
• Some example use cases of the WATERS services are
here: http://codepen.io/WATERS_SUPPORT/public/
• I used Firebug, a web development plug-in to Firefox,
to see the web service calls used by the example
software (see screen shot on next page and I will
demonstrate in class). http://getfirebug.com/
• I was told, but did not test myself, that the Chrome
DevTools could also be used for this purpose.
https://developer.chrome.com/devtools
The URL for the service
• http://ofmpub.epa.gov/waters10/Name.Service?pFullT
ext=James+River&pFullTextRegex=&pBasename=&pBa
senameRegex=&pHydrography=&pHydrographyRegex=
&pDirectional=&pDirectionalRegex=&pOperator=EQ&p
QueryLimit=&pJWThreshold=90&pResolution=3&pSou
rceTable=nhdflowline&pState=VA&pStateMod=%2C&p
CountyFips5=&pCountyFips5Mod=%2C&pSubbasin=&p
SubbasinMod=%2C&pGnisClass=&pGnisClassMod=%2C
&pFtype=&pBreakBySubbasin=false&pBreakByFcode=f
alse&optNHDPlusDataset=2.1&optCache=1415283785
917&optJSONPCallback=success
Response from Web Service
success({ "output":{ "feature_id":null,"results":[{
"styleUrl":"#.namesrv_results","feature_id":"1488853","feature_name":"JAM
ES
RIVER","feature_class":"STREAM","primary_state":"VA","primary_state_fips":
"51","primary_county":"ISLE OF
WIGHT","primary_county_fips":"093","gnis_primary_stem":"JAMES","gnis_pr
imary_hydro":"RIVER","gnis_primary_dir":null,"gnis_secondary_stem":null,"g
nis_secondary_hydro":null,"gnis_secondary_dir":null,"resolution":3,"source_
schema":"NHDPLUS","source_table":"NHDFLOWLINE_NP21","subbasin":null,
"ftype":null,"fcode":null,"gnis_centroid_geom":{"type":"Point","coordinates":
[-76.4435588,36.9415369]},"mbr_geom":{"type":"Polygon","coordinates":[[[79.8541445197531,36.9403890091464],[76.3481893915265,36.9403890091464],[76.3481893915265,37.7966000744348],[79.8541445197531,37.7966000744348],[79.8541445197531,36.9403890091464]]]},"shape":null}]},"status":{
"submission_id":null,"status_code":0,"status_message":null,"execution_time
":0.129432,"output_bytes":853}})
Comments on URL and response
• In the URL you will see
– pFullText=James+River
– pSourceTable=nhdflowline
– pState=VA
• These are the search criteria I entered in the CodePen interface
• In the response you will see
– "gnis_centroid_geom":{"type":"Point","coordinates":[76.4435588,36.9415369]},
• “Geometry representing the centroid of the result record as defined by GNIS.”
– "mbr_geom":{"type":"Polygon","coordinates":[[[79.8541445197531,36.9403890091464],[76.3481893915265,36.9403890091464],[76.3481893915265,37.7966000744348],[79.8541445197531,37.7966000744348],[79.8541445197531,36.9403890091464]]]},"shape":null}]},
• “Geometry representing the maximum bounding rectangle of the matching NHD feature.
Note that in the case that the matching NHD record is a single point, that point will be
stored in this field.”
Testing the Result using Google Maps
Note that you need to flip the coordinates provided by the service when you input them into Google maps
Challenge Problem #1
• Change the URL provided earlier to find a
water feature of interest to you.
• Enter the coordinates returned by the service
into Google Maps to verify that you got what
you expected.
Repeat Using Python
• Suppose you wanted to create a “search tool”
capable of geolocating an NHD feature given its
name and state.
• We have already shown how to get the response
from a URL using Python
– import urllib2
– response = urllib2.urlopen(<URL String Here>).read()
• There are JSON libraries in Python we can use to
extract the lat/lon coordinates of the centroid
point
– https://docs.python.org/2/library/json.html
Challenge Problem #2
• Write a Python script that returns the centroid
coordinates for an NHD feature given its name
and location.
• My solution is in the class Github site:
https://github.com/goodalljl/hydroinformatics
_class
What you learned today…
• You were introduced to REST
– Can you describe the differences between REST
and SOAP?
• You were introduced to JSON
– Can you describe the differences between JSON
and XML?
• You were introduced to the concept of using
Web services for analytics vs. simply providing
data
Plan for Thursday’s Class
• We will look at other services in the EPA WATER
Web Services
• I will discuss your assignment that will ask you to
further evaluate and build on the EPA WATER
Web Services.
– The USGS/EPA team behind these want to hear your
opinion! With your permission, we will share your
assignment solution with the NHD development team.