Transcript DAP Servers and Services

DAP Servers and Services
Section 2
APAC ‘07 OPeNDAP Workshop
12 Oct 2007
James Gallagher
Thanks to Jennifer Adams, John Caron,
Roberto De Almeida, Nathan Potter and Patrick West
Outline
• Different DAP Servers
• Hyrax Architecture and customization
• Example Configuration of Hyrax
There are several different
DAP servers
•
•
•
•
•
Hyrax, developed by OPeNDAP
TDS, developed by Unidata
GDS, developed by COLA
PyDAP, developed by Roberto De Almeida
Others we won’t cover: Dapper; FDS;
CODAR; OPeNDAP Server3; et cetera
• Terminology: We often say ‘server’ when
describing what the technical press calls a
‘web application.’
Hyrax
• Supports multiple protocols
– Data: DAP using HTTP/GET and HTTP/SOAP;
Direct access (via HTTP); WCS/WFS funded, in
development
– Catalog: THREDDS; HTML directories
• Data formats: In binary distribution: NetCDF;
HDF4; HDF5; FreeForm; many more available as
source code.
• Includes ASCII data dump, HTML data access form,
Info metadata page
Hyrax Architecture
OLFS
BES
• Two (or more) cooperating processes:
– Front-end provides DAP interface
– Back-end(s) read(s) data
• Both parts can be customized
– Front-end: different network protocols
– Back-end: different data formats/systems
• N-Tier design is flexible, secure
Data
TDS
• Java Servlet network interface
• Supports multiple protocols
– Data: DAP; WCS; NetCDF Subset; Direct access
(via HTTP)
– Catalog: THREDDS
• Data formats: NetCDF; HDF5; GRIB-1,2;
NEXRAD; DORADE; BUFR; DMSP; GINI; more in
development
• Can also read from any other DAP server
• Can serve aggregations
THREDDS Data Server
HTTP Tomcat Server
catalog.xml
THREDDS Server
•WCS
•OPeNDAP
•HTTPServer
•NetcdfSubset
NetCDF-Java
library
configCatalog.xml
IDD Data
Datasets
motherlode.ucar.edu
Application
PyDAP
• Python native implementation from the DAP2
specification; WSGI application
• Server uses run-time loaded modules
• Protocols:
– Data: DAP (HTTP/GET); WMS/KML;
EditGrid/Google Spreadsheets; JSON
– Catalog: THREDDS using a script/batch system;
full support planned
• Formats: CSV; netCDF; SQL Matlab; GRIB;
HDF5
PyDAP Server Architecture:
Plugins and Responses
GDS
• Java Servlet network interface and C/C++ back-end
• Supports multiple protocols:
– Data: DAP (HTTP/GET)
– Catalogs: THREDDS
• Formats: GRIB; HDF4; netCDF; BUFR; Binary;
GrADS Station
• Can also read from many DAP servers
• GDS can perform sophisticated server-side
operations and build interim data sets as a result.
• Server-side operations use GrADS command syntax.
GrADS Data Server
Server
datasets in any format
supported by GrADS
performs
analysis
operations
Binary
GRIB, HDF
NetCDF
Result
cache
holds temporary data
(uploaded, generated by a
previous operation, or
transferred directly from
another server) for use in
remote analysis
GrADS
batch
mode
manages sessions,
translates dataset
names
interface
code
supports extended request types for
analysis, upload
DODS server
libraries
Java
servlet
DAP Servers Summary
• Four servers, all developed by different
groups, were described (Hyrax, TDS, PyDAP
and GDS)
• Each supports DAP and THREDDS
• Each has unique features:
–
–
–
–
PyDAP: WMS, KML and python scripting
GDS: Extensive server-side operations
TDS: Aggregation, WCS
Hyrax: Customizable, SOAP interface, WCS/WFS
support funded
• These are more properly called ‘web
applications.’
Hyrax Architecture
•
•
•
•
•
Front-end (OLFS) customization
Authentication & Authorization
Back-end (BES) customization
Single/Multiple Machine Installations
Security
Hyrax Architecture
DAP2
BES Commands
OLFS
BES
THREDDS
HTML
XML- encapsulated object
Java
Servlet Engine
Unix Daemon
Optional THREDDS
catalogs
File system with
data files,
SQL Database, …
OLFS Dispatch Handlers
• The OLFS uses an ordered list
of Dispatch Handlers.
• Each handler on the list is asked
if it can handle the incoming
request.
• The first handler on the list that
claims the request gets to handle
it.
• List order is important, as some
handlers (for example THREDDS)
may claim to handle requests that
should be handled by a different
handler.
• Handlers are free to do anything
they need to handle a request:
Contact a remote
system/process, read files, spawn
threads, et cetera.
Extending the OLFS
• Extension ‘modules’ written in Java
• Added to a directory within Tomcat
• The new modules have complete access to
the request information
• Both HTTP GET and POST requests
• There is some significant processing done
before the handler is called:
– Conditional GET Requests
– Authorization & Authentication
OLFS and
Authorization/Authentication
• Authorization & Authentication (Az/Ac) actually
provided by Apache or Tomcat
• Apache/Tomcat provides a range of Az/Ac features:
– Realms: Role-based Az/Ac
• Usernames & passwords -> Authorization
• Roles -> Authentication
• Realm info in RDBs, XML docs, LDAP, …
– TLS/SSL:
• Server and client Az/Ac
• Secure data transmission
• Tomcat provides single sign-on capabilities
– Clients must support cookies
– Suitable for portals
BES
BES Framework
BES Commands/
XML Documents
NetCDF3
PPT*
Initialization/
Termination
DAP2
Access
Data
Catalogs
HDF4
FreeForm
Network Protocol and
Process start/stop
activities
Commands**
…
Data Store Interfaces
*PPT: point to point protocol
**Some commands are built
in, most load at run-time
BES Extensibility
In what ways can you extend the BES?
• New request handlers (data handlers like netcdf, hdf4)
• New response handlers (Cedar's Flat, Tab, Info responses)
• New commands (like our hello world example)
• Initialization/Termination
• Exception Handlers
• Reporters
• Container and Definition storage
• Aggregation engines
• Methods of returning your data (return as netcdf)
Possible Installations
• Both OLFS and BES run on one
machine
• OLFS on one machine and BES on
another
• One OLFS and several instances of
BES on different machines
• OLFS communicating with one or more
BESs and other backend processors
Installation Security
• A separate issue from Ac/Az
• The BES must be protected:
– With a firewall or
– TLS & Client certificates
• Running the OLFS and BES on separate
machines limits the scope of a compromise of
the OLFS
• Ensure that the BES, Tomcat and Apache all
run with limited access to the server host
Hyrax Architecture Summary
• Hyrax can be installed on one machine
or several
• Installation security merits serious
consideration
• Authentication & Authorization are
handled by the web servers (Tomcat
and Apache)
Hands on: Hyrax
Configuration
• Choices:
–
–
–
–
–
–
Single or Multiple machine
Single or Multiple back-end servers
Tomcat or Apache web server
Data formats
Catalog customization
Security
• Testing
– Command line tools for system administrators
– Web browser
APAC Workshop
Configuration
• Run a single BES and the OLFS on one
host (the virtual machine running SLAX
Linux)
• Use Tomcat running on port 8080
• Data formats: NetCDF and FreeForm
• No custom THREDDS catalog
• No firewall
APAC Hardware
• SLAX Linux VMware Virtual Appliance
• All the software needed has been built,
installed and configured
• Start the virtual machine now…
Starting the Virtual Machine
• Insert the CD-ROM
• Drag the opendap_vm and Wintools folders to the Windows
desktop
• Copy the ISO image from the CD ROM to the harddisk (in a
command window, mkdir C:\SLAX and then use copy or dd
to copy the iso to the new directory).
– The dd command is in the Wintools folder
– The C:\SLAX directory is coded into the virtual machine; use
Vmware Workstation/Fusion to edit
• Open the opendap_vm folder and double click on the
opendap_vm.vmx virtual machine configuration file or start
VMware Player/Workstation/Fusion and open
opendap_vm.vmx from there.
• The virtual host will boot and the SLAX (slackware Linux with
KDE) desktop will appear.
Installing the Software
• Already present on the virtual machine
• But, if it wasn’t, you’d go to the
OPeNDAP web site* and download:
– Binaries or source for the BES and the
data handlers needed.
– Get the OLFS web archive file (which is a
compiled java servlet).
– You also need the Tomcat servlet engine
* http://opendap.org/download/hyrax.html
Location of Server Files on the
Virtual Machine
• On your virtual machine:
– The Tomcat servlet engine is in
/usr/local/javadev/apache/tomcat-5.5.12 (this is
the value of $CATALINA_HOME)
– The BES, data handlers and related source files are in
/usr/local/src. The BES has been built and installed in
/usr/local/ ($prefix)
– The OLFS web archive file is
$CATALINA_HOME/webapps/opendap.war
– BES: bes.conf, found at $prefix/etc/bes/bes.conf
– OLFS: olfs.xml and catalog.xml, found at
$CATALINA_HOME/content/opendap
Background: Starting the
Server
• Start the BES (back-end data
processing component)
• Use bescmdln to verify it’s working
• Start Tomcat: This automatically starts
all installed servlets
– Servlets are installed by copying the .war
file to the servlet’s webapps directory
• Verify it’s working using a web browser
Start the BES
Verify the BES is running
Start Tomcat & the OLFS
•
Typical steps:
– Unpack the olfs jar-file
– Copy the opendap.war file to Tomcat’s
webapps directory
– Start Tomcat
•
Since all but the last step has been
done already, start Tomcat:
–
/usr/local/javadev/apache-tomcat5.5.12/bin/startup.sh
…terminal view
Verify Tomcat is running
http://localhost:8080
…and Hyrax
http://localhost:8080/opendap
Complete the Configuration
• Steps you would typically perform:
– Security: Set up a firewall! Limit access to
port 10002 to this host only
– Custom catalogs: Edit the catalog.xml
configuration file. By default automatic
catalogs are generated
– Logging: Edit the log4j.xml file. By default
all accesses are logged
Stopping Hyrax
First, stop Tomcat using ‘shutdown.sh’
…then stop the BES using ‘besctl’
Hyrax Configuration Summary
• For our chosen configuration, we used only
Tomcat, plus the Hyrax web application which
consists of the BES and the OLFS
• The tools bescmdln, getdap and a web
browser were used to test the installation
• There are a lot of options, but the default
settings produce a working server
• Security is a must for a web application; use a
firewall to isolate the BES so only the local
host can connect to it.
Summary
• There are a number of DAP-compliant
servers (Hyrax, TDS, PyDAP and GDS) were
described
• All of the servers (read ‘web applications’)
actually support several other protocols
• While DAP does nothing to support
cataloging data, THREDDS supports just that
• Hyrax is customizable and can be installed in
several different configurations