Transcript seacoos_data_management_details

Near Real Time
Ocean Observations
Online
the Detailed Escape of SEACOOS
(Southeastern Atlantic Coastal Ocean Observing System)
Data Management and Visualization Secrets
Software : engines

Mapserver
Open source from University of Minnesota
 USC runs a mixture of versions from 3.6 to 4.2


PostgreSQL
Open source from postgresql.org
 USC runs version 7.4.1


PostGIS
Open source from Refractions.net
 USC runs version 0.8.1

Near Real Time Ocean Observations
Online : SEACOOS
Software : additional

PHP
Open source from php.org
 USC runs version 4.3.2


Perl
Open source from perl.org
 USC runs version 5.8.0


Miscellaneous
ANiS and gifsicle
 Imagemagick

Near Real Time Ocean Observations
Online : SEACOOS
Hardware

data scout



In-situ
RS
application server




Apache 2.x
MapServer
Perl, PHP, misc.
database A


In-situ
RS
Near Real Time Ocean Observations
Online : SEACOOS

database B


In-situ
model output
Directory structure

Data that sits on USC filesystems
RS images
 cached images


All files have strict naming convention that
includes timestamp
Near Real Time Ocean Observations
Online : SEACOOS
System administration

Databases backed up nightly


Databases cleaned up and optimized nightly


requires some downtime (~ 2 hours)
significant overhead ~ 4 hours
Some backend data massaging (mainly for model
output aggregation)
Near Real Time Ocean Observations
Online : SEACOOS
Database structure

One table category per data type

e.g. in-situ winds



e.g. QuikSCAT winds


more complicated since requires aggregation and normalization
e.g. OI SST


wind_prod contains all wind data
wind_map contains wind data appropriate for maps
table containing pointers to data files on disk
Ancillary tables for specific data types

e.g. OI SST

table containing RGB to SST lookup values (for querying purposes)
Near Real Time Ocean Observations
Online : SEACOOS
Data processing overview

Data scout (netCDF)




Perl code flattens incoming netCDF into arrays which are turned into
SQL INSERT statemtents.
Triggers update the new records suitable for normalization as well as
Mapserver display elements.
Procedures run to optimize tables for display.
RS images (HDF to PNG)



begin as HDF but are availed to USC as PNG’s
Images have standard naming convention and agreed upon extents as well
as predefined RGB to value, e.g. RGB to SST, pairs.
Incoming files cause postgreSQL table to be updated with the new file
and timestamp.
Near Real Time Ocean Observations
Online : SEACOOS