Transcript Document

AGU Poster IN21B-1049
AGU Fall Meeting
December 15, 2009
The Data Management System for the Shipboard Automated Meteorological
and Oceanographic System (SAMOS) Initiative
Shawn R. Smith1, Robert Arko3, Mark A. Bourassa1,2, Jiangyi Hu1, Michael McDonald1, Jacob Rettig1, and Jeremy Rolph1
http://rvdata.us/, http://samos.coaps.fsu.edu/
1 Center
SAMOS Objective
The Rolling Deck to Repository (R2R) project was recently
launched with the ambitious goal of documenting “routine
underway data” from the US academic research fleet and
delivering those data to established national archives. Data
distributions will be submitted by 18 operating institutions
for 30 vessels, from hundreds of cruises per year.
Partnership with SAMOS data assembly center
In 2003, NOAA established a data assembly center (DAC)
at the Florida State University to provide data stewardship
for underway meteorological and near-surface
oceanographic data collected by research vessels. A new
partnership has been established between the R2R project
and the DAC to extend SAMOS data stewardship to the
US academic fleet.
SAMOS Data Management System
To collect, quality evaluate, distribute, and
ensure future access (via national archives)
to underway meteorological and near-surface
ocean data collected on research vessels
Overall, the data management system is a series of
automated processes coupled to an SQL database. The
database stores vessel metadata profiles, ship-specific
processing parameters, file tracking and version control tags,
and data quality flags.
SAMOS Data
The SAMOS data assembly center (DAC)
receives voluntary contributions from 23
vessels (as of Dec 2009; Fig. 1).
ASCII to
NetCDF
Quality
Control
Preliminary
Data Posted
SASSI:
Statistical
Quality
Control
On a 10 day delay, all files for a single ship and observation
day are combined (allowing for late file receipt)
Analyst
Notification
Problem
Secondary automated QC locates spikes, steps, and highly
variable observations and results in intermediate product.
Visual
Quality
Control
Save &
Post
Table 1
Automated QC checks data for valid ranges, ship speed, and
location; agreement with a climatology; temporal sequence,
and physical consistency.
Trained data quality analyst uses a graphical user interface to
add, modify, or remove QC flags.
Resulting research quality files are posted for users.
Internet
RSYNC
NODC
Archive
The final stage of the SAMOS DMS is to submit the data and
metadata to the National Oceanographic Data Center.
On a monthly basis, all original, preliminary, intermediate, and
research-quality SAMOS data files are uploaded to NODC.
Preliminary files are posted to web/ftp/THREDS servers.
Analyst notifies operators at sea when problems are
detected.
1.
2.
5.
Preliminary processing is fully automated.
After verifying source and format of incoming message, data
are converted to network common data form and merged with
vessel specific metadata from database.
In 2009, the SAMOS DAC forged a partnership with
the R2R to expand SAMOS data stewardship to all
vessels in the US academic fleet.
3.
4.
NOAA supports visual QC for select vessels
Monthly
Data
SAMOS-R2R Partnership
Analyst
Notification
NODC develops FGDC metadata records and provides longterm access and stewardship.
Objectives
Initiate transfers of relevant parameters (Table 1)
from all US academic fleet vessels
Increase data transfer frequency to meet needs
of operational weather forecasting community (at
least every 6 hours)
Conduct fully automated QC
Provide operators with QC feedback and
recommendations for sensor deployment
Develop automated protocol for ship-to-shore
metadata transfers
Standards
The SAMOS DAC will move to adopt R2R
vocabularies for vessel names, ports, etc.
The DAC continues to move towards the Climate
and Forecast (CF) network common data form
Plan to utilize R2R cruise inventory and cross
populate vessel profiles
Proposed Data Transfer Mechanism
Two options:
1. Large operators with sufficient IT staff will
continue to use current SAMOS data transfer
protocol.
2. SAMOS-R2R to explore development of
software/hardware to be installed on vessels
• This protocol will support smaller operators
• Recommended by UNOLS Research Vessel
Technical Enhancement Committee
• Input from technical community welcome.
Metadata Challenge
Since its inception, the SAMOS DAC has endeavored to collect
extensive ship and instrument metadata, including digital imagery
(Fig. 4), to meet the scientific goals our user community.
Initially, metadata was requested from recruited vessels using
simple forms that could be transmitted via email (Fig.2).
R2R Real-time Data Protocol
The Rolling Deck to Repository Project acknowledges support from the
National Science Foundation (NSF), Oceanographic Instrumentation and
Technical Services (OITS) Program.
Base support for the SAMOS data center is provided by NOAA’s Office of
Climate Observation.
OK
Cron/Manual
Trigger
Merge
Data
File
Merge handles temporal duplicates using preliminary QC
Email receipt by the DAC triggers automated SAMOS
processing (white list controlled).
Check
&
Verify
The one-minute interval of SAMOS data and
the tendency of research vessels to operate
outside of routine shipping lanes makes the
observations ideal for satellite and model
validation and calibration.
Presently, SAMOS data are acquired directly from
research vessels at sea via a daily email transfer protocol.
The DMS automatically tracks progress of the daily data
acquisition and quality control (QC), stores metadata on
instrumentation and ships, and provides data monitoring
capability via a user-friendly web interface. An SQL
database stores essential parameters to support tracking,
data QC, and version control throughout the process.
The SAMOS DAC plans to develop new protocols to
transfer underway meteorology and near-surface
oceanographic data from the academic fleet to shore.
Ideally, the protocol will allow the DAC to either transmit
higher frequency (e.g., 1 Hz) meteorological and surface
oceanic samples from participating vessels to real-time
servers at R2R or develop software/hardware that can be
installed on the vessels. Under either protocol, the SAMOS
DAC would support data reduction (averaging), shore-side
monitoring, quality control, metadata acquisition, data
distribution, and archival at a national data center.
Fig. 1: SAMOS ship tracks for FY2008 and FY2009 color coded by operator.
The Australians provide the only international data via the IMOS project.
User Community
The SAMOS Data Management System
SAMOS
Processing
Transmissions nominally sent at 0000UTC and include all
observations for the previous day.
Parameters collected (Table 1) vary from
vessel to vessel, but must include
observation time, position, and units.
Other uses include:
•Developing satellite retrieval algorithms
•Air-sea interaction studies
•Ocean process studies
•Primary ocean production via radiative
processes
•Validation of operational marine forecasts
Email
Analyst
Feedback
Currently, recruited SAMOS vessels use an email protocol to
transmit 1-min. averaged data to the DAC
The SAMOS Data Center
The DAC for has developed and implemented an
automated data management system (DMS) that collects,
formats, quality controls, distributes, and archives near
real-time surface marine data from research vessels. A
SAMOS is a computerized data logging system that
continuously records navigational (ship’s position, course,
speed, and heading), meteorological (winds, air
temperature, pressure, moisture, rainfall, and radiation),
and near-surface oceanographic (sea temperature, salinity,
conductivity, florescence) parameters while the vessel is at
sea. The SAMOS initiative relies on the high-quality
instrumentation purchased and deployed by the research
vessel operators and does not provide instrumentation to
the vessels. Currently, the SAMOS initiative receives
measurements recorded at 1-min intervals and derived
from higher frequency samples (on the order of 1 Hz). In
2009, 23 research vessels are providing routine SAMOS
observations to the DAC.
Satellite
Broadband
(e.g.,
HiSeasNet)
Ship
Observations are from automated instrument
systems operated and maintained by the
vessel’s home institution
SAMOS observations are one-minute
average values derived from higher
frequency (~1 Hz) instrument samples
for Ocean-Atmospheric Prediction Studies, Florida State University
2 Department of Meteorology, Florida State University
3 Lamont-Doherty Earth Observatory of Columbia University
Although operators may fill out forms initially, they still required
tedious entry into our database and updates were difficult to obtain.
New web-based forms (Fig. 3) were developed to allow operators
to directly enter and update their metadata in our database.
Fig. 2: SAMOS vessel and instrument
metadata forms (Word version).
The web forms are password protected and all updates require
approval by the DAC.
Web forms eased data entry, but are mostly used by DAC staff.
In partnership with the R2R, the DAC will move to develop a
protocol to automatically transfer necessary metadata along with
the physical observations from ship to shore on a regular schedule
It Takes a Team
Researchers
Students
Technicians
Data Managers
Fig. 3: Web-based SAMOS ship and instrument metadata forms
showing select data from the R/V Atlantis.
Fig. 4: Examples of digital imagery
from Select SAMOS vessels. (top)
metadata collage from NOAA’s
Okeanos Explorer; (lower left)
instrument mast from the L. M.
Gould; (right) Frozen anemometer
on USCG Healy.