Transcript Slide 1

Poster 3-25
IMDIS 2010
29-31 March 2010
The Shipboard Automated Meteorological and
Oceanographic System (SAMOS) Initiative
Data Management System
Shawn R. Smith1, Robert A. Arko3, Mark A. Bourassa1,2, Jiangyi Hu1,
Michael McDonald1, Jacob Rettig1, and Jeremy Rolph1
http://rvdata.us/
http://samos.coaps.fsu.edu/
SAMOS Objective: To collect, quality evaluate, distribute, and ensure future access (via national archives)
to underway meteorological and near-surface ocean data collected on research vessels.
The Rolling Deck to Repository
(R2R)
project
was
recently
launched with the ambitious goal of
documenting “routine underway
data” from the U. S. academic
research fleet and delivering those
data to established national
archives. Data distributions will be
submitted
by
18
operating
institutions for 30 vessels.
SAMOS/R2R Partnership
In 2003, NOAA established a data
assembly center (DAC) at the
Florida State University to provide
data stewardship for underway
meteorological and near-surface
oceanographic data collected by
research vessels. A partnership
has been established between the
R2R project and the DAC to extend
SAMOS data stewardship to the
U. S. academic fleet.
The SAMOS Data Center
The DAC has developed and
implemented an automated data
management system (DMS) that
collects, formats, quality controls,
distributes, and archives near realtime surface marine data from
research vessels. A SAMOS is a
computerized data logging system
that
continuously
records
navigational
(ship’s
position,
course, speed, and heading),
meteorological
(winds,
air
temperature, pressure, moisture,
rainfall, and radiation), and nearsurface
oceanographic
(sea
temperature, salinity, conductivity,
fluorescence) parameters while the
vessel is at sea. The SAMOS
initiative relies on the high-quality
instrumentation purchased and
deployed by the research vessel
operators and does not provide
instrumentation to the vessels.
Currently, 26 research vessels are
providing
routine
SAMOS
observations to the DAC.
The Data Management System
Presently, SAMOS data are
acquired directly from research
vessels at sea via a daily email
transfer
protocol.
The
DMS
automatically tracks progress of
the daily data acquisition and
quality
control
(QC),
stores
metadata on instrumentation and
ships,
and
provides
data
monitoring capability via a userfriendly web interface. An SQL
database
stores
essential
parameters to support tracking,
data QC, and version control
throughout the process.
R2R Real-time Data Protocol
The DAC is developing new
protocols to transfer SAMOS data
from the academic fleet to shore.
Ideally, the protocol will either allow
operators
to
transmit
highfrequency (~1 Hz) SAMOS data
from participating vessels to realtime servers at R2R or the DAC to
develop software/hardware that
can be installed on each vessel.
Under either protocol, the SAMOS
DAC would support data reduction
(averaging), shore-side monitoring,
quality
control,
metadata
acquisition, data distribution, and
archival at a national data center.
The Rolling Deck to Repository Project
acknowledges support from the National
Science Foundation, Oceanographic
Instrumentation and Technical Services
(OITS) Program. Base support for the
SAMOS data center is provided by
NOAA’s Office of Climate Observation.
1 Center
for Ocean-Atmospheric Prediction Studies, Florida State University
2 Department of Meteorology, Florida State University
3 Lamont-Doherty Earth Observatory of Columbia University
SAMOS Data
The SAMOS data assembly center (DAC) receives
voluntary contributions from 26 research vessels
(as of March 2010; Fig. 1).
Observations are from automated instrument systems
operated and maintained by the research vessel’s
home institution.
Parameters collected (Table 1) vary from vessel to
vessel, but must include observation time,
position, and units.
SAMOS observations are one-minute average values
derived from higher frequency (~1 Hz) instrument
samples.
User Community
The one-minute interval of SAMOS data and the
tendency of research vessels to operate outside of
routine shipping lanes makes the observations ideal for
satellite and model validation and calibration.
•
•
•
•
•
Other uses include:
Developing satellite retrieval algorithms
Air-sea interaction studies
Ocean process studies
Primary ocean production via radiative processes
Validation of operational marine forecasts
Fig. 1: SAMOS ship tracks for FY2008 and FY2009 color coded by operator.
The Australians provide the only international data via the IMOS project.
SAMOS Data Management System
Overall, the data management system is a series of
automated processes coupled to an SQL database. The
database stores vessel metadata profiles, ship-specific
processing parameters, file tracking and version control
tags, and data quality flags.
Satellite
Broadband
(e.g.,
HiSeasNet)
Ship
Email
After verifying source and format of incoming message,
data are converted to network common data form and
merged with vessel specific metadata from database.
Analyst
Feedback
SAMOS
Processing
Automated QC checks data for valid ranges, ship
speed, and location; agreement with a climatology;
temporal sequence, and physical consistency.
Transmissions nominally sent at 0000UTC and include all
observations for the previous day.
Email receipt by the DAC triggers automated SAMOS
processing (whitelist controlled).
OK
ASCII to
NetCDF
Trained data quality analyst uses a graphical user
interface to add, modify, or remove QC flags.
Analyst notifies operators at sea when problems are
detected.
Quality
Control
Preliminary
Data Posted
Analyst
Notification
Problem
Cron/Manual
Trigger
Merge
Data
File
SASSI:
Statistical
Quality
Control
Resulting research quality files are posted.
Monthly
Data
Analyst
Notification
Internet
RSYNC
NODC
Archive
On a 10 day delay, all files for a single ship and
observation day are combined
(allowing for late file receipt).
The final stage of the SAMOS DMS is to submit
the data and metadata to the National
Oceanographic Data Center.
Merge uses preliminary QC to remove temporal
duplicates.
On a monthly basis, all original, preliminary,
intermediate, and research-quality SAMOS data
files are uploaded to NODC.
Secondary automated QC locates spikes, steps, and
highly variable observations, resulting in an
intermediate product.
Preliminary processing is fully automated.
Save &
Post
NOAA supports visual QC for select vessels.
Preliminary files posted via web, ftp, and THREDDS.
Currently, recruited SAMOS vessels use an email protocol
to transmit 1-min. averaged data to the DAC.
Check
&
Verify
Visual
Quality
Control
NODC develops FGDC metadata records and
provides long-term access and stewardship.
Metadata Challenge
SAMOS-R2R Partnership
The SAMOS DAC endeavors to collect
extensive ship and instrument metadata,
including digital imagery (Fig. 2), to meet the
scientific goals of our user community.
In 2009, the SAMOS DAC forged a partnership with
the R2R to expand SAMOS data stewardship to all
vessels in the U. S. academic fleet.
Initially, metadata was requested using forms
that could be transmitted via email (Fig. 3).
1.
2.
Although many operators fill out forms initially,
tedious entry into our database was required
and updates were difficult to obtain.
3.
4.
New web-based forms (Fig. 3) were
developed to allow operators to directly enter
and update their metadata in our database.
Web forms eased data entry, but are mostly
used by DAC staff.
In partnership with the R2R, the DAC will
move to develop a protocol to automatically
transfer necessary metadata along with the
physical observations from ship to shore.
5.
Objectives
Initiate transfers of relevant parameters (Table
1) from all U. S. academic fleet vessels
Increase data transfer frequency to meet needs
of operational weather forecasting community (at
least every 6 hours)
Conduct fully automated quality control (QC)
Provide operators with QC feedback and
recommendations for sensor deployment
Develop automated protocol for ship-to-shore
metadata transfers
Standards
The SAMOS DAC will move to adopt R2R
vocabularies for vessel names, ports, etc.
Fig. 3: Original
MS Word
(above) and
interactive webbased (right)
SAMOS
instrument
metadata forms.
The DAC continues to move towards the Climate
and Forecast (CF) network common data form
Plan to utilize R2R cruise inventory and cross
populate vessel profiles
Fig. 2: Examples of digital imagery from select
SAMOS vessels. (far left) metadata collage from
NOAA’s Okeanos Explorer; (left) Frozen
anemometer on USCG Healy; (right) instrument
mast from the L. M. Gould.
Proposed Data Transfer Mechanism
Two options:
1. Large operators with sufficient technical staff will
continue to use current SAMOS data transfer
protocol.
2. SAMOS-R2R exploring development of
software/hardware solution for other vessels
• This protocol will support smaller operators
• Recommended by UNOLS Research Vessel
Technical Enhancement Committee
• Wecoma and Endeavor are test vessels
• Input from technical community welcome.