Transcript Campaign Database

Envisat Validation Workshop
Atmospheric Chemistry Validation Team
Ground-Based Measurements and Campaign Database Subgroup
Campaign Database
Terje.Krognes, [email protected]
Aasmund.Fahre.Vik, [email protected]
Norwegian Institute for Air Research (NILU)
Envisat Validation Workshop, Campaign Database, 12/12/2002
Page 1
Presentation Outline
Data Centre Status
Tools
 Local end-user tools
 WWW search tools
Commissioning Phase Activities
Envisat Validation Workshop, Campaign Database, 12/12/2002
Page 2
nadir web
DATA ORIGINATOR
DOCS
DATA
table.dat
ASC2HDF
http or ftp
Error email
http
Data flow
 Signed protocol required
for access to tools, docs
and files
 ECMWF data requires
additional protocol
HDF2ASC
OK?
FILES
DATABASE
SEARCH
DATA USER
ftp to zardoz
Envisat Validation Workshop, Campaign Database, 12/12/2002
Page 3
Data Centre Status (1/3): Hardware
 The machines have been swapped
between production and spare servers
 The hardware has not been changed
spare
Internet
nadir
http
Storage
Firewall
ftp,
telnet
zardoz
Envisat Validation Workshop, Campaign Database, 12/12/2002
Page 4
Data Centre Status (2/3): Software
 Web-server (nadir) runs Linux Redhat 7.2 with
up2date ~weekly (6.2 at 2nd rehearsal)
 File server (zardoz) runs Solaris 7, as before
 Nusphere MySQL







Apache 1.3.22 (1.3.14)
MySQL 3.23.44 (3.23.32)
PHP 4.0.6 (4.0.4pl1)
IDL 5.4, as before
HDF 4.1r3, as before
No re-boot on nadir since April 2002
Uptime on zardoz since April 2002 is approximately
99,98% (2 half-hour maintenance + re-boot)
Envisat Validation Workshop, Campaign Database, 12/12/2002
Page 5
Data Centre Status (3/3): Metadata
 Metadata Guidelines updated by metadata board on 24
April 2002
 During commissioning phase, updates of various
metadata table entries, without affecting structure:
 Project-related information
 Actual descriptors
 Guidelines still applicable, but more clarifications to be
added.
Envisat Validation Workshop, Campaign Database, 12/12/2002
Page 6
New user tools
 Major re-write of the ASC2HDF file generation and
checking tool around launch
 6 smaller upgrades of the ASC2HDF package after launch
 Entire ASC2HDF source code (FORTRAN) published to
users. HDF output routines may be combined with user’s
own input and processing modules
 IDL-tool to generate HDF data files (without error
checking): IDL2HDF
 IDL tool for reading and plotting HDF data files
(HDF2IDL), to be downloaded and used locally
 IDL subroutine to read HDF Cal/Val data, for integration
into user-developed software (nilu_hdf_read_user.pro)
Envisat Validation Workshop, Campaign Database, 12/12/2002
Page 7
New web functionality
- ECMWF
 ECMWF plots (NH 2,5º*2,5º
and T106)
 ECMWF 10-day back
trajectory plots
 T106 extractions to HDF files
Envisat Validation Workshop, Campaign Database, 12/12/2002
Page 8
New web functionality - search and plot
 Use of defaults
 Facility to save and retrieve a search
 Provision for downloading multiple files in tarball via web
interface
 Improved on-line plotting (also for multidimensional data)
 Corrected display of comments to variables
Envisat Validation Workshop, Campaign Database, 12/12/2002
Page 9
File searching (1/4)
 Defaults are marked with ***
 Page initially opens without
defaults
 Page may be reset or set to
saved defaults bys selecting
a previously saved query
 Files may be filtered by a
large number of metadata
parameters
Envisat Validation Workshop, Campaign Database, 12/12/2002
Page 10
File searching (2/4)
 The default criteria page lists all
criteria sets that this user has
previously saved
 Entries may easily be removed
 The main search page will be
loaded with the selected defaults
when launched from this page
 Defaults may include 4D spacetime parameters and variable
name wild-cards
Envisat Validation Workshop, Campaign Database, 12/12/2002
Page 11
File searching (3/4)
 The NUMERICAL SIMULATION entry is
now marked with *** since we selected it
as default through the saved query
page
 Variable name / mode / descriptor have
wildcard entries automatically added to
the drop down lists.
 Variable name and variable mode
accept multiple selections
 4D space-time criteria may be entered
as a single point, or as a max/min box
 All files that include or touch the 4D
criteria are listed as matches
Envisat Validation Workshop, Campaign Database, 12/12/2002
Page 12
File searching (4/4)
 Files may be selected by
posting date
 Files updated after a given
date may be selected
 The file list may be sorted by
file name, by submitter or PI,
by posting date, or by AOID
 If no criteria are selected, the
entire file list will be
requested
Envisat Validation Workshop, Campaign Database, 12/12/2002
Page 13
File List
 The file list displays the current
selection criteria
 A single file may be downloaded
immediately
 Files may be marked and
downloaded in a compressed
tarball
 Search criteria may be saved for
later re-use (type a meaningful
description before saving)
 The Variables link brings up a
variable list and other
documentation contained in the file
Envisat Validation Workshop, Campaign Database, 12/12/2002
Page 14
Variable List
 The variable list page displays all
parameters contained in a file
 Variable name, mode and descriptor are
displayed
 The measurement unit and
corresponding SI unit are displayed
 All documentation fields attached to the
file are displayed
 Additional documentation fields attached
to a single variable may be displayed
through the Comments link
 Variables may be selected to generate an
on-line plot (next slide)
 Observe the dependencies listed for
each variable – not all variable
combinations will produce meaningful
plots
Envisat Validation Workshop, Campaign Database, 12/12/2002
Page 15
Variable plotting (on-line browse)
Envisat Validation Workshop, Campaign Database, 12/12/2002
Page 16
Administrative tools
 Web tools to report uploads/downloads
 Web tools to archive/retrieve outdated
data files
 Web tools to check and maintain index
database consistency and integrity
Envisat Validation Workshop, Campaign Database, 12/12/2002
Page 17
User activities
 Users have sent approximately 200 emails with requests for accounts
or updates to metadata, or with questions to HDF file generation
 After launch focus has slowly shifted to requests for help with HDF file
formatting issues and removal of outdated files
 From May to November 2002 users have submitted 5880 HDF data
files (570 of these were rejected due to formatting problems, and 238
have later been archived).
 From May to November 2002 users have downloaded 4000 HDF data
files (totalling 215 MB) (ftp downloads via zardoz not included)
 There are currently 4741 ACVT files open to users (22 active AOIDs)
 There are currently 462 MAVT files open to users (5 active AOIDs)
Envisat Validation Workshop, Campaign Database, 12/12/2002
Page 18
User activities: Up and Downloads
1200
1000
800
uploads
downloads
600
400
200
0
35 36 37 38 39 40 41 42 43 44 45 46 47 48 49
The download statistics only covers WWW downloads from nadir, not FTP
downloads from zardoz
Envisat Validation Workshop, Campaign Database, 12/12/2002
Page 19
Nadir-team activities
 300+ interactions with users (of which 50% during Commissioning Phase)
 250+ interactions with ESA and the Metadata Board (10% during
Commissioning Phase)
 One major re-write and 6 updates of the ASC2HDF and HDF2ASC programs
 Maintained and improved shell scripts for automatic file checking and storage
 Created additional IDL and FORTRAN tools
 Performed extensive web program development
 Performed user administration updates
 Added metadata entries and updated the table.dat file 23 times since launch
 Developed and maintained scripts and data flow to provide pre-processed
ECMWF data extractions and plots on the web
 Monitored the automatic file reception system (now containing nearly 5000 files
totalling 275 MB + index database 20 MB)
Envisat Validation Workshop, Campaign Database, 12/12/2002
Page 20
Metadata issues
Although many quality checks are performed on the data upon submission to
the database:
 The metadata definition is currently not complete enough to perform
automatic checks of the validity of reported measurement units
 Some components and units may still be inconsistently named or mis-used
Several files have indeed been submitted containing such errors due to
misinterpretation of the guidelines. The data submittors will be contacted to
correct these files.
Envisat Validation Workshop, Campaign Database, 12/12/2002
Page 21