Transcript The EDUCE database

Tasks within WP4
&
The EDUCE database - A summary
WP1, WP2, WP4 and WP5 tasks
Jussi Kaurola, Niilo Siljamo, Antti Arola
Ozone and UV-Radiation Research Group
Finnish meteorological Institute
Contributions: Tim Martin (UG), Brian
Gardiner&Peter Kirsch (BAS), Harry Slaper&Peter
den Outer (RIVM), Ola Engelsen (NILU), Martin
Stanek (CHMI), Ann Webb&Richard Kift (UMIST),
Massimo Ferraguto (SSF), ...
Tasks within WP4 (deliverables):
4.1. Installation of new hardware, permitting... (FMI)
4.2. Installation of WWW data submission tools (FMI)
4.3. Addition of further Oracle structures for
non-spectral and ancillary data (FMI, BAS)
4.4. Provision of web pages to provide search access to
the Oracle database and guidance to users (FMI)
4.5. Update and maintain metadata pages for the
database, including flag for WMO guidelines (UMIST)
4.6. Testing of and report on all database search and
access tools, and publishing of user's guide and
manual, with copy made available at web site (UG)
The EDUCE database - A summary
WP1, WP2, WP4 and WP5 tasks:
WP1 & WP2
recent changes in the database
screen shots
submission facilities
retrieval facilities
data integrity
hardware issues
plans for the future
FMI tasklist
summary
FMI task in WP1:
"RAF for ozone and albedo"
Antti Arola
"The irradiance ratio of snow-covered to snow-free
conditions tells how much snow enhances the
irradiance on average.
The average albedo RAF in Sodankylä, calculated
from 11 years of data, is about 1.19. In other
words, snow enhancement is about 19%"
Report 1.4 (11pages):
http://www.muk.uni-hannover.de/~martin/
results/deliverables/1.4/wp_1_4.pdf
FMI task in WP2:
"2.1An evaluation of the impact of atmospheric
variability on the detection of trends in UV"
Antti Arola
The developed methodology offers a way of separating the effect
of a single factor, e.g., ozone, cloudiness, aerosols or albedo, on
the short- and long-term variability of UV, using measurements of
spectral irradiance
Report 2.1 (13 pages):
http://www.muk.uni-hannover.de/~martin/results/
deliverables/2.1/wp2.1_finalreport.pdf
Arola A., K. Lakkala, A. Bais, J. Kaurola, C. Meleti, and P. Taalas,
"Factors affecting short- and long-term changes of spectral UV
irradiance at two European stations", JGR,
doi:10.1029/2003JD003447, 2003.
Recent changes in the database
quality flags:
old data flagged "manually"
incoming data flagged automatically
submission:
auxiliary data
retrieval of data:
flags included in the retrieval of spectra
broadband data
auxiliary data: daily values (hourly values not yet)
better information of recent submissions,
enhanced user options for output (CHMI)
Data submission facilities
spectral and broadband submission has been in
operation since 2001
the submission of auxiliary data was delayed due to
problems with (old) software
now the submission works also for aux data
"short and friendly guides" for spectral and
auxiliary data
spectral guide should be OK (after several iterations)
auxiliary guide was revised two weeks ago
planned changes:
speeding up of submission process
make it work more as a background process
Data retrieval facilities
changes made since the last meeting
new oracle table: measurement_type
(spectral, broadband, ozone, cloud, dursun, snowcover)
broadband data can be retrieved
daily auxiliary can be retrieved (currently not hourly data)
flags included in headers and can be retrieved separately
some fixes of previously reported problems
planned changes
fix problems with hourly auxiliary data
new user requirements for WPI design, give us feedback!
spectral retrieval somewhat slow and error-prone (queue
problems). A completely new way to make SQL query and
consequent data handling is planned
flagging of one spectrum
••
••
••
••
••
149263#056 GREEN
GREEN Cloudy 3.639e+02 -1.000e+05 8.380e+01 -1.000e+05 -1.000e+05
GREEN -1.200e-02 14
GREEN 1.700e-02 67
GREEN 1.010e-05 6.600e-06
GREEN 1.850e+00 0 0 0
GREEN
CLOUDS 2.860e-01 3.207e-01
SOME_VARIATION_SCAN 9.240e+00
9.766e-04 slf00567
/UVROOT/DB/DBroot/no_ntnu_trh/1997/ol752/11010000.gxa 56
Data integrity: duplicates
• what is a duplicate?
– a measured spectrum which has several copies in
the database
– several different spectra have the same
measurement time
• what is an overlapping spectrum?
– several different spectra have overlapping
measurement time
• both types are included in the data retrievals
• both types need to be removed
Data integrity: duplicates
duplicates caused by errors in submitted data
automatic identification: very complex task, not
possible with current software
manual work is needed
current status (August run by the BAS tool):
belsk bre064 128
bru
jytot
148855
spurious zeroes
during the flagging spectra with only zero
values in irradiance column were recorded
41345 spectra found
only 93 cases with sza < 90 degrees
at_boku_sonnblick
at_boku_vienna
be_iasb_bru
fi_fmi_sod
fr_ustl_vda
gb_umist_reading
it_jrc_ispra
no_nilu_andoya
2
1
53
2
5
11
27
8
Hardware status
the current hardware is becoming old
(purchased in 2000 with $40 000)
disks are almost full, Oracle disk at critical level
recent submissions have been very large
total disk space 106 GB
Oracle data size 56 GB
Flexstor data size 23 GB
CPU power is adequate, flagging might be lagging a
bit in the case of large submissions
disk failure on last Thursday, which was handled very
well by the FMI IT department
Planned changes
some hardware upgrading is needed:
poor mans solution to disk problem:
new disks: 4 x 150 GB (4x600 euro)
to current disk box (2 for data, and 2 for mirror)
better solution: new disk array (about 15 keuro)
and some new disks. Much more tolerant for errors.
best solution: new server (20 keuro), new disk array (15
keuro) and new disks
a short break in service, a longer break in uploading of new
data to Oracle
operating system upgrade in November-December
causing 2-3 days break
upgrading of e.g. WPI related software (TOMCAT)
Planned changes...
currently everything is in ozone2.fmi.fi server
which is outside FMI firewall ("grey area")
an alternative solution:
server with Flexstor and WPI in "grey area"
Oracle inside firewall
a better integration of EUVDB Oracle database to
FMI other Oracle databases. Benefits:
security
maintenance
even performance?
FMI task list
• follow the link on "General Information Page"
• updated regularly
• check the "wish list" if you have ideas for new
improvements
Some remarks:
-campaign data
• manual submission by FMI
• retrieval from www pages needs planning and
work
Summary
• personal opinion: now the functionalities of the
database are rather good
• some FMI milestones were delayed considerably
• the whole project time was more or less
development period, which certainly has caused
headache to some users
• remaining work: hourly auxliary data
• planned improvements:
– smoother submission of large datasets
– faster retrieval of spectral data
– improved WPI design
• our interest is to continue with the database
maintenance. Funding to be discussed later...