Transcript Metadata and Data Management Activities

OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Metadata and Data
Management Activities winthin NA 3-7
S. Iona (HCMR), D. Schaap (MARIS),
L. Rickards (BODC), F. Nast (BSH)
www.seadatanet.org
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 Outlines
• SeaDataNet Objectives
• Training and Capacity building
• Discovery System
• Maintenance
• Upgrade
• Current Status
• Summary
Bologna, 19 September 2008
2
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 SeaDataNet Objectives
• To network existing oceanographic data centres already
nationally funded
• To develop an efficient distributed pan-European marine
data management Infrastructure (a “unique Virtual Data
Centre”)
• To provide on-line access to integrated databases of
standardised quality by using adapted communication &
information technology
Bologna, 19 September 2008
3
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 NA3-Training and Capacity building
Aims:
• to ensure that the data and metadata to be integrated in the
system will be formatted, checked for quality and disseminated
according to the common protocols developed during the
project
• to transfer expertise and to train IT experts of the SeaDataNet
data centers in the basics, installation and operation of the
SeaDataNet technical components
Bologna, 19 September 2008
4
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 NA3-Training and Capacity building
Training Workshops,
Oostende, (Belgium):
(
T
r
a
i
n
i
n
g
m
a
t
1. February
e
r
1
i
2
a
l
a
n
d
p
r
e
s
e
IOC
n
t
a
t
i
o
n
Rroject
s
a
v
a
i
l
a
b
l
e
Office
o
n
S
D
N
E
for
x
t
r
a
n
e
IODE,
t
)
-17 2007
Bologna, 19 September 2008
5
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 NA3-Training and Capacity building
Training Workshops,
Oostende, (Belgium):
(
T
r
a
i
n
i
n
g
m
a
t
e
r
i
a
l
a
n
d
p
r
e
s
e
IOC
n
t
a
t
i
o
n
Rroject
s
a
v
a
i
l
a
b
l
e
Office
o
n
S
D
N
E
for
x
t
r
a
n
e
IODE,
t
)
2. June 4-5 2007:
• dedicated on generating XML records with the use of
MIKADO tool
Bologna, 19 September 2008
6
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 NA3-Training and Capacity building
Training Workshops, IOC Rroject Office for IODE,
Oostende, (Belgium):
(
T
r
a
i
n
i
n
g
m
a
t
e
r
i
a
l
a
n
d
p
r
e
s
e
n
t
a
t
i
o
n
s
a
v
a
i
l
a
b
l
e
o
n
S
D
N
E
x
t
r
a
n
e
t
)
3. June 16-19 2008:
•
use of the new V1 formats, interfaces and maintenance
tools (MIKADO, online CMS, Web services, Validation
services, Vocabularies)
•
data quality control and assessments, using ODV
software
•
analysis and data presentations, using ODV – DIVA
software
Bologna, 19 September 2008
7
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 Discovery System
The SeaDataNet Discovery System is an integrated catalogue
service, aiming at facilitating marine data searching, location and
retrieval.
The Objectives are to:
• Maintain and expand the national metadata-bases.
• Standardize the information using common vocabularies and
reference tables.
• Interconnect the national inventories in a common Pan-European
directory.
Bologna, 19 September 2008
8
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 Discovery System
It is composed of several thematic inventories of different levels:
• European Directory of Marine Environmental Datasets
(EDMED)
• Cruise Summary Reports (CSR)
• European Directory of the initial Ocean-observing Systems
(EDIOS)
• European Directory of Marine Environmental Research Projects
(EDMERP)
• European Directory of Marine Organizations (EDMO)
• Common Data Index (CDI)
Bologna, 19 September 2008
9
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 Discovery System Structure
Bologna, 19 September 2008
10
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
11
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 Discovery System Structure
Common Reference Tables – EDMERP, EDMO
hold research projects and organizations
metadata common to higher directories
Bologna, 19 September 2008
12
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 Maintenance
Version 0 – 2006-2007
• Continuation and maintenance of existing Sea-Search system :
the data access needs several different requests to each data
centres
and the data sets are delivered in different formats
Bologna, 19 September 2008
14
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 Discovery System Upgrate
Version 1 – 2008-2010
• Setup of the integrated online data services to users :
networking of 10 “interoperable” data centres of the Technical Task
Team
unique request to the interconnected data centres
and the data sets are delivered with a unique format
Progressive integration of 10 data centres by end of 2008
Bologna, 19 September 2008
15
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 Current Status
• Frozen directories
• Content upgrade from Version 0 to Version 1 using on-line (like CMS
forms) and off-line tools that produce XML ISO 19115 compliant exchanges
and developed by the Technical Task Team in the joint research activities
JRA1,JRA2:
Bologna, 19 September 2008
16
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 Cruise Summary Reports
Cruise Summary Reports (CSR = former ROSCOPs) are the
usual means for reporting on cruises or field experiments at
sea. Traditionally, it is the Chief Scientist's obligation to submit
a CSR to his/her National Oceanographic Data Centre
(NODC) not later than two weeks after the cruise. This
provides a first level inventory of measurements and samples
collected at sea.
Bologna, 19 September 2008
17
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 New features in SDN V1
The major differences to previous V0 version are:
CSR Local ID given by Data Centre for future updates  makes it easier to modify/improve existing reports
Most entities now have defined vocabularies
eg. EDMO for organisation, EDMERP for projects and
many more
 no more spelling/typing mistakes or deviant
interpretations
Mandatory fields
 improves quality of report
Bologna, 19 September 2008
18
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 Structure of Report
Similar to the former ROSCOP forms the Cruise Summary
Report has 4 basic parts:
• General Cruise Information
• Mooring Description
• Sampling/Measurement Description
• Information on Geographical Coverage
Bologna, 19 September 2008
19
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 How to submit CSRs?
2 methods for generating CSRs
• Online for individual entries (CMS)
• XML files for bulk submission with the use of MIKADO tool
Both tools can be applied for new entries as well as updates !
Bologna, 19 September 2008
20
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 Online Content Management System
http://seadatanet.bsh.de/csr/on_line/V1_index.html
Link to
CSR
Discovery
Bologna, 19 September 2008
21
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 How to update existing reports?
Online update:
• contact BSH-DOD requesting CSRs to be updated
• BSH-DOD loads requested CSRs to entry database
• E-Mail with list of CSR ref. no. and passwords
• modify CSRs in entry database
• save and submit
Bologna, 19 September 2008
22
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 How to update existing reports?
Offline – XML update:
• contact BSH-DOD requesting CSRs to be updated
• BSH-DOD sends requested CSRs as XML V0 files
- 1 file/cruise using BSH CSR ref. no. as file name
- all free text information included
• submit modified CSRs in XML V1 format to BSH-DOD
- use BSH CSR ref. no. as central identifier (CSR ID)
- include local identifier (from NODCs) for future updates
Bologna, 19 September 2008
23
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 User Interface - Discovery
http://seadatanet.bsh.de/csr/retrieve/V1_index.html
Results
Bologna, 19 September 2008
24
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 User Interface - Report
Download in
V0 format
Bologna, 19 September 2008
25
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 Content Conversion into V1
Status of preliminary conversion for 39839 cruise reports:
•Responsible Laboratories (EDMO codes)
~ 20%
• EDMO codes of Chief Scientists
~ 20%
• Ship/Platform (ICES codes)
~ 60%
• Port of departure/return (C381):
~ 30%
• General Ocean Area (C161+C162):
~ 90%
• Sampling units (common vocab. L181)
~ 20%
• EDMO codes of Principal Investigators
~ 3%
Bologna, 19 September 2008
26
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 Continuing operation and population
Heraklion/Triest we got 1071 CSRs
Since Triest we got 1263 CSRs
• Online 33 %
• Online 52 %
• XML
34 %
• XML
• ICES 25 %
• ICES
• Others 8 %
• Others 0 %
Bologna, 19 September 2008
44 %
4%
27
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 Status
Number of entries (in an Oracle DB): 39 839
Characteristics:
From 1873-2008, 2105 ships from 48 countries
Bologna, 19 September 2008
28
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 CSR Contents
Country
Albania
Algeria
Belgium
Bulgaria
Croatia
Cyprus
Denmark
Estonia
Faroe Islands
Finland
France
Georgia
Germany
Greece
Iceland
Ireland
Israel
Italy
Latvia
Lebanon
Lithuania
Morocco
Netherlands
Norway
Poland
Portugal
Romania
Russian Federation
Slovenia
Spain
Sweden
Tunisia
Turkey
Ukraine
United Kingdom
United States
T otal CSR submissionsUpgraded to V1
1
7
22
41
75
2
5
8
45
15
577
5
261
49
2
102
9
17
35
7
14
1
14
117
14
3
24
32
14
145
152
91
38
176
27
9
2156
New V1 submissions
2156 CSR Entries during SDN Project
(1.4.2006 - 15.09.2008)
1
V1 in operation since 21. July 08
2
4
• Russia and Spain with online
upgrades
3
7
3
1
13
18
6
16
42
Bologna, 19 September 2008
• 42 already using new V1 from 8
countries,
• Lithuania first after the 3rd SDN
workshop in Oostende,
29
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 European Directory of Marine Organizations
• The directory lists the organization profiles of all (1000+) Data Holding
Centres, Research Institutes, Monitoring Agencies and Research
Vessel operators, that have an active role in one or more of the
SeaDataNet Discovery services (EDMED - data sets, EDMERP research projects, CSR - research cruises, EDIOS - observing
stations/ systems, and CDI - index to data).
• Direct crosslinks are provided to their entries in these directories.
• The organization entries are maintained online per country by the
SeaDataNet partners.
• new Web service for retrieving EDMO entries in XML
Bologna, 19 September 2008
30
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 Online Content Management System
http://seadatanet.maris2.nl/vu_organisations/welcome.asp
Bologna, 19 September 2008
EDMO CMS geo-locator via Google maps
31
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 EDMO V1 search and retrieval
http://seadatanet.maris2.nl/edmo
Bologna, 19 September 2008
32
EDMO
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Country
 EDMO Contents
•
Number of EDMO entries is quite
stable
•
As part of producing CDI records a
lot of new organisations have been
added; also existing entries have
been
altered
due
to
reorganisations, double entries etc.
•
Content upgrade from V0 to V1 is
finished
st
After 1 year SeaDataNet
No. of EDMO
entries
Albania - PUT
Algeria
Belgium
Bulgaria
Croatia
Cyprus
Denmark
Estonia
Finland
France
Georgia
Germany
Greece
Iceland
Ireland
Israel
Italy (OGS +ENEA)
Latvia
Lebanon
Lithuania
Malta
Morocco
Netherlands
Norway
Poland
Portugal
Romania
Russia (SIO + RIHMI)
Slovenia
Spain
Sweden
Tunesia
Turkey
Ukraine
United Kingdom
Other countries
TOTAL
Bologna, 19 September 2008
0
1
63
14
5
3
36
5
9
197
13
58
59
3
104
3
66
9
2
7
7
3
11
7
39
43
9
94
1
47
29
1
25
12
136
1122
After 2nd year SeaDataNet
No. of EDMO entries
1
1
69
12
5
3
31
5
10
198
14
58
57
3
46
2
60
6
2
6
7
3
22
23
40
45
8
88
1
68
32
1
26
10
153
17
1134
33
EDMO
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 European Directory of Marine Environmental
Data
EDMED is a directory of data sets relating to the marine
environment. It covers a wide range of disciplines including
marine meteorology; physical, chemical and biological
oceanography; sedimentology; marine biology and fisheries;
environmental quality monitoring; coastal and estuarine studies;
marine geology and geophysics etc.
Bologna, 19 September 2008
34
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 EDMED Contents
Currently, EDMED describes: over 3500 datasets from 700 data holding centres
Bologna, 19 September 2008
35
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 Total No. of Datasets and Data Holding Centres
Bologna, 19 September 2008
36
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 Progress
• Content Management System will be launched soon
(Content upgrade will start then)
• Web interface is under development
Bologna, 19 September 2008
37
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 European European Directory of Marine
Environmental Research Projects
EDMERP is a European directory of research projects
relating to the marine environment. It covers a wide range of
disciplines including marine meteorology; physical, chemical
and biological oceanography; sedimentology; marine biology
and fisheries; environmental quality; coastal and estuarine
studies; marine geology and geophysics etc.
Bologna, 19 September 2008
38
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 EDMERP developments
• double entries have been taken out, because projects are entered
and maintained by the country of the coordinator, who can add via
EDMO entries all related partners
• capability
of creation of sub-accounts for institutes in the NODC’s
country, while the NODC safeguards the quality by having the chief
editor role before publishing
• new Web service EDMERP entries in XML (export and import).
Bologna, 19 September 2008
39
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 EDMERP Retrieval
Browse list
Bologna, 19 September 2008
Additional details
40
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
http://seadatanet.maris2.nl/vu_edmerp/welcome.asp
 EDMERP – CMS
Bologna, 19 September 2008
41
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
After 1st year SeaDataNet
After 2nd year SeaDataNet
Country
Total No. of
EDMERPs
Albania
Algeria
Belgium
Bulgaria
Croatia
Cyprus
Denmark
Estonia
Finland
France
Georgia
Germany
Greece
Iceland
Ireland
Israel
Italy (OGS +ENEA)
Latvia
Lebanon
Lithuania
Malta
Morocco
Netherlands
Norway
Poland
Portugal
Romania
Russia (SIO + RIHMI)
Slovenia
Spain
Sweden
Tunesia
Turkey
Ukraine
United Kingdom
Other countries
TOTAL
Total No. of
EDMERPs
0
3
209
32
9
20
40
11
16
93
86
18
114
16
179
12
52
31
5
28
31
5
71
3
42
26
18
84
0
91
30
0
24
62
46
1507
2
4
128
35
7
9
41
9
14
90
102
23
117
13
160
4
48
29
12
21
30
4
67
4
35
26
16
103
4
79
15
3
28
79
236
 EDMERP Contents
• Number
of
EDMERP
entries reduced during
upgrading
because
of
removal of duplicates
1600
Bologna, 19 September 2008
42
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES

European Directory of Ocean Observing Systems
• EDIOS is the European Directory of the Ocean-observing
System, a unique searchable metadatabase of observing
systems operating repeatedly, regularly and routinely in
European waters.
• It contains metadata on European observing systems such as
platforms, repeated ship-borne measurements, buoys, remote
imagery, etc.
Bologna, 19 September 2008
43
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 Progress
• No new input has been requested over the year
• Awaiting new technological developments (XML schema and
new version of Mikado)
• BODC has produced an improved Oracle (database) schema
• Supported by common vocabularies, EDMO and EDMERP
Bologna, 19 September 2008
44
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
New User Interface:
http://seadatanet.maris2.nl/v_edios/search.asp
Bologna, 19 September 2008
45
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 Common Data Index
The CDI provides an index (metadatabase) to individual data
sets. For comparison: the present European Directory of Marine
Environmental Datasets (EDMED) gives an overview of
datasets at a high metalevel. Each EDMED data set description
covers a broad set of individual measurement data. The CDI
gives references to these individual measurement data,
providing a more detailed insight into the available datasets.
Bologna, 19 September 2008
46
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 New interface
Bologna, 19 September 2008
47
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Country
 CDI Contents
• Partners are now working with
the new Mikado to upgrade V0 to
V1
• The IIrd training workshop has
been very useful for transferring
expertise and instructing partners
in use of CDI format and tools.
Albania - PUT
Algeria – ISMAL
Belgium – MUMM
Belgium – VLIZ
Bulgaria – IO-BAS
Croatia – IORS
Cyprus – CYODC
Denmark – NERI
Estonia – MSI
Finland – FIMR
France – IFREMER
France - CLS
Georgia – TSU
Germany – BSH
Greece – NCMR
Iceland – MRI
Ireland – MI
Israel – IOLR
Italy - OGS
Italy – ENEA
Latvia – IAE-UL
Lebanon – NCRS
Lithuania – CMR
Malta – IOI-MOC
Morocco – INRH
Netherlands - NODC
Norway – IMR
Poland – IMGW
Portugal – IHPT
Portugal - IST
Romania – NIMRD
Russia – RIHMI
Slovenia – MBS-NIB
Spain – IEO
Sweden – SMHI
Tunesia – INSTM
Turkey – METU – IMS
Ukraine – MHI
United Kingdom – BODC
Bologna, 19 September 2008TOTAL
After 1st year
SeaDataNet
No. of CDI
entries
0
0
21
0
0
0
0
13676
0
0
71314
0
0
1594
10588
0
0
0
37926
0
0
0
0
0
0
5270
13546
0
0
0
0
2668
0
0
0
0
0
0
40122
195131
After 2nd year SeaDataNet
No. of CDI
entries
0
17
21
912
606
7111
1056
17042
0
1667
74762
8
99
29295
11588
0
4727
4817
37926
5395
104
7
810
0
0
41588
14099
0
0
396
2461
12587
2709
15547
2600
1497
2477
7476
40122
341499
See comments
See comments
See comments
See comments
See comments
See comments
See comments
See comments
48
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
 Summary
2006/2007
2007/2008
EDMED: 3.000
EDMED: 3.500
CSR: 38.525
CSR: 39.648
EDMERP: 1.507
EDMERP: 1.600
EDMO: 1.122
EDMO: 1.134
CDI: 195.131
CDI: 341.499
A very important increase in the CDI contents but
extra coverage of national data is need
Bologna, 19 September 2008
49