Transcript SeaDataNet tools NEMO, MIKADO, VALIDATOR

Training course – Ostende – 2-6 July 2012
SeaDataNet tools
NEMO, MIKADO, VALIDATOR
M. Fichaut, IFREMER, France
Training course – Ostende – 2-6 July 2012
SDN tools directory
•
On your desktop, SDN tools directory :
\presentations\SDN_tools.pps
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Overview
•
•
•
•
SeaDataNet tools in their context
NEMO : Reformatting tool
– Practical session : reformatting a set of stations
NEMO and SDN Download Manager – coupling table, NEMO
and MIKADO – CDI summary file
– Practical session : generation of a CDI summary file and of a
coupling table
MIKADO : XML files generator, Web validation service, Link
between MIKADO and SDN Download Manager – coupling
table
– Practical session :
1. Automatic creation of XML files for CDI, Web validation of these
files, creation of the corresponding coupling file
2. Manual input for EDIOS program, series and platform
www.seadatanet.org
[email protected]
Automatic creation of–XML
files for EDIOS
Training course – Ostende – 2-6 July 2012
XML validator
SeaDataNet
Vocabulary
CSR
Metadata
In
database
New
metadata
Metadata
in Excel
files
MIKADO
MIKADO
Meta data
at XML
ISO19155
EDIOS
Coupling
table
Data in
database
New
data
Data in
ASCII file
Format X
NEMO
EDMED
Collection
of ASCII files
at
SDN Format
EDMERP
CDI
Local copy of
data for
downloading
[email protected] – www.seadatanet.org
Data centres
Data request
Downloading
S
E
A
D
A
T
A
N
E
T
P
O
R
T
A
L
AAA
RSM
European portal
Training course – Ostende – 2-6 July 2012
Data Centre
CSR
Data in
database
Data
Coupling
Local ID+
File
table
address or
SQL Select
EDIOS
Collection
of ASCII
files at
Data
SDN format
EDMED
EDMERP
Download
Local
ID
Local
ID
Local
ID ++
manager
File
SQLaddress
Select
Local copy
of data
for
Downloading
downloading
Data request
AAA
S
E
A
D
A
T
A
N
E
T
ODV
User
Central ID
Zip file
P
O
R
T
A
L
CDI
Central
Central ID
ID +
+
ID
local
+ EDMO
Local
ID+
EDMO code
code
RequestStatus
Zip file
Manager (RSM)
[email protected] – www.seadatanet.org
DIVA
Training course – Ostende – 2-6 July 2012
Survey about the tools
 29 answers
Do you know the tool?
Yes
No
NEMO
4
25
MIKADO
16
13
•
NEMO : question about a Cruise name field
•
Questions on the coupling table : how to generate it, using
which tool?
•
Those who know, please help those who don’t know
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
NEMO – Reformating software
Training course – Ostende – 2-6 July 2012
XML validator
SeaDataNet
Vocabulary
CSR
Metadata
In
database
New
metadata
Metadata
in Excel
files
MIKADO
MIKADO
Meta data
at XML
ISO19155
EDIOS
Coupling
table
Data in
database
New
data
Data in
ASCII file
Format X
NEMO
EDMED
Collection
of ASCII files
at
SDN Format
EDMERP
CDI
Local copy of
data for
downloading
[email protected] – www.seadatanet.org
Data centres : SeaDataNet nodes
S
E
A
D
A
T
A
N
E
T
P
O
R
T
A
L
AAA
Data request
Downloading
RSM
European portal
Training course – Ostende – 2-6 July 2012
Objectives
 NEMO is a reformatting software used to
generate ASCII files at MEDATLAS or ODV formats which are
defined as SeaDataNet formats for data exchange between
SeaDataNet partners. Can be downloaded from SeaDataNet
Web site :
http://www.seadatanet.org/standards_software/software/nemo

It was created in IFREMER for internal purposes, and adapted
to take into account the SeaDataNet formats and vocabularies.
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Technical characteristics
•
•
•
•
•
•
Release : 1.4.4
Written in Java Language (Version >= 1.6)
Bilingual (French, English)
Available under multiple environments :
– Microsoft : Windows 2000, WINDOWS 7, XP, VISTA , APPLE
– Unix - Solaris
– Linux
Interactive and batch modes available
Use of SeaDataNet common vocabularies web services
– to update lists of values of the SeaDataNet common
vocabularies
• need network connections in order to have up to date lists
of values.
– www.seadatanet.org
• [email protected]
But NEMO works offline
once the lists are up-to-date
Training course – Ostende – 2-6 July 2012
NEMO main features
•
Reformat ASCII text file of vertical profiles,
time-series or trajectories to a SeaDataNet
ASCII format (ODV, MEDATLAS).
• The input ASCII files can be:
• one file per station for vertical
profiles or time series
• one file for one cruise for vertical
profiles, time series or trajectories
•
Interact with Mikado, to be able to generate
ISO-19115 XML descriptions of the data.
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
NEMO principles
•
•
•
NEMO must be able to read almost any ASCII format to
translate it to MEDATLAS, ODV
Users of NEMO describe the entry files format so that NEMO
is able to find the information which is necessary in the
SeaDataNet formats
One mandatory pre-requirement is that in the set of entry
files the information about the stations must
– be located at the same position : same line in the file,
same position on the line or same column if CSV format
– be in the same format
– For example : for all the stations the latitude is
• on line 3 on the station header,
• from character 21 to character 27, or 3rd column in CSV
• the format is +DD.ddd
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Running NEMO
•
4 or 5 steps to convert the entry files to a SeaDataNet format
1.
2.
Describe the type of file(s)
[Describe the cruise, if the files are related to 1 cruise]
NEMO is able to read XML CSR generated with MIKADO
3.
4.
5.
•
Describe the station information
Describe the measured parameters
Convert the file
One more step to
•
Save the description of the format (Model)
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
NEMO - Description of entry files (1)
• Where are the files?
• Is it a cruise?
– Is it one file per cruise or
one file per station ?
• Is it a collection of station files ?
– grouped by cruises or not?
• Are the files with separators?
– Tabulations? Semicolon?
Comma?
• Are they vertical profiles, time
series or trajectories data?
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
NEMO - Description of entry files (2)
File to convert is
displayed in
NEMO window
The principle is
to describe the
input file by
selecting the
information in the
file window
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Station header : constant number of lines
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Station header : number of lines not constant
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Cruise description
Can be:
Manually input
Imported from a
SeaDataNet XML
CSR
Imported from a
database
(IFREMER only)
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Station
description
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Station
description
Select the date
in the file, in the
left part of the
screen
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Station
description
Input the date
format
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Station
description
Click on the
Set button to
tell NEMO the
start and end
position of the
date
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Station
description
Click on the
Test button to
verify your
input
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Station
description
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Data
description
Choose the
parameter
list
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Data
description
Choose the
parameter
list
Give the
code, unit,
position ,
format and
input default
value of each
parameter
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Data
description
Select
« Parameter
list » in the
context menu of
the measured
parameters table
(right click to
open the menu)
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Data
description
Add all the
measured
parameters
and their unit
(P061) using the
context menu
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Data
description
Select the
value of the
measured
parameter in
the file, in the
upper left part
of the screen
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Data
description
Select « Set
start /end » in
the context
menu of the
measured
parameters
table
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Data
description
Start and End
position of
depth are added
in the table
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Data
description
Describe the
format and
default value of
the parameters
(output default
value only if
output format is
MEDATLAS)
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Data
description
Check your
input : Select
« Update test »
in the context
menu of the
measured
parameters
table
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Data
description
The values
read by NEMO
are displayed
in the Test
column of the
table. User is
able to check
that his inputs
are correct
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Data
description
Tell NEMO
which
parameter is
the vertical
reference
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
File
conversion
Output
formats
•
MEDATLAS
– Multi-station
•
ODV
– Mono-station
– Multi-station
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Non numeric
parameters
For sediment
description in
Sediment Core
Compatible with ODV
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
NEMO in batch mode
• MIKADO can be run in batch mode, from NEMO install
directory, using existing models
• Several arguments can be added on the command line
• Log file in NEMO install directory
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Files which can’t be reformatted by NEMO
•
•
•
Binary format : like EXEL, WORD …
– File have to be first reformatted to text file
Files which do not respect NEMO pre-requirements
– be located at the same position : same line in the file, same
position on the line or same column if CSV format
– be in the same format
Files which are not :
– Vertical profiles (with depth or pressure as vertical reference)
– Time series (with time as reference)
– Trajectories (with latitude, longitude and time as reference)
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
1201 310219590650370-029-330100
2 19660719140058999000000000000080161 013
30029800000258+
+ +
+ 201214
42010
28
0000
6000000036043261501461904
6000090030943320801460604
6000240039043409401465404
6000470054343452101472604
6000710047843449101470404
6000940052043472301472704
6001180059643491301476404
6001410065743509201479404
6001650060343506601477704
6001880051943491601474504
6002120052343496101475104
6002340046543491801473104
1201 31025246064008000060530000
2 19660806190058999000000000000170161 013
30030900000298+
+ +
+ 201214
40213
0000
6000000147043307601503504
6000100133643417301500604
6000250119043467401496504
Like for example old
historical tape formats
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
NEMO – User manual
•
•
User manual is provided :
File : sdn_Nemo_UserManual_V1.4.4.pdf
Detailed explanation for NEMO installation and use,
lots of snapshots
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
NEMO next releases
•
V1.4.5 (soon)
– With correction of some know bugs
•
V1.5.0 (scheduled in autumn)
– BODC vocabulary, version 2
• Pre-requirement : Importation tools must manage BODC V2
vocabulary
– Include a Cruise name field that can be read in the input file
•
V1.5.1 (not scheduled yet)
– Generation of files at NetCDF format
– Pre-requirement
• NetCDF format final definition
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
[email protected] – www.seadatanet.org
Training Workshop – Ostende – 1-4 March 2010
Training course – Ostende – 2-6 July 2012
Practical work on NEMO
Converting files
M. Fichaut
Training course – Ostende – 2-6 July 2012
From ASCII data to XML files
Collection
of
ASCII files
ODV
files
XML
CDI files
summary_CDI_NEMO.xml
Delivered with NEMO
Coupling table
Export
CDI summary
CSV file
Coupling table
[email protected] – www.seadatanet.org
MIKADO
SeaDataNet
CDI
Training course – Ostende – 2-6 July 2012
NEMO installation
•
•
•
•
•
Open the install_nemo_1.4.4_windows.zip file (on the
desktop)
Unzip it
Double click on launcher_nemo.bat
Follow the installation procedure
Don’t forget to add the NEMO shortcut on your desktop
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Collection of ASCII files
•
Training data set
– 6 CTDs raw files under
SDN tools/Practical work/data/input files
– ASCII files with measured parameters in
column
– Measured parameters = Depth, Temperature,
Salinity, Fluorescence and Dissolved oxygen
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
First step : conversion to ODV with NEMO
• Run NEMO to convert the files to ODV format.
1. Modify the settings of NEMO
• Default directories for data set to
SDN tools/Practical work/data
– Default directories for data set to
SDN tools/Practical work/models
– Enter your EDMO ID in the field below using the
search function (right click)
2. Proceed the 4 steps of NEMO
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
NEMO – File description
•
•
•
•
•
•
•
•
Choose the type of file (cruise directory, cruise file, …)
Browse your PC to input the data files directory
Once the file is open in NEMO, have a look to its content
Choose file type : Profile
Choose the output format : ODV, one unique file for all
stations or One file per station
Enter the Station header information
– there is a specific character string at the end of each
Station header : *END*
The Data termination indicator is EOF (End of file)
Validate the step when description is OK
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
NEMO – Station description
•
•
•
•
•
•
•
•
•
Station number is given on line “**Station”
Data type is ‘CTD stations’
Time is U.T.
Latitude format is NDD MM.mm
Longitude format is NDDD.MM.mm
Bottom Depth is not available
Input all the mandatory fields : station number, time, date,
latitude, longitude
Use the Test function to verify your input
Validate the step when description is OK
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
NEMO – Data description (1)
•
•
•
•
Select the parameter list to use : P011 via P021
Add the parameters : P021 codes are
– Depth : AHGT, Temperature : TEMP, Salinity :
PSAL, Fluorescence : FVLT, Dissolved oxygen :
DOXY, Input the units of the parameters (for
salinity, use Dimensionless)
– Add a text label to each parameter
Set Start and End positions for the parameters
Tell NEMO which code is the vertical reference
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
NEMO – Data description (2)
•
•
•
•
Format is mandatory
– Describe the output format that you want
– %6.1f : means 6 digits total (decimal point included),
1 digit for decimal, f for float (decimal number)
– Use the contextual menu ‘Select the format’ for
predefined format
Use the test fields to validate your input
Validate the step when description is OK
Save you model in the model directory
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
File conversion
•
•
•
•
•
You will be asked to give :
A dataset reference (can be a cruise name, for
example)
A name for the output file (all the stations in
the same file) or name for the output
directory (one station per file) in the
SDN tools\Practical work\output directory
Have a look to the output files
Drag and drop the ODV files in ODV4 to test
their conformity
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
NEMO and MIKADO interaction– CDI summary file
NEMO and SDN Download Manager – coupling table
Training course – Ostende – 2-6 July 2012
While converting …
•
•
•
NEMO is able to
Generate a SeaDataNet CDI Summary
– Text file containing the minimum mandatory
information needed in the CDI ISO-19115 description
– This file can be converted to an Excel file
– Which can be read by MIKADO to generate the XML
CDI files
Generate a coupling table that will be used by the
Download manager of SeaDataNet
– It is the link between the LOCAL_CDI_ID and the file
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
CDI summary
•
Further information must be added in
NEMO settings (menu Options of
NEMO)
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
CDI SUMMARY File
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Interaction with MIKADO
Collection
of
ASCII files
ODV
files
XML
CDI files
summary_CDI_NEMO.xml
CDI summary
CSV file
[email protected] – www.seadatanet.org
MIKADO
SeaDataNet
CDI
Training course – Ostende – 2-6 July 2012
Coupling table for Download Manager
•
Further information must be
added in NEMO settings (menu
Options of NEMO)
– The coupling table makes the link
between each LOCAL_CDI_ID and
the file(s) which contains it
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Coupling table
• If the “Generate mapping” option is clicked :
• NEMO will insert a record in the coupling table for
each vertical profile, time series or trajectory that
it converts.
• The coupling table is managed by a local
database imbedded in NEMO
• The coupling table can be :
– Edited (for modification, insertion or deletion)
– exported (to be used by the download manager)
– imported (from previous version of NEMO, for example)
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Coupling table content (1)
LOCAL_CDI_ID
Modus
1 : mono-station
3 multi-station
Filename
Date of creation
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Coupling table content (2)
•
One unique LOCAL_CDI_ID in one mono-station ODV file
•
The same LOCAL_CDI_ID in 2 different files :
– One multi-station MEDATLAS file
– One mono-station ODV file
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Practical work on NEMO
Generating CDI summary and coupling table
M. Fichaut
Training course – Ostende – 2-6 July 2012
From ASCII data to XML files
Collection
of
ASCII files
ODV
files
XML
CDI files
summary_CDI_NEMO.xml
Delivered with NEMO
Coupling table
Export
CDI summary
CSV file
Coupling table
[email protected] – www.seadatanet.org
MIKADO
SeaDataNet
CDI
Training course – Ostende – 2-6 July 2012
Creation of the CDI-SUMMARY and the
coupling table
•
Input the information in NEMO SETTINGS
– Mapping
• Click in the box Generate mapping for SeaDataNet
download Manager
– SeaDataNET CDI summary
• Unfold the SeaDataNet CDI Summary block
• Click in the box Generate SeaDataNet CDI summary
• Fulfill all the mandatory fields (in red), using right click for
the Search function
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Run the file conversion
•
•
•
•
•
•
Same file than in the previous exercise
You will be asked to give :
A dataset reference (can be a cruise name, for example)
A name for the CDI summary file
A name for the output file (all the stations in the same file)
or name for the output directory (one station per file)
An output directory prefix (for the coupling table). This
prefix will be subtracted from the file name in the mapping
table.
For example if
– output file name =
C:\username\NEMO\cruise_name\file_name
– and output directory prefix = C:\username\NEMO
– file name in the mapping table will be :
cruise_name\file_name
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
CDI summary file
•
Convert it to an EXCEL file (for future use by MIKADO)
– Open the CDI summary text file with EXCEL (field
delimited by tabulations)
– Rename the spreadsheet into : STATION
– Save it under XLS or XLSX format, it will be used
later on during the practical work on MIKADO
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Coupling table
•
•
Have a look to the coupling table
– Menu : Coupling_table>Edit
– It gives the link between the LOCAL_CDI_ID and the file in
which the LOCAL_CDI_ID will be found by the download
manager of SeaDataNet
– MODUS = 1 if the LOCAL_CDI_ID is in a mono-station file
– MODUS = 3 if the LOCAL_CDI_ID is in a multi-station file
– One LOCAL_CDI_ID and one format is unique in the coupling
table
– Add and delete records
Export the coupling table in a flat file
– Menu : Coupling_table>Export
– Have a look at the flat coupling.txt file
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO
Tool for the generation of XML descriptions of
SeaDanaNet catalogue records
Training course – Ostende – 2-6 July 2012
XML validator
SeaDataNet
Vocabulary
CSR
Metadata
In
database
New
metadata
Metadata
in Excel
files
MIKADO
MIKADO
Meta data
at XML
ISO19155
EDIOS
Coupling
table
Data in
database
New
data
Data in
ASCII file
Format X
NEMO
EDMED
Collection
of ASCII files
at
SDN Format
EDMERP
CDI
Local copy of
data for
downloading
[email protected] – www.seadatanet.org
Data centres: SeaDataNet nodes
S
E
A
D
A
T
A
N
E
T
P
O
R
T
A
L
AAA
Data request
Downloading
RSM
European portal
Training course – Ostende – 2-6 July 2012
Objective

MIKADO is used to generate XML catalogue descriptions, it creates
XML ISO-19115 files using SDN common vocabularies for metadata
exchange of
 CSR - Cruise Summary Reports
 EDMED - Marine Environmental Data sets
 CDI - Common Data Index
 EDMERP - Marine Environmental Research Projects
 EDIOS – Permanent Ocean-observing System
 SensorML and O&M for seismic data
 Planned Cruise Report (not used in SeaDataNet)

Is freely available on SeaDataNet Web site
http://www.seadatanet.org/standards_software/software/mikado
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Technical characteristics
•
•
•
Release 2.3
Written in Java Language (Version >= 1.6)
Available under multiple environments :
– Microsoft : Windows 2000, WINDOWS7, XP, VISTA , APPLE
– Unix - Solaris
– Linux.
•
•
Interactive and batch modes available
Use of the SeaDataNet common vocabularies
web services to update lists of values
– needs network connection in order to have up to date
lists of values.
– www.seadatanet.org
– [email protected]
Mikado works offline
once the lists are up-to-date
Training course – Ostende – 2-6 July 2012
MIKADO main features (1)
•
•
MIKADO can be used in 2 different ways
– One manual way, to input manually information for
the catalogues and CDI in order to generate XML
files.
– One automatic way, to generate XML descriptions
automatically, from information catalogued in a
relational database or in an Excel file. Automatic way
is needed for those who have many entries
referenced in a relational database
Only one interface for all catalogues
– Same look
– Same principles
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO main features (2)
Manual
MIKADO
Java code
Automatic
XML files
for SeaDataNet
catalogues
CSR
EDMED
EDMERP
CDI
EDIOS
DATABASE
JDBC
Java DataBase Connectivity
Native Drivers
MYSQL
ORACLE
POSTGRES
SYBASE
MSServer
Bridge Drivers
using Microsoft ODBC
ACCESS, EXCEL, SQL SERVER, Open Office
Other Drivers
Downloaded from ad hoc Websites
[email protected]
www.seadatanet.org
(Copied in –the
dist/lib MIKADO directory)
EXCEL File
Training course – Ostende – 2-6 July 2012
MIKADO and SDN vocabulary lists (2)
List
code
List Name
List
code
List Name
C16
SeaDataNet Sea Areas
L061
SeaDataNet Platform Classes
C77
ICES ROSCOP data types
L071
SeaDataNet data access mechanisms
C174
SeaDataNet CSR ship metadata
L081
SeaDataNet Data Access Restriction Policies
C320
ISO countries
L101
SeaDataNet geographic co-ordinate reference frames
C371
Ten-degree Marsden Squares
L111
Height and Depth Vertical Co-ordinate Reference Datum
C381
Ports Gazetteer
L181
ROSCOP sample quantification units
L05
SeaDataNet device categories
L231
SeaDataNet metadata entities
L021
SeaDataNet Geospatial Feature Types
L241
SeaDataNet data transport formats
L031
SeaDataNet Measurement Periodicity Classes
P021
BODC Parameter Discovery Vocabulary
EDMERP
European marine projects
P081
SeaDataNet Parameter Disciplines
EDMO
European marine organisation
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO and SDN vocabulary lists (2)
Automatic check of the version
of the vocabulary lists : once
when MIKADO starts
– If “On” is clicked in the
Vocabulary Update Menu
– MIKADO downloads locally
the latest version of each list
Possible to enable-disable the
automatic check
– If “Off” is clicked
Manual check
– Update once now
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO – Manual input
•
•
•
•
Available for 5 catalogues : EDMED, CSR, CDI, EDMERP, EDIOS
(Programme, series and platform)
Each input generates an XML file that can be sent to the central
catalogue
For EDMERP and CSR : EDMERP CMS and CSR online can also be
used, but MIKADO is useful
– if you have problems with the NETWORK connection
– if you want to keep locally an XML description of your catalogues
For EDMED, EDIOS and CDI, there is no online input tools.
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO – Manual input
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO manual : LOCAL Identifier
•
•
•
This LOCAL ID exists for all the catalogues and is
under the responsibility of the data centre who
generates the XML descriptions
The LOCAL ID must be unique for a catalogue and
a data centre (EDMO code)
The LOCAL Identifier is vital because it is kept in
the central catalogue and is the entry point to know
if the record is new or if it is an update.
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO manual – vocabulary lists (1)
•
•
Common
vocabulary BODC,
list C320 (Country
ISO codes)
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO manual – vocabulary lists (2)
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO – Automatic XML generation
•
Principle
•
Read the information about CSR, EDMED, EDMERP, EDIOS or
CDI in a database or in an Excel file
MIKADO has predefined variables which correspond to the XML
tags definition for each catalogues
MIKADO helps user to write the SQL orders to fulfill these
variables with the information available in the database or in the
Excel file
•
•
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO – Automatic XML generation
•
4 STEPS
•
•
Connect to a database or an Excel file and test the connection
Write the queries to retrieve information in the database or in the
Excel file, test the queries
Save the queries in a “Configuration file”
Generate the XML files using the “Configuration file”
•
•
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO – Automatic XML generation
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO automatic - connection
•
•
•
Help for the
connection to
the database
Pre-filled
information for
some
databases
Check of the
connection
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO automatic – connection OK
• Green
message
in the
Check
box
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO – automatic – connection KO
• Red
message
in the
Check
box
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO automatic – queries
Expendable trees
Main query
Return the LOCAL ID
Single subqueries
Return 1 row
Multiple subqueries
Return 1 to n rows
Single an multiple
queries related to each
LOCAL ID returned by
the main query.
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO automatic – queries
Write the queries
SQL syntax (for
Oracle, Excel,
MySQL, …) and SQL
variables must be
adapted to your own
data base
Check the
Queries
Green OK
Red KO : read the
error message
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO automatic – single queries
All the XML variables
are listed in the
expendable tree
• In bold : mandatory
fields
• 1 to n single subquery
can be written
• Green ticks: fields
already fulfilled
• Add or delete
variables in a query
• Delete a full query
• Check the query
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO automatic – multiple queries
All the XML variables
are listed in the
expendable tree
• Number of queries is
pre-defined
• The list of variables
for each of these
multiple queries is
also pre-defined
• In bold : mandatory
field
• Green ticks : fields
already fulfilled
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO automatic – multiple queries
•
•
In a group of
variables (same
XML block,
same predefined set of
variable in a
query), the non
mandatory
variables can
be left to null
Check all
function
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO automatic - Save the queries
•
•
When all the
queries are
written
Saved in an
XML file
(configuration
file) to be reused later on
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO automatic - generate the XML files
1. Select the
catalogue you
want to generate
2. Open the
corresponding
configuration file
3. Choose the output
directory
4. Choose the type
of export files
5. Export the XML
files
• Progress bar
• Cancel
allowed
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO automatic - local mapping
•
•
•
•
While generating the XML files for all the catalogues
Each time that MIKADO does not recognized a value
(entrykey or entryterm) which should come from the
common vocabulary, it asks the user for mapping
MIKADO manages a demand-driven continuous
(incremental) extension of a local mapping : mapping of
the local database to the common vocabulary
Mapping tables can be modified
– Delete rows
– Modify the LOCAL value
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO automatic - local mapping
Example
CSR
generation
• Mapping of
the platform
type
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO automatic - local mapping
Modification of
the local
mapping
If wrong entries
have been input,
it is possible to:
• Delete one entry
• Delete all the
entries
• Change the local
code
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO in batch mode
• MIKADO can be run in batch mode using existing
configuration files
• Several arguments can be added on the command
line
• Java –Djava.endorsed.dirs=”dist/lib” –jar
dist/Mikado.jar mikado-home=[path] argument2= …
argumentn=
• Log file to register the errors
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Coupling file for Download manager
•
•
MIKADO is able to generate this coupling file
The coupling file is used by SeaDataNet download
manager to make the mapping between a
LOCAL_CDI_ID (one profile, one time-series or one
trajectory) and
– the name of the file containing this LOCAL_CDI_ID
(MODUS1 and 3) : if the metadata is in a data base and
the data in files or
– the SQL Query to retrieve the meta data and the data
of this LOCAL_CDI_ID in the local database (MODUS
2)
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Coupling file for Download manager (2)
• The principle to create this coupling file is
the same than to create XML files for
catalogue descriptions
• User has
– to create a configuration file that will be used
for the generation of the coupling file.
– to write the queries to retrieve the filename or
the data for each LOCAl_CDI_ID
• Retrieve the filename: if the data are stored as files
• Retrieve the data: if the data are stored in a database
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Coupling file generation
• 3 STEPS
• Create a configuration file
• Connect to a database or an Excel file and test the connection
• Write the queries to retrieve information in the database or in the
Excel file, test the queries
• Save the queries in a “Configuration file”
• Generate the coupling table using the
“Configuration file”
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Configuration file for coupling
•
Create a new
one
– From nothing
– Using a
MIKADO CDI
configuration
file
•
Open an
existing one
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Configuration file for coupling
•
•
Connection to the
database or excel
file
Main query to
retrieve all
LOCAL_CDI_IDs
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Configuration file for coupling: MODUS 2
•
•
•
•
Data are stored in a
database
Write the query to
retrieve the data in
the database
– Temperature
– Salinity
– Oxygene …
Test your query
Give the name of the
mapping file used by
DM to generate ODV
files
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Configuration file for coupling: MODUS 1 or 3
•
Data are stored in
files
– Modus 1: on the shelf
– Modus 3: to be
converted by the
Splitter
•
•
•
•
•
Choose the file format
Write the query to
retrieve The adress of
the file
Test your query
Preview the coupling
file
Save your
configuration file
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Generation of the coupling table
•
Using the
configuration
file for
coupling
– Select the
configuration file
– Select the output
directory
– Select the write
mode
• Append
• Overwrite
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Generation of the coupling table
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO – User manual
•
•
•
User manual is provided :
File : SDN_MIKADO_UserManual_V2.3.pdf
Detailed explanation for MIKADO use, lots of
snapshots
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
MIKADO next release: version 3
• ISO 19139 for CDI and CSR
– CDI available, to be finalised
• BODC vocabulary, version 2
– New services to be included
– Backwards compatibility with old XML files : Read of old
URN and conversion to new ones
• Pre-requirements
– Importation tools must be ready for ISO-19139
catalogues and must manage BODC V2 vocabulary
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Coupling table : NEMO or MIKADO?
• Depending of your data organisation
– If data in data base
• MIKADO
– If metadata in database, and address of the file in a
specific table
• MIKADO is recommended
– If no database, data in flat files generated using NEMO
• NEMO
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
[email protected] – www.seadatanet.org
Training Workshop – Ostende – 1-4 March 2010
Training course – Ostende – 2-6 July 2012
XML validation tool
Web service for validation of the XML files generated
by MIKADO
Training course – Ostende – 2-6 July 2012
XML validator
SeaDataNet
Vocabulary
CSR
Metadata
In
database
New
metadata
Metadata
in Excel
files
MIKADO
MIKADO
Meta data
at XML
ISO19155
EDIOS
Coupling
table
Data in
database
New
data
Data in
ASCII file
Format X
NEMO
EDMED
Collection
of ASCII files
at
SDN Format
EDMERP
CDI
Local copy of
data for
downloading
[email protected] – www.seadatanet.org
Data centre
S
E
A
D
A
T
A
N
E
T
P
O
R
T
A
L
AAA
Data request
Downloading
RSM
European portal
Training course – Ostende – 2-6 July 2012
•
Select the
catalogue
you want
to validate
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
•
•
•
View all CDI
schema
versions
View statistics
about
detected
errors
Run the
validation with
the last
version of
XML schema
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
•
•
Upload
the file
you want
to validate
Validation
works
only file
per file
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
• The file is
displayed
on the
screen
• Press
Validate
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
• Errors and
warning are
displayed
• Warning are
not obstacles
for XML
delivery
• Errors must be
corrected
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
VALIDATION of XML files
• XML validation Services have been developed in
the frame of SeaDataNet by the Russian NODC
• It is a Web validation Service available at
http://www.seadatanet.org/validator
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
[email protected] – www.seadatanet.org
Training Workshop – Ostende – 1-4 March 2010
Training course – Ostende – 2-6 July 2012
Practical work on MIKADO
Creation of XML files for CDI
M. Fichaut
Training course – Ostende – 2-6 July 2012
Exercise 1
a.CDI Automatic generation from
the CDI summary file created this
morning with NEMO – Validation
of the XML files
b.Creation of the corresponding
coupling file
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
From ASCII data to XML files
VALIDATOR
a
b
Collection
of
ASCII files
ODV
files
Coupling
file for
DM
XML
CDI files
summary_CDI_NEMO.xml
Delivered with NEMO
Coupling table
Export
CDI summary
CSV file
Coupling table
[email protected] – www.seadatanet.org
MIKADO
SeaDataNet
CDI
Training course – Ostende – 2-6 July 2012
a) Configuration file for CDI XML
•
•
Run MIKADO
Open the configuration file delivered with NEMO :
– MIKADO => Automatic/Open/CDI
the configuration file is in the HOME_DIRECTORY of NEMO,
but for the training it is under \Practical work\config , file
summary_CDI_NEMO.xml
•
•
•
•
•
Make the connection with the Excel file that you have
created with NEMO (the Excel file must be closed to avoid
conflict access)
Check the connection (you must have a green message)
Go to the Queries tag and have a look at the queries
Check them (green message after check all)
Save your configuration file
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
a) XML file generation
•
Run Automatic/Generate/CDI
– Open the configuration file
\Practical work\config\summary_CDI_NEMO.xml
• Choose the export directory (where CDI XML files will
•
be written)
– Choose the type of export file you want :
• XML files
• Zip file containing the XML files
• Both (XML and Zip)
Open one XML export file to have a look at it with MIKADO
Manual>Open>File>CDI
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
a) Web validation service: Validator
•
•
•
•
•
•
Check the XML file with the Web validation Service at :
http://www.seadatanet.org/validator
Select the CDI catalogue
Validate file with CDI Xml schema
Upload one of the XML file that has been generated
Validate the file
Look at the result of the validation : you must have no error,
you can have some warnings depending on the vocabulary
versions.
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
b) Coupling configuration file
•
•
Run MIKADO
Open the configuration file delivered with NEMO :
– MIKADO => Tools/Coupling table for download manager/ Import
CDI configuration
the configuration file is the same than in a) ) in the config directory
•
Check the connection to the excel file
\Practical work\config\CDI_summary_with_address.xls
•
•
•
•
•
Go to the main query tag and check the query (you must have
a green message)
Go to Modus 1 or Modus 3, choose the output format and
fulfill the query which retrieve the file address in the excel file
Check it
Preview the coupling file
Save your coupling
configuration
file
[email protected]
– www.seadatanet.org
Training course – Ostende – 2-6 July 2012
b) Coupling file generation
•
Run the generation of the coupling file
– MIKADO => Tools/Coupling table for download manager/
generate/ Modus 1 or Modus 3 depending on your ODV file(s)
•
•
•
•
•
Select the configuration file for coupling that you just create
Select the output directory
Choose the mode for writing (Append or Overwrite)
Generate the coupling file
Open it with a text editor
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Exercise 2
• EDIOS manual input of
– 1 Program
– 1 Series
– 1 Platform
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Under SDN_tools\practical_work
•
•
Under \EDIOS__POSEIDON_Example\EDIOS_Data
Open the EDIOS_POSEIDON.xls file describing the EDIOS program,
series and platforms (1 program, 15 series,15 platforms)
• Input the program information into MIKADO and create the
corresponding XML file
• Input 1 series information into MIKADO and create the corresponding
XML file
• Input the platform related to the previous series in MIKADO and create
the corresponding XML file
•
Why can’t you first enter a platform, then a series then a
program?
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Exercise 3
• EDIOS Automatic generation of XML
files for:
– Program
– Series
– Platforms
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Under SDN_tools\practical_work
•
•
•
\EDIOS__POSEIDON_Example\EDIOS_Data
Open the EDIOS_POSEIDON.xls file describing the EDIOS program, series
and platforms (1 program,15 series, 15 platforms)
Run MIKADO to create the automatic query to retrieve the program
information
– Or open the already written XML configuration for MIKADO for EDIOS program :
\EDIOS__POSEIDON_Example\EDIOS_Data\queries\poseidon_edios_program.xml
•
Run MIKADO to create the automatic query to retrieve the series information
– Or open the already written XML configuration for MIKADO for EDIOS program :
\EDIOS__POSEIDON_Example\EDIOS_Data\queries\poseidon_edios_series.xml
•
Run MIKADO to create the automatic query to retrieve the program
information
– Or open the already written XML configuration for MIKADO for EDIOS platform :
\EDIOS__POSEIDON_Example\EDIOS_Data\queries\poseidon_edios_platform.xml
[email protected] – www.seadatanet.org
Training course – Ostende – 2-6 July 2012
Generate the XML EDIOS files
• EDIOS PROGRAM
– Run MIKADO : Automatic>Generate>EDIOS Program
• Choose the \queries\poseidon_edios_program.xml
• Choose the export directory \results\program
• EDIOS SERIES
– Run MIKADO : Automatic>Generate>EDIOS Series
• Choose the \queries\poseidon_edios_series.xml
• Choose the export directory \results\series
• EDIOS PLATFORMS
– Run MIKADO : Automatic>Generate>EDIOS platforms
• Choose the \queries\poseidon_edios_platforms.xml
• Choose the
export directory \results\platforms
[email protected]
– www.seadatanet.org