Transcript MIKADO tool - SeaDataNet

3rd SeaDataNet training course – Ostende – 16-19 June 2008
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Tools for data management in
the frame of SeaDataNet
project
M. Fichaut
www.seadatanet.org
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
SeaDatanet
Vocabulary
Metadata
in
Database
Metadata
Input
Metadata
In
Excel files
XML Validator
MIKADO
XML
Metadata
Files
MIKADO
CSR
Coupling
table
Data in
Database
Data Input
EDMED
Collection
Collection
NEMO
of
of
ASCII files
ASCII files
Format SDN
Format X Med2MedSDN
EDMERP
CDI
ODV
www.seadatanet.org
S
E
A
D
A
T
A
N
E
T
Local copy
of data
to download
Partner system : pilot data centre
Data request
Data download
P
O
R
T
A
L
European portal
2
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
SeaDatanet
Vocabulary
Metadata
Input
Metadata
In
Excel files
MIKADO
Metadata
in
Database
XML Validator
MIKADO
XML
Metadata
Files
Data in
Database
Data Input
CSR
EDMED
Collection
Collection
NEMO
of
of
ASCII files
ASCII files Med2MedSDNFormat SDN
Format X
EDMERP
Data request
by email
CDI
ODV
www.seadatanet.org
S
E
A
D
A
T
A
N
E
T
Manual
preparation
of data
Local copy
of data
to download
Partner system : other data centre
P
O
R
T
A
L
Data download
European portal
3
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 Overview (1)
• MIKADO
• Objectives and technical characteristics of Mikado
• Mikado’s main features
• Manual input of metadata
• Use of the common SDN vocabularies
• Automatic XML generation
• Connection to the local database
• Query writing
• Mapping to the local database
• Batch mode
• Coupling table for download manager
• XML VALIDATION SERVICE
www.seadatanet.org
4
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 Overview (2)
• Med2MedSDN
• Objectives and main features
• NEMO
• Objectives, main features and principles of NEMO
• Technical characteristics
• Description of the different steps to follow to be able to reformat
ASCII files to SeaDataNet format
• Link between NEMO and MIKADO
• Link between NEMO and SDN Download Manager – coupling table
www.seadatanet.org
5
3rd SeaDataNet training course – Ostende – 16-19 June 2008
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
MIKADO
Tool for the generation of XML
descriptions of SeaDanaNet
catalogue records
www.seadatanet.org
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 Objective
 MIKADO is used to generate XML catalogue descriptions, it
creates XML ISO-19115 files using SDN common vocabularies
for metadata exchange of
 CSR - Cruise Summary Reports
 EDMED - Marine Environmental Data sets
 CDI - Common Data Index
 EDMERP - Marine Environmental Research Projects
 [EDIOS – Permanent Ocean-observing System]
 Is freely available on SeaDataNet Web site
http://www.seadatanet.org/standards_software/software/mikado
www.seadatanet.org
7
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 Technical characteristics
• Written in Java Language (Version >= 1.6)
• Available under multiple environments :
• Microsoft : Windows 2000, XP, VISTA , APPLE
• Unix - Solaris
• Linux.
• Use of the SeaDataNet common vocabularies web services
to update lists of values
• needs network connection in order to have up to date
lists of values.
• but Mikado works offline once the lists are up-to-date
www.seadatanet.org
8
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO main features (1)
MIKADO can be used in 2 different ways
• One manual way, to input manually information for
the catalogues and CDI in order to generate XML
files.
• One automatic way, to generate XML descriptions
automatically, from information catalogued in a
relational database or in an Excel file. Automatic way
is needed for those who have many entries
referenced in a relational database
Only one interface for all catalogues
• Same look
www.seadatanet.org
• Same principles
9
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO main features (2)
Manual
MIKADO
Java code
Automati
c
XML files
for SeaDataNet
catalogues
CSR
EDMED
EDMERP
CDI
[EDIOS]
DATABASE
JDBC
Java DataBase Connectivity
Native Drivers
MYSQL
ORACLE
POSTGRES
SYBASE
MSServer
www.seadatanet.org
Bridge Drivers
using Microsoft ODBC
EXCEL File
(ACCESS, EXCEL, SQL SERVER)
Other Drivers
Downloaded from ad hoc Websites
(Copied in the dist/lib MIKADO directory)
10
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO and SDN vocabulary lists (2)
List
code
List Name
List
code
List Name
C16
SeaDataNet Sea Areas
L061
SeaDataNet Platform Classes
C77
ICES ROSCOP data types
L071
SeaDataNet data access mechanisms
C174
SeaDataNet CSR ship metadata
L081
SeaDataNet Data Access Restriction Policies
C320
ISO countries
L101
SeaDataNet geographic co-ordinate reference frames
C371
Ten-degree Marsden Squares
L111
Height and Depth Vertical Co-ordinate Reference Datum
C381
Ports Gazetteer
L181
ROSCOP sample quantification units
L05
SeaDataNet device categories
L231
SeaDataNet metadata entities
L021
SeaDataNet Geospatial Feature Types
L241
SeaDataNet data transport formats
L031
SeaDataNet Measurement Periodicity Classes
P021
BODC Parameter Discovery Vocabulary
EDMERP
European marine projects
P081
SeaDataNet Parameter Disciplines
EDMO
European marine organisation
www.seadatanet.org
11
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO and SDN vocabulary lists (2)
•Automatic check of the version of
the vocabulary lists : once when
MIKADO starts
• If “On” is clicked in the
Vocabulary Update Menu
• MIKADO downloads locally the
latest version of each list
•Possible to enable-disable the
automatic check
• If “Off” is clicked
•Manual check
• Update once now
www.seadatanet.org
12
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO – Manual input
• Available for 4 catalogues : EDMED, CSR, CDI, EDMERP
• Each input generates an XML file that can be sent to the
central catalogue
• For EDMERP and CSR : EDMERP CMS and CSR online can
also be used, but MIKADO is useful
• if you have problems with the NETWORK connection
• if you want to keep locally an XML description of your
catalogues
• For EDMED and CDI, there is no online input tools.
www.seadatanet.org
13
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO – Manual input
www.seadatanet.org
14
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO manual : LOCAL Identifier
• The LOCAL Identifier is vital because it is kept in
the central catalogue and is the entry point to know
if the record is new or if it is an update.
• This LOCAL ID exists for all the catalogues and is
under the responsibility of the data centre who
generates the XLM descriptions
www.seadatanet.org
15
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO manual – vocabulary lists (1)
Common vocabulary BODC,
list C320 (Country ISO
codes)
www.seadatanet.org
16
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO manual – vocabulary lists (2)
www.seadatanet.org
17
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO – Automatic XML generation
Principle
• Read the information about CSR, EDMED, EDMERP or
CDI in a database or in an Excel file
• MIKADO has predefined variables which correspond to
the XML tags definition for each catalogues
• MIKADO helps user to write the SQL orders to fulfill
these variables with the information available in the
database or in the Excel file
www.seadatanet.org
18
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO – Automatic XML generation
4 STEPS
• Connect to a database or an Excel file and test the
connection
• Write the queries to retrieve information in the database
or in the Excel file, test the queries
• Save the queries in a “Configuration file”
• Generate the XML files using the “Configuration file”
www.seadatanet.org
19
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO – Automatic XML generation
www.seadatanet.org
20
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO automatic - connection
• Help for the
connection to
the database
• Pre-filled
information for
some
databases
• Check of the
connection
www.seadatanet.org
21
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO automatic – connection OK
• Green message
in the Check box
www.seadatanet.org
22
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO – automatic – connection KO
• Red message in
the Check box
www.seadatanet.org
23
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO automatic – queries
Expendable trees
• Main query
Return the LOCAL
ID
• Single queries
Return 1 row
• Multiple queries
Return 1 to n rows
• Single an multiple
queries related to each
LOCAL ID returned by
the main query.
www.seadatanet.org
24
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO automatic – queries
Write the queries
• SQL syntax (for Oracle,
Excel, MySQL, …) and
SQL variables must be
adapted to your own data
base
Check the Queries
• Green OK
• Red KO : read the error
message
www.seadatanet.org
25
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO automatic – single queries
All the XML variables
are listed in the
expendable tree
• In bold : mandatory
fields
• 1 to n single subquery
can be written
• In green : fields already
fulfilled
• Add or delete variables
in a query
• Delete a full query
• Check the query
www.seadatanet.org
26
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO automatic – multiple queries
All the XML variables
are listed in the
expendable tree
• Number of queries is
pre-defined
• The list of variables for
each of these multiple
queries is also predefined
• In bold : mandatory
field
• In green : fields already
fulfilled
www.seadatanet.org
27
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO automatic – multiple queries
• In a group of
variables
(same XML
block, same
pre-defined set
of variable in a
query), the non
mandatory
variables can
be left to null
www.seadatanet.org
28
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO automatic - Save the queries
When all the
queries are
written
• They can be
saved in an
XML file to be
re-used later
on
www.seadatanet.org
29
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO automatic - generate the XML files
1. Select the
catalogue you want
to generate
2. Open the
corresponding
configuration file
3. Choose the output
directory
4. Choose the type of
export files
5. Export the XML files
•
•
Progress bar
Cancel allowed
www.seadatanet.org
30
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO automatic - local mapping
While generating the XML files for all the catalogues
• Each time that MIKADO does not recognized a value (entrykey or
entryterm) which should come from the common vocabulary, it asks
the user for mapping
• MIKADO manages a demand-driven continuous (incremental)
extension of a local mapping : mapping of the local database to the
common vocabulary
• Mapping tables can be modified
• Delete rows
• Modify the LOCAL value
www.seadatanet.org
31
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO automatic - local mapping
Example
CSR generation
• Mapping of
the platform
type
www.seadatanet.org
32
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO automatic - local mapping
Modification of
the local
mapping
If wrong entries
have been input,
it is possible to:
• Delete one entry
• Delete all the
entries
• Change the local
code
www.seadatanet.org
33
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO in batch mode
• MIKADO can be run in batch mode using existing
configuration files
• Several arguments can be added on the command line
Java –Djava.endorsed.dirs=”dist/lib” –jar dist/Mikado.jar mikado-home=[path]
argument2= … argumentn=
• Log file to register the errors
www.seadatanet.org
34
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 Batch mode : mandatory arguments (1)
batch-type : XmlFiles, Zipfile, Both
batch-mode : CDI, EDMERP, CSR, EDMED
conf-file : path and name of the configuration file with the SQL
queries
output-dir : path and name of the output directory where the
XML files will be written
continue-on-error : true or false
If true : if one record with mapping missing, or one record with
mandatory field(s) null in the database, warning for this record in the
logfile, and MIKADO processes next records..
If false : MIKADO stops at the first error.
www.seadatanet.org
35
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 Batch mode : optional arguments (2)
log-file : Path and name of the Log file of MIKADO. By default,
mikado.log in Mikado home directory
trace : to have a summary of the time response of each SQL
query (useful for tuning of the queries bad if time responses)
max-file-in-zip : number maximum of XML files per zip file
generated by Mikado. By default, one zip file contains 1000
XML files.
zip-prefix : To personalize the names of the Zip files generated
by Mikado. By default the zip file name are
SeaDataNet_[catalogue]_[x].zip
www.seadatanet.org
36
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 Coupling file for Download manager (1)
• MIKADO is able to generate this coupling file
• The coupling file is used by SeaDataNet download manager to
make the mapping between a LOCAL_CDI_ID (one profile,
one time-series or one trajectory) and
• the name of the file containing this LOCAL_CDI_ID
(MODUS1 and 3) : if the metadata is in a data base and
the data in files
• the SQL Query to retrieve the meta data and the data of
this LOCAL_CDI_ID in the local database (MODUS 2)
www.seadatanet.org
37
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 Coupling file for Download manager (2)
• The principle to create this coupling file is the same than to create
XML files for catalogue descriptions
• User has
• to create a configuration file that will be used for the generation of
the coupling file.
• to write the queries to retrieve the filename or the data for each
LOCAl_CDI_ID
www.seadatanet.org
38
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 Coupling file for Download manager (3)
www.seadatanet.org
39
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 MIKADO – User manual
User manual is provided :
File : SDN_MIKADO_UserManual_v1_1_4.pdf
• Detailed explanation for MIKADO use, lots of
snapshots
www.seadatanet.org
40
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
ANY QUESTION?
www.seadatanet.org
41
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
 VALIDATION of XML files
• XML validation Services have been developed in the frame of
SeaDataNet by the Russian NODC
• It is a Web validation Service available at
http://www.seadatanet.org/validator
www.seadatanet.org
42
3rd SeaDataNet training course – Ostende – 16-19 June 2008
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
XML validation tool
Web service for validation of the
XML files generated by MIKADO
www.seadatanet.org
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
• Select the
catalogue
you want
to validate
www.seadatanet.org
44
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
• View all CDI
schema
versions
• View statistics
about
detected errors
• Run the
validation with
the last version
of XML
schema
www.seadatanet.org
45
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
• Upload the
file you
want to
validate
• Validation
works only
file per file
www.seadatanet.org
46
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
• The file is
displayed
on the
screen
• Press
Validate
www.seadatanet.org
47
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
• Errors and
warning are
displayed
• Warning are
not obstacles
for XML
delivery
• Errors must
be corrected
www.seadatanet.org
48
OBSERVATIONS
& PRÉVISIONS CÔTIÈRES
Training Workshop – Ostende – 26-29 October 2009
ANY QUESTION?
www.seadatanet.org
49