Transcript SDMX Registry - CIRCABC

6. SDMX:
A non-technical overview of the
SDMX architecture and IT tools
Raynald Palmieri
Eurostat
Unit B5: “Central data and metadata services”
1
SDMX Basics course, March 2016
Eurostat
The SDMX
Components
SDMX
Information
Model
Content
Oriented
Guidelines
IT Infrastructure
for exchange and
sharing
Eurostat
2
Describing the data exchange
Who?
When?
How?
What?
What's next?
Eurostat
Who?
EUROSTAT SDMX TOOLS FOR
DESIGNERS
4
Eurostat
Eurostat SDMX tools for designers
5
Tools offered
Use
Data Structure Wizard
Used to create, edit and test SDMX objects
Maintenance of SDMX registry
SDMX Registry
A central repository for storing and sharing
SDMX objects
Eurostat
Eurostat SDMX tools for designers
Data Structure
Wizard
Web service
Create/Edit
6
Eurostat
SDMX Registry
Store/Edit
Work roles and process
SDMX Registry
Interpret
SDMX
messages
Consult and
get DSD
COLLECTING
ORGANISATION
Publish DSD
Consult and get DSD
Create DSD
7
Send SDMX
messages
Format / structure
SDMX messages
Data Structure Wizard
• Creation / edition of DSDs
• Validation of DSD
• Local repository
• Artefacts are stored in a local repository
• Remote repository
• Can be used for retrieving existing artefacts
8
DSW: Management of SDMX artefacts
9
DSW: Artefact example: the codelists
10
SDMX Registry
• Metadata registry which implements the SDMX registry
specifications
• Tasked with providing structure, organisation,
maintenance and query interfaces for the SDMX
components required to support the data sharing vision
• Provides a web-based user interface as well as web
services for interacting with the SDMX structural metadata
objects in use within the collecting organisation and with
statistical partners.
11
Eurostat
SDMX Registry
• Only concerned with providing information need to access
the data and reference metadata sets.
• An application which wants a particular data or metadata
set would then query the registry for the structure, and
then go and retrieve the data or metadata set directly from
the provider's web server.
• Example of existing SDMX registries:
• Euro SDMX Registry (2.1)
• SDMX Global Registry (2.1)
12
Eurostat
New Euro SDMX Registry (2.1)
 Eurostat's implementation of the SDMX
Registry
 Based on SDMX 2.1 (as published on April
2011) Also capable of importing and exporting
SDMX 2.0 artefacts.
 Allows browsing, searching, editing and
subscribing to artefacts.
 Advanced access control mechanism of
artefacts controlling their visibility
(Public/Private).
Eurostat
New Euro SDMX Registry (2.1)
Home page
Access to the
content of the
Registry text
search
Access to the
content of the
Registry by type
Most recent items
Eurostat
New Euro SDMX Registry (2.1)
Registry Content
Filtering list
Number of objects
of type selected
Eurostat
SDMX Global Registry(SGR)
• Launched in 2011 by the SDMX Sponsoring
institutions
• Originally to support the world-wide technical
implementation of Balance of Payments and
National Account implementations
• To make metadata material (DSDs, concept
schemes, MSDs, code lists, etc.) publicly and
centrally available used and maintained by
national and international organisations.
Who should use the SDMX Global
Registry (SGR)?
• Any national or international agency that
wants to use SDMX standards for data
management or for exchanging data with other
organisations using SDMX standard.
• Any end-user of statistical data can access the
data structures posted on the SGR.
How to use the SDMX Global
Registry (SGR)?
• All information published in the SGR is publicly
available and freely viewable and
downloadable.
• No need to log in to view or to download
information - SDMX Global Registry Content.
• Users can view the full content of the Global
Registry
Exploring by agency
Available SDMX
objects
Click on a type
Click on Export to
choose a SDMX
object to download
Enter a text to be
found in the
Registry
Access to the web
service
Codelist Content
2. Select an object
1. Choose a DSD in
the list
Dimensions,
measure and
attributes
Corresponding list of
codes
Example of result
for "NA-MAIN"
5 versions found
Web Service Query
Select the SDMX version, the
Object type and the
identifiers of the SDMX object
(ID, agency, version)
Select which referenced
objects (e.g. Code lists) you
want to include
View the xml file in the
browser or Download the file
The URL generated is the SDMX
query to be sent to the web service
to get the xml file corresponding to
the selected object
View the xml file in your
browser (DSD NA_MAIN)
SDMX TOOLS FOR DATA PROVIDERS
25
Eurostat
SDMX Tools for Data Providers
Tools offered
Use
Action
Web Forms (EU)
Excel-like templates
Transmission of low
volumes of data
No costs for EU
organisations
Manual work for senders
(type, copy/paste)
SDMX Converter
(sender’s PC)
Converts data files
between SDMX formats
and other file formats
Installation on sender’s PC
Manual work for senders
(convert)
Installation on server
SDMX Converter
(batch mode)
SDMX Reference
Infrastructure
A set of tools that
Installation on server
allows to connect your
Mapping of database to
IT systems to the SDMX DSDs
world
Own SDMX
Implementation
26
Local development in
organisations
Eurostat
Data provider view:
One goal – different possibilities
Webforms
EDAMIS
Single Entry
Point
SDMX-ML file
Excel
sheets
SDMX Converter
SDMX-ML file
Database
export
SDMX Converter
SDMX-ML file
Database
SDMX Reference
Infrastructure
Eurostat
SDMX-ML file
SDMX Web Service
Same
DSD

Same
Format
Data provider view:
One goal – different IT Architectures
Webforms
Web-based
Push
mode
Excel
sheets
SDMX Converter
Local conversion
Push
mode
Database
export
SDMX Converter
Local conversion
Push
mode
Database
SDMX Reference
Infrastructure
Push
or Pull
mode
Eurostat
SDMX Converter
https://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/SDMX_Converter
• File based conversion
• Open source and platform independent (Java)
• Different ways of using it:
•
•
•
•
Graphical user interface
Batch file (server or client side)
Web Service interface
Reusing source code in your own Java application
• Formats: SDMX-ML 2.0 & 2.1, SDMX-EDI
(Gesmes), FLR, CSV, Google DSPL, predefined
Excel templates
Eurostat
GUI
(Converter 5.2.1)
1. Selection of the
input/output files and
their format
2.a Select the
DSD in the local
drive
2.b Identify a DSD to
download from the
SDMX Registry
(configuration required)
2. If the local DSD
includes multiple
versions, we can
specify the one
desired
2.c Identify a dataflow
linked to the DSD to
download from the
SDMX Registry
(configuration required)
3. Excel parameter file
3. SDMX header
(.prop file)
CSV parameters
Only for flat and
excel files
4. Mapping and
Transcoding
6. SDMX (output)
validation
5. CSV quotation
XML parameters for
SDMX output formats
Eurostat
31
Eurostat
32
Eurostat
Exercice: conversions
• From CSV to SDMX Compact
• From XLS to SDMX Compact
• From SDMX Compact to CSV
33
Eurostat
What is SDMX-RI
• Framework for SDMX data provision
• Set of pick-and-choose reusable building blocks
allowing a statistical office to disseminate SDMX
data to the external world
• Designed to provide data and structural
metadata based on mappings to each
organization's dissemination data warehouse
• Uses SDMX standards including the one for Web
Services
34
Eurostat
Why SDMX-RI
• Developed to simplify the exchange of data
• Provides standard software and components,
allowing individual statistical organizations to
interact and exchange their data using the same
software and methodology
• Modular approach, users can
o use part or the entire infrastructure,
o extend it by adding new modules or modify it in
any other way to suit their own purposes
• Developed in both Java and .NET
35
Eurostat
SDMX Reference Infrastructure
Census-HUB architecture –
Eurostat to NSI
Web service
SDMX Query
(XML File)
SDMX
DataMessage
(XML File)
SDMX
-RI
National Statistics Institute
Eurostat Census Hub
Eurostat
Supported DBs
Oracle
•
•
•
•
•
Oracle
SQL Server
MySQL
Pc Axis
ODBC
SQL
Server
ODBC
Supported
DBs
PC
Axis
37
Eurostat
MySQL
SDMX-RI components
• Mapping Assistant
 Graphical tool to create the mapping between the
DSD and the dissemination database.
• Test Client
 Used to test your dataflow locally
• NSI Web Service
 Allows you to share your dataflow
• NSI Client
 Web interface to interact with the Web Service
38
Eurostat
Mapping
Store
DSD
Local
environment
39
Is my dataflow
working? Is my WS
working?
Test Client
(Desktop
Software)
Eurostat
NSI Client
Mapping
Web Service
Dissemin
ation DB
Data Consumer (external)
SDMX-RI process
Dissemination Database
• The SDMX-RI is configured to directly map the
dissemination Database
• An SDMX output is generated by the SDMX-RI
using the PULL mode (the Receiver Organisation
directly requests data and receives an SDMX file
with no intervention of the Sender Organisation)
40
Eurostat
Dissemination Database
41
Eurostat
Mapping Assistant
• Facilitates the mapping between the structural
metadata provided by an SDMX-ML Data Structure
Definition (DSD) and those that reside in a
database of a dissemination environment
• Maintains a Mapping Store for keeping the
mappings between the SDMX and the local data
storage scheme
• In the SDMX Reference Infrastructure, provides
mapping information to the Data Retriever
42
Eurostat
43
Eurostat
44
Eurostat
Eurostat
Transcoding
46
Eurostat
Test Client
• Test mappings of data flows configured through
the Mapping Assistant
• Test the NSI Web Service is correctly set up
• Export data sets to file in SDMX format
47
Eurostat
Eurostat
Web Client
• Web Interface to directly request data.
• Graphical view of the results
• Generates the output in different formats
49
Eurostat
NSI Web Client : Welcome screen
50
Eurostat
Eurostat
Support for data providers
• SDMX Reference Infrastructure
 Installed in ~32 countries for Census Hub
 Can be extended to other domains
• 1st contact point
-
Local Transmission Coordinators
 In National Statistical Institutes
 Experience with EDAMIS and SDMX
• 2nd contact point
-
Eurostat support teams
Eurostat
Tools Relationships – Example
Produce
Store
DSW
SDMX Registry
DSD
DSD
Produce
use
SDMX
Converter
SDMX-RI
SDMX-ML
Dataset
Mapping
Assistant
Produce
CSV
File
53
use
Eurostat
Mapping
National
DB
SDMX Tools – Summary
http://www.sdmxtools.org
• Eurostat tools  Eurostat SDMX Info Space
• SDMX Converter
• SDMX Reference Infrastructure & Mapping Assistant
• SDMX Registry & Data Structure Wizard
• Other tools available in the community
•
•
•
•
•
Fusion Weaver (validation, transformation, creation)
Fusion Transformer (command line conversion)
Fusion Registry
ECB Checker/Loader (conversion and validation)
FAO OpenSDMX project
Eurostat
Thank you for your attention!
Questions
Eurostat