Transcript SDMX-RI
SDMX Reference Infrastructure
(SDMX-RI)
Work in progress, status and plans
Bengt-Åke Lindblad, Adam Wroński
Eurostat
Eurostat Unit B3 – IT and standards for data and metadata exchange
1
Content
What is SDMX-RI
Why use SDMX-RI
Usage status
Development strategy
Release calendar
Eurostat Unit B3 – IT and standards for data and metadata exchange
2
What is SDMX-RI
Universal framework for modern data provision
Set of pick-and-choose reusable building blocks
allowing a statistical office to expose data to the
external world based on access rights
Designed to provide data and structural metadata
based on mappings to each organization's
dissemination data warehouse
Uses SDMX standards incl. one for Web Services
Eurostat Unit B3 – IT and standards for data and metadata exchange
3
Eurostat Unit B3 – IT and standards for data and metadata exchange
4
Why SDMX-RI
Developed to simplify the exchange of data
Provides standard software and components,
allowing individual statistical organisations to
interact and exchange their data using the same
software and methodology
Modular approach, use part or the entire
infrastructure, extend it by adding new modules or
modify it in any other way to suit their own
purposes
Developed in both Java and .NET
Eurostat Unit B3 – IT and standards for data and metadata exchange
5
Usage
As of Sept 2012 deployed in 22 EU countries
Tested across EU Member States (2011
population census)
Running in Mexico
Expression of interest
Latin America, the Caribbean, OECD and Russia
July 2013:
expected to run in EU27 member states
in Eurostat dissemination Web Service upgrade
Eurostat Unit B3 – IT and standards for data and metadata exchange
6
SDMX-RI versions
SDMX-RI
Initial Solution
Streaming
SDMX-RI
Intermediate
Solution
Eurostat Unit B3 – IT and standards for data and metadata exchange
Common API
SDMX v2.1
SDMX-RI
Ultimate
Solution
7
Development strategy 2012-2013
Architectural changes are needed (two step approach)
Intermediate solution to solve performance & out of
memory errors
Ultimate solution to provide a common API and
implement SDMX 2.1
Implement new user requests and correct defects
Widen the scope and usage of SDMX-RI among data
providers
ESSVIP project such as Web INfrastructure for Statistics
Reuse for other statistical data collections
DSWS (Eurostat dissemination web service)
Eurostat Unit B3 – IT and standards for data and metadata exchange
8
Why an intermediate solution
Problems needed to be solved within short time for
existing users
Performance
High memory consumption and response times
constrained organizations from putting it in
production
Out Of Memory errors
Impossible to serve large data requests.
Eurostat Unit B3 – IT and standards for data and metadata exchange
9
What changed?
Streaming of data in the service
Migration to JAX-WS in Java
IO revised with Streaming Writers
Data Retriever API changed (streaming)
Eurostat Unit B3 – IT and standards for data and metadata exchange
10
Benefits
Better performance
Improvement up to 4 times faster for .Net
Improvement up to 7 times faster for Java
Solution to Out Of Memory problems for large
datasets
No memory constrains
Data are streamed to the client
Eurostat Unit B3 – IT and standards for data and metadata exchange
11
Impact
Organizations that have installed initial solution
Only re-install the Web Service Package
Existing users of Web Service package not affected
Same v2.0 interface remain
Organizations using the components API
API has changed due to streaming support
Migration required
Organizations has done modifications to the source code
Have to make the changes again if they want to use the
intermediate version.
Eurostat Unit B3 – IT and standards for data and metadata exchange
12
Why an ultimate solution?
Use of common API (that supports various versions and
flavors of SDMX)
Interfaces which allow implementation using the
components and also the interfaces for component-tocomponent communication
Better inter-organization component reusability
Ability to support new versions of SDMX without
changing our applications
Support of SDMX 2.1
New messages (data representation, queries)
New Interfaces (SOAP/REST)
Eurostat Unit B3 – IT and standards for data and metadata exchange
13
What will be changed?
All modules migrated to the common API
To use of new SDMX Model/IO
Components APIs will be changed
Support of SDMX 2.1 messages and new query features
Extend to support 2.1 standardized SOAP and RESTful APIs
New Web Service end points
Co-existence with v2.0 service
Support of v2.1 error codes
Eurostat Unit B3 – IT and standards for data and metadata exchange
14
Impact
Mapping Assistant users will be able to reuse their Mapping Stores
Mapping Store upgrade will be supported
Will be required to re-install MA only
Organizations that have installed initial solution
Only to re-install the Web Service Package
Existing users of Web Service package will not be affected
Same v2.0 interface will remain
Organizations using components API
API has changed due to streaming support
Migration will be required
Migration guidelines will be provided
Organizations done modifications to the source code
Will have to make the changes again if they want to use the
new version.
Eurostat Unit B3 – IT and standards for data and metadata exchange
15
Status - release
Common API development (jar/dll expected end 2012)
Java: ongoing
.NET: not started yet (API should be ported to C# and then
implemented)
SDMX 2.1 support (expected Q1 2013)
Not yet started (dependent on API)
Enhancements and bug corrections (independent)
Mapping Assistant (Sept 2012)
Improved transcoding, error handling, QueryEditor, etc
Mapping Assistant (end 2012)
Implementation of 11 request mainly linked to the user
interface and the usage
Test Client
using encrypted user id and password
Eurostat Unit B3 – IT and standards for data and metadata exchange
16
SDMX-RI contacts
[email protected]
[email protected]
Eurostat Unit B3 – IT and standards for data and metadata exchange
17
Web Service Package
extension to SDMX 2.1 and new interfaces
Web Service
Provider
SOAP
Request
v2.0
SOAP
Request v2.1
NSI_Service_2.0
Structure
Retriever
API
Implementation
REST
Request v2.1
NSI_Service_2.1
NsiRestService
Controller
Common
API
Eurostat Unit B3 – IT and standards for data and metadata exchange
Data Retriever
(streaming)
18