Transcript SDMX-RI

SDMX Reference Infrastructure
(SDMX-RI)
Work in progress, status and plans
Bengt-Åke Lindblad, Adam Wroński
Eurostat
Eurostat Unit B3 – IT and standards for data and metadata exchange
1
Content





What is SDMX-RI
Why use SDMX-RI
Usage status
Development strategy
Release calendar
Eurostat Unit B3 – IT and standards for data and metadata exchange
2
What is SDMX-RI
 Universal framework for modern data provision
 Set of pick-and-choose reusable building blocks
allowing a statistical office to expose data to the
external world based on access rights
 Designed to provide data and structural metadata
based on mappings to each organization's
dissemination data warehouse
 Uses SDMX standards incl. one for Web Services
Eurostat Unit B3 – IT and standards for data and metadata exchange
3
Eurostat Unit B3 – IT and standards for data and metadata exchange
4
Why SDMX-RI
 Developed to simplify the exchange of data
 Provides standard software and components,
allowing individual statistical organisations to
interact and exchange their data using the same
software and methodology
 Modular approach, use part or the entire
infrastructure, extend it by adding new modules or
modify it in any other way to suit their own
purposes
 Developed in both Java and .NET
Eurostat Unit B3 – IT and standards for data and metadata exchange
5
Usage
 As of Sept 2012 deployed in 22 EU countries
 Tested across EU Member States (2011
population census)
 Running in Mexico
 Expression of interest
 Latin America, the Caribbean, OECD and Russia
 July 2013:
 expected to run in EU27 member states
 in Eurostat dissemination Web Service upgrade
Eurostat Unit B3 – IT and standards for data and metadata exchange
6
SDMX-RI versions
SDMX-RI
Initial Solution
Streaming
SDMX-RI
Intermediate
Solution
Eurostat Unit B3 – IT and standards for data and metadata exchange
Common API
SDMX v2.1
SDMX-RI
Ultimate
Solution
7
Development strategy 2012-2013
 Architectural changes are needed (two step approach)
 Intermediate solution to solve performance & out of
memory errors
 Ultimate solution to provide a common API and
implement SDMX 2.1
 Implement new user requests and correct defects
 Widen the scope and usage of SDMX-RI among data
providers
 ESSVIP project such as Web INfrastructure for Statistics
 Reuse for other statistical data collections
 DSWS (Eurostat dissemination web service)
Eurostat Unit B3 – IT and standards for data and metadata exchange
8
Why an intermediate solution
 Problems needed to be solved within short time for
existing users
 Performance
 High memory consumption and response times
constrained organizations from putting it in
production
 Out Of Memory errors
 Impossible to serve large data requests.
Eurostat Unit B3 – IT and standards for data and metadata exchange
9
What changed?
 Streaming of data in the service
 Migration to JAX-WS in Java
 IO revised with Streaming Writers
 Data Retriever API changed (streaming)
Eurostat Unit B3 – IT and standards for data and metadata exchange
10
Benefits
 Better performance
 Improvement up to 4 times faster for .Net
 Improvement up to 7 times faster for Java
 Solution to Out Of Memory problems for large
datasets
 No memory constrains
 Data are streamed to the client
Eurostat Unit B3 – IT and standards for data and metadata exchange
11
Impact
 Organizations that have installed initial solution
 Only re-install the Web Service Package
 Existing users of Web Service package not affected
 Same v2.0 interface remain
 Organizations using the components API
 API has changed due to streaming support
 Migration required
 Organizations has done modifications to the source code
 Have to make the changes again if they want to use the
intermediate version.
Eurostat Unit B3 – IT and standards for data and metadata exchange
12
Why an ultimate solution?
 Use of common API (that supports various versions and
flavors of SDMX)
 Interfaces which allow implementation using the
components and also the interfaces for component-tocomponent communication
 Better inter-organization component reusability
 Ability to support new versions of SDMX without
changing our applications
 Support of SDMX 2.1
 New messages (data representation, queries)
 New Interfaces (SOAP/REST)
Eurostat Unit B3 – IT and standards for data and metadata exchange
13
What will be changed?
 All modules migrated to the common API
 To use of new SDMX Model/IO
 Components APIs will be changed
 Support of SDMX 2.1 messages and new query features
 Extend to support 2.1 standardized SOAP and RESTful APIs
 New Web Service end points
 Co-existence with v2.0 service
 Support of v2.1 error codes
Eurostat Unit B3 – IT and standards for data and metadata exchange
14
Impact
 Mapping Assistant users will be able to reuse their Mapping Stores
 Mapping Store upgrade will be supported
 Will be required to re-install MA only
 Organizations that have installed initial solution
 Only to re-install the Web Service Package
 Existing users of Web Service package will not be affected
 Same v2.0 interface will remain
 Organizations using components API
 API has changed due to streaming support
 Migration will be required
 Migration guidelines will be provided
 Organizations done modifications to the source code
 Will have to make the changes again if they want to use the
new version.
Eurostat Unit B3 – IT and standards for data and metadata exchange
15
Status - release
 Common API development (jar/dll expected end 2012)
 Java: ongoing
 .NET: not started yet (API should be ported to C# and then
implemented)
 SDMX 2.1 support (expected Q1 2013)
 Not yet started (dependent on API)
 Enhancements and bug corrections (independent)
 Mapping Assistant (Sept 2012)
 Improved transcoding, error handling, QueryEditor, etc
 Mapping Assistant (end 2012)
 Implementation of 11 request mainly linked to the user
interface and the usage
 Test Client
 using encrypted user id and password
Eurostat Unit B3 – IT and standards for data and metadata exchange
16
SDMX-RI contacts
[email protected]
[email protected]
Eurostat Unit B3 – IT and standards for data and metadata exchange
17
Web Service Package
extension to SDMX 2.1 and new interfaces
Web Service
Provider
SOAP
Request
v2.0
SOAP
Request v2.1
NSI_Service_2.0
Structure
Retriever
API
Implementation
REST
Request v2.1
NSI_Service_2.1
NsiRestService
Controller
Common
API
Eurostat Unit B3 – IT and standards for data and metadata exchange
Data Retriever
(streaming)
18