Transcript SDMX ISTAT FRAMEWORK

SDMX ISTAT FRAMEWORK
Francesco Rizzo (ISTAT - Italy)
GENEVE 10 -11 May 2007
OECD SDMX Expert Group
SDMX Istat
Framework
Summary
Why Istat is starting to use SDMX
SDMX Istat Framework – an overview
SDMX data Web Service (demo)
SDMX Web Navigator (demo)
Geneva 10-11 May 2007
OECD SDMX Expert Group
SDMX Istat
Framework
Why Istat is starting to use SDMX
The Istat Information System is made up of several
production Directorates sub-systems that, independently
one from another, cover the full life cycle of statistical data,
from collection to dissemination.
Production Directorates
Data Collection
Data Editing
Data Aggregation
metadata
thematic
DB
web navigator
Geneva 10-11 May 2007
OECD SDMX Expert Group
web navigator
Integrated Output Management System project
Production Directorate
Data Collection
Data Editing
Production Directorate
Data Collection
Data Editing
Data Aggregation
Data Aggregation
metadata
validated
microdata
thematic
DB
Integrated Output
Management System
thematic
DB
web navigator
web navigator
web navigator
web navigator
Geneva 08-10 May 2007
Meeting on the Management of Statistical Information Systems
SDMX Istat
Framework
Why Istat is starting to use SDMX
For this reason Istat is following the evolution of the SDMX
initiative with interest :
Tactically:
through the participation in the EUROSTAT SODI task
force
Strategically:
through the building of a special working group whose
main objective is to analyze and verify the use of SDMX in
the internal architecture of ISTAT Information System
Geneva 10-11 May 2007
OECD SDMX Expert Group
SDMX Istat
Framework
Why Istat is starting to use SDMX
The participation in the EUROSTAT SODI initiative has
involved Istat to gain experience in the use of XML and
Web Services technologies for integration purpose inside
Statistical Information Systems.
In order to facilitate this objective, we are developing
several software modules organized in a framework
(current version is 1.0)
Geneva 10-11 May 2007
OECD SDMX Expert Group
SDMX Istat
Framework
SDMX Istat Framework version 1.0
Characteristics:
• compatible to SODI needs
• useful to introduce SDMX for both internal purposes and
external information exchange
• can be used entirely from the reporting phase to the
dissemination phase, or alternatively using modules
separately, integrating them into one information system
• in the future it might be possible to distribute the
framework under Public Licence
Geneva 10-11 May 2007
OECD SDMX Expert Group
Reference
metadata
manager
Reference metadata RSS
Reference metadata SDMX Query
Reference
metadata
web navigator
Reference
metadata web service
Reference
metadata
database
STS
ESA
mapping
Word extractor
IMF
Eurostat
SDMX web navigator
STS-ESA
time series
database
SDMX data
web service
.ist
gesmes
.dat
Loader
Check
RSS builder
&
query builder
ConIstat modules
new modules
SDMX Queries
SODI RSS
fix formatted record file
under construction modules
GENEVE 10-11 May 2007
ConIstat
web navigator
OECD SDMX Expert Group
SDMX Istat
Framework
SDMX Istat Framework version 1.0
Overview of changes in the Loader module:
New dissemination functions were added in order to satisfy
some SODI requirements:
• to publish an RSS;
• to publish on or more SDMX Query file(s);
• to publish one or more SDMX Compact file(s)
(optional)
Geneva 10-11 May 2007
OECD SDMX Expert Group
SDMX Istat
Framework
SDMX Istat Framework version 1.0
Overview of the SDMX Data Web Service module:
XML
validate
SDMX Query
SDMX Compact
query
Parser
SDMX
Compact
builder
sql query
builder
Database
Response-time test:
an entire Dataflow containing 685 time series, 120 months
long (developing 82.200 observation) < 20 seconds
demo
Geneva 10-11 May 2007
OECD SDMX Expert Group
SDMX Istat
Framework
SDMX Istat Framework version 1.0
Overview of SDMX Web Navigator module:
This module acts as a client towards the web service and
represents a web graphic interface that allows the following
functions:
• building a SDMX Query using DSDs as analysis
dimensions
• sending the SDMX Query to the web service for
querying the database
• Testing the SDMX Queries that a user can build
starting from SDMX Queries created with the graphic
interface
demo
Geneva 10-11 May 2007
OECD SDMX Expert Group
SDMX Istat
Framework
SDMX Istat Framework version 1.0
Overview of Manager and Navigator SDMX reference
metadata module:
Database
Production
Directorates
Reference
metadata
manager
XSLT
XML
Reference
HTML
metadata
web navigator
Reference
metadata
web service
Word
extractor
Geneva 10-11 May 2007
OECD SDMX Expert Group
SDMX
Eurostat
IMF
SDMX Istat
Framework
SDMX Istat Framework version 2.0
A generalized SDMX Query parse module (beta version):
This module accepts only queries with DataWhere section
but also accept queries regarding:
• different Dataflows
• subsets of different DataFlows
A SDMX Query is processed by the following steps:
• rewrites the XML stream using Boolean logic rules, and adding to
each node an attribute that acts as a “unique key”. The scope of the
unique key is to set a hierarchy between “parent nodes” and “child
nodes”;
• converts the XML stream in a memory tabular data structure
whose columns represent all types of nodes (Time, Dimension,
Attribute, Dataflow) except for And and Or;
• re-organizes the in-memory tabular data structure so that all
columns have an And relation, and all rows have an Or relation;
• converts the in-memory tabular structure in a SQL query
Geneva 10-11 May 2007
OECD SDMX Expert Group