Transcript Using SDMX for data reporting, publishing and modelling

Statistical Data Exchange
Platform
MarketMap Analytic Platform
Agenda
 Introduction to MarketMap Analytic Platform
 Sample Statistical Data Exchange Platform
 Economic Data Manager
 SDMX Driven Roadmap
2
MarketMap Analytic Platform
USERS
MarketMap Analytic
Language
C
Available 3rd Party Interfaces
Forecasting, Analysis &
Modeling Environment
Onsite
Server
• Fast DB for time series
data storage
• Built in Analytical platform
• Time intelligence
Web
Services
Web
Reports
APPLICATIONS
Managed Data
Services
Data
Loaders
Out-of-the-box development
and user interfaces
SQL
Access
Pathfinder Cross
Symbology
MarketMap Analytic Platform
3
Key, Value Pair Time Series Data Storage
 Vector object database with
coupled analytical engine
 Store, retrieve, and manipulate
large numbers of rapidly
accessible facts
‒ ibm.close
‒ sp500.TotalReturn
‒ PCT(s779255.sales)
Btree
Data
 Apply a structured programming
language geared towards
manipulation of vector objects
4
Analytic layer with an embedded concept of time




Regular frequencies: events recorded millisecondly, daily, monthly, etc
Pattern frequencies: specialized, but regular (e.g. market hrs)
Aperiodic: for event-driven data capture (trades, corporate actions)
Able to automatically convert frequencies
GLOBAL
FORMULAS
Business
Monthly
Quarterly
Return
Automatic Frequency Conversions
Monthly
Monthly
Monthly
Monthly
5
Multiple Analytic Databases house all required data
This MarketMap Analytic
database persists historical
market data objects
This MarketMap Analytic
database persists historical
macro economic objects
6
MarketMap Economic Data Manager
 Extract, transform and load data stored in Excel workbooks
 Dual database system
‒ Time series data stored in FAME databases
‒ Meta data about these series stored in SQL container
 Web Access layer used to report and graph time series data and
also display descriptive data about the series
7
Economic Data Collected in Excel Workbooks
8
Common "access point" to Time Series & Metadata
Relational Database
Client Applications
Client Applications
MAP
Web Access
Server
HTTP Request
Service Request
Web Browser
Business Process
Downstream
MAP Database
Web
Service
(WSDL)
HTML / CSV / XML
Time IQ / Result Set
Remote
MAP
Web
Access
Server
Output
Providers
Proprietary Database
9
Statistical Data Warehouse Use Case
10
Sample statistical production process
Statistics Production Environment
Collect
Compile
Disseminate
publications
web
ESCB-Net/EXDI
Production system (FAME)
SDW
ESCB-net
SDMX Data Model
11
11
Data structures
 Statistical data can be grouped together at
‒ the observation level (the measurement of some phenomenon);
‒ the series level (the measurement of some phenomenon over
time, usually at regular intervals);
‒ the group level (a group of series – a well-known example being
the sibling group, a set of series which are identical, except for
the fact that they are measured with different frequencies); and
‒ the dataset level (made up of several groups, to cover a specific
statistical domain for instance).
 Dimensions are grouped into keys, which allow the
identification of a particular set of data (a series, for example).
 Key values are attached at the series level and given in a
fixed sequence. Partial keys can be attached to groups.
12
12
Example: Monetary aggregate M3
BSI.M.U2.N.V.M30.X.I.U2.2300.Z01.A
 BSI = Key Family,
 M = Monthly series,
 U2 = euro area aggregate,
 N = Non-adjusted,
 V = MFIs + Govt,
 M30 = M3,
 X = Unspecified maturity,
 I = Index,
 U2 = residence of counterpart is euro area,
 2300 = other residents sector,
 Z01 = denominated in all currencies,
 A = Annual growth/change
13
13
Use of a (SDMX-based) structured data format
 In the exchange, storage and dissemination of all statistical data
and associated metadata
 In the internal system and the communication with partner
institutions and the general public (web services, SDMX-ML
based extractions from the web site)
 Covering most domains of economic statistics (e.g. monetary
and financial statistics, balance of payments, price indices,
short-term statistics, real sector, government finance statistics,
securities, etc.)
14
14
Browsing for data series
Organize object names and categories based on the SDMX
standard
15
Demo: Key family search
16
Demo: Context sensitive selection boxes
17
Demo: Report on the January 2006 data
18
Demo: Detailed CPI search results
19
Demo: Select objects of interest
20
View objects in monthly frequency
21
Demo: View selected objects in annual frequency
22
Demo: Multiple ways of aggregating data
23
Demo: PCT
24
Demo: Graph selected objects
25
Demo: Administer user entitlements
26
SDMX Oriented Roadmap
MarketMap Analytic Platform
27
Central Bank Use Case
 Using FAME since 1993 to not only store time series but also
to produce statistics from primary to derived information, to
disseminate statistical information, internally and on the
internet, in different ways (statics, and dynamically) and formats,
and, to exchange statistical information with national and
international organizations using the SDMX standard.
 Core tools
‒ Hundreds of functions and procedures using Fame/4GL, TimeIQ
and the C-HLI
‒ Working with FAME 9.3 and migrating to FAME 10.1
28
High priority database improvements
 Integration of FAME vectors, formulas, functions and procedures
with the SDMX information model and SDMX ML format
‒ Specific objects to manage data structural definitions (DSD)
information
‒ Referential integrity between series names and the DSD
‒ Creation of a matrix object to deal with observation-level attributes
 New second sub-index referencing the attribute so that attribute series
values and their observation level attributes can be stored together
 Concurrent database updating
‒ Current update mechanism locks database at the database level
which prevents users from updating data at the same level
concurrently
29
High priority end user improvements
 Friendly menu-driven environment for end users & developers
‒ Take a page from SAS, eViews and others who provide a menu
driven experience for building statistical studies
‒ Take a page from Eclipse or Visual Studio .NET who provide an
integrated development environment complete with syntax wizards,
auto completion facilities and debuggers
 Attribute searching facilities
‒ Wildcard searching on the information stored in attributes that
perform with the same level of efficiency as the searching facilities
for series name and series alias
30
Statistical Data & Metadata eXchange
 SDMX is not just tagging of data with XML. Its intent is to be a
standard for data interrogation (SDMX Registry) and common format
for publishing (SDMX XLM output) .
 SDMX Data – Query Response provider in Web Access
‒ Included a POC SDMX registry.
‒ The Metadata for a few Forex based times series using SDMX
structures were placed in this registry by hand. The Registry was
implemented in MySQL.
‒ The SDMX Custom Service provider implemented the SDMX V1.0
standard.
‒ It included SDMX supported query response. The idea here is you hit
the MYSQL that has the SDMX Registry to discover what data is
available at that site.
‒ Create a second call that had the SDMX Data information. That
second call for the data was in SDMX xml form and the CSP received
that request, looked at the SDMX Registry which had a hard mapping
to the Forex time series.
‒ The response was sent back using SDMX xml V1.0 format.
31