Transcript AE Presentation

Metadata Driven Data Services
for SOA
Paul Anderson
Technical Sales Director
5 April 2016
Agenda
•
•
•
•
•
2
SOA Review
Data Integration
Data Services
Use Cases
Wrap-Up
Overview
Data architecture and its
management are evolving
toward an data services
management system that
delivers information as a
service.
3
SOA Timeline
• Personal involvement since 1998
– “Candle's Roma product of 1998 is the ESB's most
direct ancestor" (Roy Schulte, Gartner)
• SOA concepts have existed for a long time
– Business needs did not drive SOA adoption
– Fragmented technology slowed adoption
• Gained widespread acceptance in last 5 years
• Web Service Technology has enabled broad
adoption of the concept
4
What is SOA?
• Gartner
– "Web services are about technology specifications, whereas SOA is a
software design principle.” (Yefim V. Natis)
• SOA Concepts
– Decoupling of service production from consumption
– Service Interchange
– Service Discovery
• SOA is an architectural pattern
– The architectural pattern has/does appeared in many guises
– ESB tools represents the most widely accepted toolset for SOA
implementation
– Just because you use an ESB does not mean you have an SOA
architecture
• SOA benefits from the platform neutrality of Web Services
– Enables interchange of services
– Requires decoupling
– Provides discovery
• The changing business environment is driving SOA
– The scope of business functionality is changing
5
SOA Business Drivers
• Business Process enhancements
–
–
–
–
Migration of customer management to relationship management
Risk Management across/between organizations
Order/Shipment tracking across/between organizations
Tighter Partner Management
• Cross-Sell / Up-sell
– Based on history/demographics/relationship/value
• Business Agility
–
–
–
–
Services are re-usable building blocks
Cataloging of available services
Enhanced time to market
Focus on information rather than data
• Integrated Businesses
– Transactions span multiple organizations
6
Characteristics of SOA
• Requires Horizontal integration
– Scope crosses lines of business boundaries
– Scope crosses corporate boundaries
• Requires Horizontal understanding
– Requires understanding of multiple lines of business
– Requires development of common vocabulary
• Require concept generalization
– Exploitation of commonality
– Drilldown into specifics
• Requires easy access to information
– Need to remove the complexity of differing semantics,
location, and access methods
7
SOA often slowed by
• Vertical Fragmentation
– Differing technologies across verticals
– Differing semantics across verticals
• Infrastructure Fragmentation
– Application fragmentation
– Hardware fragmentation
– Data fragmentation
• LOB focus
• Multiple technologies
• Replicated data
• Organizational fragmentation
– Ownership Issues
– Organizational issues
– Governance issues
8
Common Misconceptions #1
SOA is an technology issue:
• Web Services are a suite
technology specifications
• You can implement a web
service based stove pipe
• SOA does not have to be web
service based
• Looks like an arcitectural
diagram but is just a list of
standards
9
Common Misconceptions #2
SOA is an application issue:
• Focuses on services but ignores
the underlying data
• Characteristics of an application
layer are very different from the
characteristics of a data layer
• Focuses on reuse of
functionality and ignores reuse of
data
10
Common Misconceptions #3
An ESB tool is all you need:
11
Common Misconceptions #3 (Continued)
• Fragile implementation
– Data access is embodied in the business process
– Adding an additional data source requires business process
change
– The data services are source specific
• Poor performance
– Data reduction is likely happening in the Business Process layer
– XML well suited for business process layer but not as well suited
to the data services layer
• Unmanageable
– Difficult to audit data access
– Difficult to secure data access
12
SOA reality
• SOA is fueled by data
– SOA combines People, Machines
– SOA embodies Business Process
– SOA is based on data
• SOA is fueled by metadata
– You can only reuse what you know exists
– You can only reuse what you understand
• SOA based applications need to co-exist
with existing applications
13
Data Integration Implementation Evolves Toward Data
Services – Gartners View
Processes
Applications
Business
Services
Get customer
Close account
Archive history
Users
Calc lifetime value
Get single view of product
Request for
data operations:
(Type, Format,
Latency, Quality)
Metadata
Data
Services
Access
Data
14
Response:
(Data or
Metadata)
Sync
Aggregate
Profile
Data
Transform
Move
Data
Deriving Value From Enterprise Information

Leading organizations incorporate information assets
into their portfolio management process.
– Treat information as an organizational asset
• Optimize availability and quality
• Balance accessibility vs. security
• Improve information supply chain performance
and integration
– Squeeze more value out of information
•
•
•
•
Optimize and eliminate overlapping processes
Improve enterprise performance
Trade information for goods and services
Strengthen enterprise relationships (partners,
employees, customers/constituents, suppliers)
 Information must be managed as a currency — save it, invest it, spend it,
manage it and account for it.
15
Agenda
•
•
•
•
•
16
SOA Review
Data Integration
Data Services
Use Cases
Wrap-Up
Enterprise Information Problem
The current state of corporate data
customers
sales
supplier data
?
Data Services
Disparate
financial
billing
Information
Silos
accounts
shipping
customers
Enterprise Information
Consumers
17
Enterprise Information
Sources
Data Integration Challenges
• Organizations cannot address basic questions
–
–
–
–
–
What information do I have?
Where is it?
How do I access the most current information?
How do I manage it?
What is the impact of change during and after
implementation of my system?
• Information is stored in myriad data stores
across the Enterprise.
– Different formats/data types
– Different structures/semantics
– Different access methods (API’s)
18
Data Integration Landscape
EAI - Target is the application





EAI
CRM
ERP
ETL - Target is the database
Reporting /
Dashboard
DW /
ODS
EAI
CRM
ERP
Even Driven data movement between apps
Data movement pre-wired
Message based.
Requires application coordination / workflow.
Requires complex translation / programming.
ETL






Scheduled extraction of data
Bulk loading of data warehouses.
Batch driven, Large Volumes.
Inflexible, unable to meet business agility needs
Historical data analysis.
One-way data movement, read-only access.
Data Services - Target is the end-user
Web
Services
BI
BAM
Apps
DS
DW /
ODS
EAI
CRM
19
ERP
ETL








On-demand delivery of information
Real-time integration of disparate data.
Universal data access layer.
Integrates multiple data sources.
Push / Pull any data across the enterprise.
Single View of Customer.
Supports dynamic / evolving reporting.
Supports / Enables SOA / Web Services.
Integration Technologies
Data
Integration Style
Process
Data Integration Timeliness
Real Time
Data
Services
EAI
ETL
Batch
20
What if……….
Web Services,
Custom
Applications Business Processes
Packaged
Applications
EAI, ESB, BPM
Reporting,
Analytics
Enterprise Data
Services
<sale/>
<value/>
</ sale >
Client sees single database
containing enterprise data
Clients are freed from the
specifics of underlying sources
21
Virtual Data
Each client has their own
desired “view”
Clients issue queries against
their “view”
Enterprise Data Services
ESB/
Portals
Applications /
Dashboards
Reporting /
Analytics
• High Performance Infrastructure
• Universal data access layer.
• Integration of structured/unstructured data.
• Data access security enforcement.
Web Service
Web Service/
Relational
Relational
• Insulates data consumers from sources.
• Real-time integration of disparate data.
• Federation of operational & historical data.
• Provides a virtual representation of data.
• Extract & Combine information on-demand.
• Supports dynamic reporting / dashboards.
• Read & Write to any data source.
APPS
22
DBMS
FILES
XML
DW
Model-driven Data services
Understand, relate, harmonize, rationalize, and use
EAI, Data warehouses
Enterprise Information
Data Server
geo-spatial
Packaged Apps
real-time
discover,
access,
update
<sale/>
<value/>
</ sale >
real-time
access & update
services
Virtual Data Server
warehouses
Web Services,
Business Processes
package, deploy, version
databases
Metadata
Catalog
spreadsheets
Custom Apps
Reporting, Analytics
model,
describe,
relate
share, version,
manage, discover
model,
describe,
relate
xml
rich media
…
Modeler
23
<sale/>
<value/>
</ sale >
Agenda
•
•
•
•
•
24
SOA Review
Data Integration
Data Services
Use Cases
Wrap-Up
SOA Challenges
• Majority of data is held in relational sources
– Relational structure optimized for LOB usage
– Probably do not want the existing data structure to drive data
service structures
– Performance concerns may rule out XML federation
• Granularity Issues
– Granularity of data needed to support SOA may be different
than current use cases
– Typically there will be data reduction moving from existing data
sources to data service presentation
• Web-Service data is XML based
– Data is organized hierarchically
– Need to map from relational to Hierarchical
• Scope of application typically wider that traditional
applications
– Spanning lines of business and/or corporate boundaries
25
SOA Challenges (continued)
• Performance
– XML based data is exceptionally verbose
– Exposing existing data in XML clearly not viable
• Security
– Broader access to data drives security concerns
– Applications that span political boundaries may be
subject to local confidentiality regulations
– May need to integrate data security with application
security infrastructure
• Auditability
– Regulations demand access audit trails
26
What is a Data Service?
• Decouple data sources from
application
– Data implementation shielded
from application
SOAP/XML
• Semantic/Format Mediation
Data Service
– Standard vocabulary
• Single access point
SQL
SQL
– Web Service/XML
– SQL
• Federation
• Scalability
– Security, performance
28
Bridge the
Gap
Master
Data
Divisional
Operational
Application
API
Call
SAP
Data Service Layer in SOA
Client Process & Applications
App
App
App
App
App
App
Business Process Services
Business Services
Message Services (ESB)
Data Service
Data Service
Data Service
Data Service
Data Services Layer
Data Sources
29
Data Service
Data Service
Obtain XML From Non-XML Sources
GIVEN:
Data Sources containing
Information to integrate
GIVEN:
Fixed XML Schema
WANT:
Data complying to schema
«XML»
<customerPositions>
<accounts>
<account ID=…>
…
</account>
</accounts>
</customerPositions>
30
«Relational»
NEED:
Mapping from
Data to XML
«Application»
?
<<XML Doc>>
XML Modeling to achieve business agility
The Problem:
Solution:
EIS Data
XML Document
EIS Data
<X>
<X>
XML Document
…
T
</X> <X>
</X>
<X>
…
XML
Schema
31
</X>
</X>
T
Data Service Design
Bottom
Up
WSDL
WSDL descriptor
Web Svc
Operation
Web service operation
XSDs - in/out
XSD
IN
XSD
OUT
<X>
XML views
Relational views
Source models
Start
32
Import sources
VDB Container
Three ‘typical’ use scenarios:
• Bottom-up: expose relational-style sources in ‘table-like’ form
• Business view: starting from XSD/XML-based business views
• Top-down: starting from WSDL definition of Web service operations
Data Service Design
Business
View
WSDL
WSDL descriptor
Web Svc
Operation
Web service operation
XSDs - in/out
Start
XSD
OUT
<X>
XML views
Relational views
Source models
Import sources
33
XSD
IN
VDB Container
Three ‘typical’ use scenarios:
• Bottom-up: expose relational-style sources in ‘table-like’ form
• Business view: starting from XSD/XML-based business views
• Top-down: starting from WSDL definition of Web service operations
Data Service Design
Top
Down
WSDL
WSDL descriptor
Start
Web Svc
Operation
Web service operation
XSDs - in/out
XSD
IN
XSD
OUT
<X>
XML views
Relational views
Source models
Import sources
34
VDB Container
Three ‘typical’ use scenarios:
• Bottom-up: expose relational-style sources in ‘table-like’ form
• Business view: starting from XSD/XML-based business views
• Top-down: starting from WSDL definition of Web service operations
Data Service Design
Bottom
Up
Business
View
Top
Down
WSDL
WSDL descriptor
Start
Web Svc
Operation
Web service operation
XSDs - in/out
Start
XSD
OUT
<X>
XML views
Relational views
Source models
Start
35
XSD
IN
Import sources
VDB Container
Three ‘typical’ use scenarios:
• Bottom-up: expose relational-style sources in ‘table-like’ form
• Business view: starting from XSD/XML-based business views
• Top-down: starting from WSDL definition of Web service operations
Data Services – Design Time
Metadata
Modeler
Metadata
Reports
• Model and Manage metadata.
• Combines technical & business metadata
• Enrich models with custom metadata.
• Infer relationships.
• Infer ownership.
• Maintain consistent data dictionary
Metadata Relational View
• Understand overlaps in business areas
Security
• Reduce time to perform impact analysis.
Virtual Databases
• Metadata open standards
Metadata
Repository
Runtime
Metadata
Design time
Metadata
• MOF
• XMI
• Import metadata from :
• Rational Rose
• ERWIN
• System Architect
UML
APPS
36
DBMS
XML
DW
Data Services – Run Time
MetaData
Modeler
Console
(Admin)
Query
Builder
• Model driven Integration
• GUI based modeler
• Metadata Repository
• Data Connectivity via
• SQL-92
• JDBC
• ODBC
• SOAP/HTTP
• SOAP/JMS
• Robust security and access control
• Row & column level
• Supports existing authentication solutions
Security/Audit
Virtual Databases
• Distributed query optimization & processing
Metadata
Repository
Query Processing /
Optimise Engine
Data Cache
Connector Framework
APPS
37
DBMS
FILES
XML
DW
• Data Caching vs. Real Time
• Connectivity to
• Databases
• ERP
• MOM
• Data Warehouses
• Apps / Legacy
• Files
Data Services benefits
• On-demand information
– Real time data integration
– Information sharing between business units
• Federation of disparate Information
– Structured, unstructured
– Relational + XML + Enterprise Apps + Legacy
• Faster time to market
– Integrated information in days, weeks
– Tight coupling of design & implementation phases
– Leveraging the skill-set of the data architects for integration
• Costs across application lifecycle reduced
– Model-driven abstraction layer between information sources and
applications eases development and maintenance
38
Agenda
•
•
•
•
•
39
SOA Review
Data Integration
Data Services
Use Cases
Wrap-Up
Typical Use Cases
•
Financial Services
– Market Reference Data
– Risk Management: BIS - Basel II
– Transaction Monitoring: Anti-money Laundering, Patriot Act
•
Supply Chain
– Visibility
– Reporting
•
Customer Service
– Single View of Customer
– Account Aggregation & Cross-marketing
•
Financial Reporting
– Corporate Governance & Compliance
– Sarbanes-Oxley, Executive Dashboard
•
Homeland Security
– Watch Lists
40
Securities Reference Data
Challenge
•
•
•
Securities data is inconsistent
Consistent pricing & other data needed for
better trades & accounting
“Hard-coding” was too expensive and
inflexible
Front-End Applications
IBM MQ Series
Solution
•
•
•
•
Models define common “view” of securities
data
Transform data to XML formats
Publish to pub/sub system
Support over 50 front-end applications
MetaMatrix Server
ROI
•
•
•
41
Consistent data = better trades, better
accounting, better risk management
New data sources online faster (weeks)
$3 million in IT savings
Security
Price
Security
Price
Security
Description
Implementing Web Services & Service Oriented Architecture
• Business Drivers
– Information sources available and discoverable as locationindependent Services on the CSFB Network
– CSFB Lines of business have access to information when and where
it is needed
– Web Services Loosely Coupled IT Architecture
• Technology Use Case – Universal Data Services
– Web Service consumers integrated to discoverable Services
– Encapsulate existing/legacy functionality
• Solution Benefits – ROI
–
–
–
–
–
42
70% of IT Budget spent on integrating CSFB systems
Found new uses for old data
Squeezed more value out of legacy systems
Embrace heterogeneity
Increase business agility
Single View of Customer
Challenge
•
•
•
•
•
Call center application
Full view of customer interaction
Data distributed across multiple systems
Data inconsistent
Need a enterprise-wide data layer for more
applications
Call Center
Application
MetaMatrix
MetaMatrix Solution
•
•
•
•
•
Real-time access
Virtual views of data
Federated queries across systems
Common re-usable data definitions
Enterprise-wide data layer for re-use
ROI
• Better service of customers
• Lower data integration costs
43
Data Sources
Information
Consumption
Conceptual Architecture
Application
Integration
XML
XML / Binary
XML /
Binary
XML
XML / Binary
Metadata Management
Information Bus
Data Quality
Information Services
Enterprise Information Integration
Extract, Transform and Load
Information Transport & Protocols
Information Assets
Near Real Time
44
Rules
Data
Marts
Data
Warehouse
ODS
(structured)
Content
(unstructured)
LOB
App(s)
LOB
Data
Data
Staging
External
Data
Meta
Data
Agenda
•
•
•
•
•
45
SOA Review
Data Integration
Data Services
Use Cases
Wrap-Up
Recommendations
• Start small, incrementally grow toward
information as a service
• Build the infrastructure on a project by project
basis
• Data security must be a top priority
• Ensure performance as you go
• Keep it in SOA context — information fabric is a
means to an end, promoting information as a
service
• But ability to access and use information is a
longer-term vision
46
Thank You
MetaMatrix
Paul Anderson
Technical Director
508 720 9266
[email protected]
48