Transcript PM4Data - GUIDE Share France – Groupe de travail WebSphere MQ

Powering Enterprise Process Advantage®
Metastorm
PM4Data
Managed File Transfer
Chris Howarth
2005 Metastorm, Inc.
[email protected]
Powering Enterprise Process Advantage®
WebSphere Business Integration
Reference Architecture
Development Platform
Business Performance Management Services
Interaction
Services
WBIC
Partner
Services
Process
Services
Information
Services
WBIMB WMQ ESB WMQ
PM4DATA and PM4CICS
Application and Data
Access Services
Business
Application
Services
Business Application and Data Services
Enterprise Applications and Data
Infrastructure Services
2005 Metastorm, Inc.
2
Powering Enterprise Process Advantage®
PM4Data is the File and Database Adapter for the
Enterprise Service Bus
Utilize MQ
Messaging to
access and
provide
managed
transfer facilities
for Files and
Databases
XML enable
Files,
Databases
WMQ & XML
Onramp
2005 Metastorm, Inc.
Portal
Service
SOAP
Service Request
(e.g. J2EE, .NET)
B2B Interactions
Service
Flow
Data
Existing
Applications
New
Service Logic
3
Powering Enterprise Process Advantage®
What is PM4Data?
• PM4Data is:
– A high-performance, highly scalable solution for “managed”
high volume data transfer of almost any type of data
between two or more distributed heterogeneous systems
(Managed File Transfer Solution).
– A solution that leverages the award winning IBM
transactional MOM (WebSphere MQ ®) for assured, high
performance, reliable and secured asynchronous delivery
of data.
– A extendible, customizable, reliable and scalable
foundation for a modern visible process-centric WMQ
based data transfer & integration solution.
– A powerful WBI companion and WMQ implementation
accelerant tool
– A WebSphere MQ and XML Power Tool.
2005 Metastorm, Inc.
4
Powering Enterprise Process Advantage®
Supported Platforms
• Mainframe z/OS (running OS/390 or z/OS)
• Unix
– AIX
– HP-UX
– Sun Solaris
– Linux
• OS/400
• Windows
– NT, 2000, XP, 2003
• Tandem NSK
• IBM 4690 Point of Sale System
– Optional “Continuous Extract Facility” for TLOG Trickle Feed
• Both WebSphere MQ Client or Server supported
2005 Metastorm, Inc.
5
Powering Enterprise Process Advantage®
Features and Functions Summary
High performance, secured, bulk data transfer solution that delivers real-time
and batch File or Database Integration capabilities.
Features
Usability
• Eclipse® based Workbench IDE for developing and
•Customizable process flows for end to end status
tracking.
testing multiplatform integration Scripts & XML Policies
• Accelerates file and message based integration.
• Centralized management
•XML-enabler (Files and Databases).
• Built-in Status and Statistics Tracking
•XML Control interface.
• Common multi-platform scripting language
•Distribution list for one-to-many transfers.
• Web based Command/Control Center and Status
• Transfer list support for many-to-many transfers.
• User profile and access control
• Service-oriented architecture.
• File and Database Integration Service
• Accelerates integration with IBM WMQ Broker
• File Transfer request repository
• Logical record processing
• Interactive retry / fail selection of failed transfers
•SSL based data encryption (WMQ SSL)
• Performance analysis reports and graphs
•Policy driven I/O:
• Native multi-platform Interface
• File or Database to Message
•TSO, JCL, Command, Message, Shell script,
bat/cmd
• Message to File or Database
• Simple file extract, transfer, and load capabilities.
• Directory Monitoring: Sense creation/arrival of
file to initiate a file transfer
• Highly scalable, both in volume and platform
capabilities
• Automated exception handling facility
2005 Metastorm, Inc.
6
Powering Enterprise Process Advantage®
PM4Data new features List in V8
– Central Registry
• Management and configuration artifacts
– MQ Dashboard
• Easily Monitor Enterprise MQ load and Usage
• Daily Summary reports
– Process Flow Editor to create customized Process Flows
– JMX based Systems Management
infrastructure
• JMX (Java Management Extensions) infrastructure and
support
– Web Browser Component Launcher Console
• LOG4J (STDOUT / STDERR) log management
• Email notifications for exception
• Alert Management integration to Tivoli Console
2005 Metastorm, Inc.
7
Powering Enterprise Process Advantage®
Managed Data Movement
Simplified Data Movement Scenario
Data broken
into messages
Message
Message
Message
Messages
received and
assembled
Target
platform
Source
Platform
End-to-end detailed transfer status




Far more sophistication, reliability and performance than putting a data in a
single message
Platform file differences handled automatically during transfer
Extended functionality with customizable exits
End-to-end data transfer visibility
2005 Metastorm, Inc.
8
Powering Enterprise Process Advantage®
PM4Data Architecture
Process
Monitor
User Database
Apache
Web Server
User
Service
Registry File Store
Registry
Service
Status Database
Status Query
Server
Status Logger
MQSeries Bus
PM4Data Node
File Transfer
Service
File Transfer
Service
File Transfer
Service
File Transfer
Service
2005 Metastorm, Inc.
Exit Processor
Directory Monitor
Service
9
Coordinator
Powering Enterprise Process Advantage®
PM4DATA
Files, Databases and Queue Integration Simple Example
•
Example
–
–
–
Source Data needs to be moved to one or more targets
This can be caused by application updates and sensed by PM4DATA
End-to-end visibility with Status and Notification
Queue
or
PM4Data
New or Append
file
Queu
e
DB
or
or
PM4Data
Database
update
Data transfer
request
Queu
e
File
or
PM4Data
New file
or
Database
update
2005 Metastorm, Inc.
10
Powering Enterprise Process Advantage®
Process Flow Editor
Business Provides the
“What needs to be
done”
IT provides the “How it
gets done”
Exception Handler
Routing
File Transfer
as a sub Flow
Activity
2005 Metastorm, Inc.
13
Powering Enterprise Process Advantage®
PM4Data Registry
Application1
– Central HFS based Artifact Registries
• Eclipse Workbench IDE
– Web Browser access for transfer request / policies
• Version and Change Control Support (CVS or similar
product)
• Customized or template based Node configuration support
• Support for “Application” or “Project” level Registry
• Support for MQ / TCP and File based registry drivers
• Support for Registry chaining and search order control
Messages
Process Registry
Contents
Programs
XMScript
XML
Request
Group1
Group2
Policies
<RegistryList>
Search Order
<registry location="C:\CQDemos\data\Registry">Emergency</registry>
<registry location="C:\CQDemos\data\Registry">Application1</registry>
<registry location="C:\CQDemos\data\Registry"> Application2 </registry>
<registry location="C:\CQDemos\data\Registry"> Application3 </registry>
<registry location="C:\CQ\data\Registry">CommerceQuest</registry>
</RegistryList>
Physical
Location
2005 Metastorm, Inc.
14
“Registry Name”
Typically used to
group artifacts for
a project or
“Application”
Powering Enterprise Process Advantage®
Transfer Process Monitor
•
•
•
•
•
Web Server support
for:
WebSphere,
JBOSS (included)
– Status
– Component
Configuration /
Management
(JMX)
– Component Logs
(LOG4J)
Status display
– End-to-End
transfer status
Access Control
– User
management,
with roles & rights
Transfer request
management
– Ping nodes
– Create, save, and
submit transfer
requests
– Transfer grouping
– Distribution List
Management
– Directory
monitoring
2005 Metastorm, Inc.
16
Powering Enterprise Process Advantage®
Conclusion
• Satisfies Enterprise Managed File Transfer requirements
• Strategic Component of the Enterprise Service Bus
Architecture
• Accelerates the On Demand Strategy
2005 Metastorm, Inc.
19
Powering Enterprise Process Advantage®
Mapping XML to XMObject’s
and
Vise Versa
<order>
<customer>
<name>I. Wannabuy</name>
<address> … </address>
</customer>
<item>
<description>Small Rodent, Generic</description>
<quantity>35</quantity>
<price>29.99</price>
</item>
<item>
<description>Catapult</description>
<quantity>1</quantity>
<price>149.95</price>
</item>
</order>
order
customer
name
address
…
item
•
XMOBJECT
item[0]
description
quantity
price
item[1]
description
quantity
price
2005 Metastorm, Inc.
20
Powering Enterprise Process Advantage®
Things that are important in XML
which XMObject takes care off
•
•
•
•
Well defined order semantics
– What is property order for new object?
– Where are new properties added?
– What is impact of deleting properties?
Operators for controlling order
– Specify property order
– Modify property order
– Preserve property order
Operators for creating and manipulating additional XML
artifacts
– Attribute
– Namespaces, Mixed content
Specialized methods for managing Objects.
– Creating, Serializing, Cursor-ring, Searching, Printing
– Renaming, Copying, Allocating and de-allocating
memory
– Encoding, Compressing, Code Page conversions
etc.
2005 Metastorm, Inc.
21
order
Type = “This order is …
customer
custid = “007”
name
address
…
item
item[1]
id = “342”
description
quantity
price
item[2]
Powering Enterprise Process Advantage®
XMObject’s Objective
• Provide a simple, familiar, general purpose programming
model for XML that:
– Leverages existing skills and knowledge
– Reuses familiar concepts, operators and syntax
– Flattens the learning curve
– Minimizes need for specialized skills and knowledge
– Enables developers immediately with little or no training
• Ultimately, provide a simple object abstraction for creating,
navigating and manipulating XML
2005 Metastorm, Inc.
22
Powering Enterprise Process Advantage®
Metastorm
DB XMObjects
2005 Metastorm, Inc.
Powering Enterprise Process Advantage®
The Data format dilemma:
The Fix = DB Objects
Payment
Order
Supplier Product
Account
Account
Customer
Product
Payment
Customer
Product
SQLServer
Oracle
DBObject
IMS
DB2
DBObject
Payment
Account
DBObject
Supplier
Account
AS/400
Payment
Product
Order
Customer
DBObject
<?xml version='1.0'?>
<Account>
<CustomerID>123451</CustomerID>
<ContactFirstName>John</ContactFirstName>
<ContactLastName>Smith</ContactLastName>
<BillingAddress>123 Front Street</BillingAddress>
<City>Toronto</City>
<StateOrProvince>Ontario</StateOrProvince>
</Account>
Customer
DBObject
Product
Z/OS
DBObject
ODBC
DBObject
IDMS
DBObject
NCR
Warehouse
2005 Metastorm, Inc.
24
Supplier
Powering Enterprise Process Advantage®
DB Objects Database Wrapper
• DB Objects is a database independent XMObject Interface
• Language and Platform independent API support, including XMScript
• Supports any ODBC accessible database
• XMObjects or XML formatted messages for input and output
• Native database access support included for:
• DB2 ( OS/400, Windows,Unix & OS/390)
• All Popular RDBMS via Data Direct Drivers On Unix Systems
• SQL Server (Windows)
• Allows for Database access via MQSeries network
• Reduces or eliminates ODBC client connections
• Instantly XML and Message enables almost any database
• Serves up data as XML from any database
• Turns Stored Procedures into “Message Enabled Components”
• Full read, update, create, merge, commit, backout & insert support
2005 Metastorm, Inc.
25
Powering Enterprise Process Advantage®
Metastorm
2005 Metastorm, Inc.
Process Manager for Data
Architecture Overview
Powering Enterprise Process Advantage®
PM4Data Architecture
Process
Monitor
User Database
Apache
Web Server
User
Service
Registry File Store
Registry
Service
Status Database
Status Query
Server
Status Logger
MQSeries Bus
PM4Data Node
File Transfer
Service
File Transfer
Service
File Transfer
Service
File Transfer
Service
2005 Metastorm, Inc.
Exit Processor
Directory Monitor
Service
27
Coordinator
Powering Enterprise Process Advantage®
Process Flow Manager (Coordinator)
1
WBI
Message
Broker
Request Message
Initiating
Application
Process Flow Definition
Status Tracking
Audit Sub System
2
Process Flow
Manager
3
5
Message
Message
6
4
Response
Business
Process
Definition
Repository
3
5
Message
Message
4
Response
Response
Synchronous External
Service, including WS, EJB
2005 Metastorm, Inc.
Asynchronous
Workflow
Enabled
Component
(response required)
Asynchronous
Workflow
Enabled
Component
(NO response req)
28
Synchronous In-line
Workflow Enabled
Service Program
Or Shared Library
Powering Enterprise Process Advantage®
Dispatch Manger for the Coordinator
Workflow
Workflow
Applications
Workflow
Applications
PM
Applications
Applications
Incoming
“XML” WMQ
Messages
UNIX / Windows / AS400 / ZOS
XML
Console
WSD
L
Console
Management
Thread
Dispatch
Manager
For
Coordinator
Coordinator
Management
Thread
Coordinator
Management
Thread
Coordinator
Instance
2005 Metastorm, Inc.
Coordinator
Instance
29
Q Depth
Monitor
Thread
Coordinator
Management
Thread
Coordinator
Instance
Powering Enterprise Process Advantage®
Auditor Sub System
PM4Data
Apache
Web Server
MQ
XML
Status
Query Request
Status Query
Server
Control Center
JSP Application
MQ
XML
Status
Query Response
DB2
MQ
XML
Status
Datagram
Process Flow
(Coordinator)
Server
2005 Metastorm, Inc.
Status Loader
MQ
XML
Status
Datagram
User
User
Components
Components
30
Powering Enterprise Process Advantage®
Metastorm
2005 Metastorm, Inc.
PM4Data Components
Powering Enterprise Process Advantage®
Coordinator Service
• Directs all file-transfer activity
– Receives transfer request in XML format from the PM4Data
interface
• Process Coordinator
• Command Line
• XML Interface
– Manages file transfers from end to end
• Sends transfer request to sending File Transfer Service (FTS)
• Uses XML to direct file transfer processing
• Receives status information back from the receiving FTS
• At least one must exist in the enterprise
– More than one can exist in the enterprise
– More than one can exist on a host
2005 Metastorm, Inc.
32
Powering Enterprise Process Advantage®
File Transfer Service (FTS)
• Can send and receive files, based on direction from the Coordinator
– A single instance of the FTS can both send and receive files
• Each host that performs transfers must have at least one FTS
– The FTS must have access to the source and destination file
system
• Multiple instances of the FTS can run on one host
– Typically, two instances of the FTS are placed on a host at
installation
– More instances of FTS can be added to provide additional
throughput
2005 Metastorm, Inc.
33
Powering Enterprise Process Advantage®
Exit Service
• Responsible for running PM4Data exits
• Can be accessed from any exit point in the transfer process
– Receives XML message directing execution
– Contains name of script and entry point in script
– Returns result element to PM4Data
• Uses XMScript modules
– User can easily add additional methods
• Exits run at any node
– Sender, Receiver, Process Coordinator
– Pre-process, Post-process
• Multiple exits at any exit point
– Exits can be chained
2005 Metastorm, Inc.
34
Powering Enterprise Process Advantage®
Auditor Service
– Status Query Server
• Provides interface between the GUI interface and the status
database
– Process Monitor – web-based
– Explorer – MMC-based
• Obtains status information for display
2005 Metastorm, Inc.
35
Powering Enterprise Process Advantage®
Auditor Service
– Status Logger
• Provides interface between each component that reports status and
the status database
– Receives status information from the following components:
•
•
•
•
Process Coordinator
Sending FTS
Receiving FTS
Exit Service
– Puts header and detail status information in the status database
2005 Metastorm, Inc.
36
Powering Enterprise Process Advantage®
User Service
• Provides interface between the Process Monitor web interface and the
user database
• Manages user accounts for the web interface
– Govern access to create and submit transfers and view status
• Typically only used in planning and administration
2005 Metastorm, Inc.
37
Powering Enterprise Process Advantage®
Registry Service
• Stores transfer-related artifacts created in the Process Monitor
– Transfer requests
– Destination entries
– Destination lists
– Directory Monitor entries
– Policies Object
– Scripts
– Process Flows
– Message Object
– Meta Data
2005 Metastorm, Inc.
38
Powering Enterprise Process Advantage®
Metastorm
2005 Metastorm, Inc.
Policy based I/O
Powering Enterprise Process Advantage®
Why use policy based I/O?
Issues
Solutions
Benefits
Integration requires custom
“glue” code at the end points.
Describe end points or data stores
with a policy.
Improve time to market by replacing
coding with configuration.
Files are “familiar” and nontransactional, while
databases, queues or other
data targets require
transactional properties for
integration.
Use file metaphor for all data targets
including databases, queues, TCP/IP
sockets, HTTP protocol.
Bridge file based “legacy”
applications with queue, database, or
other data stores applications.
File based integration
requires costly operational
support.
A managed file transfer with
flexibility for non-file based data
stores.
Simplifies development, without
burdening operations.
2005 Metastorm, Inc.
40
Powering Enterprise Process Advantage®
2005 Metastorm, Inc.
41