Transcript Implementations - Databases and Distributed Systems

Enabling Online Analysis
of Notification Services
Sebastian Salvucci
Advisor: Mariano Cilia
Co-Advisor: Ricardo Orosco
Facultad de Ciencias Exactas
Universidad Nacional del Centro
de la Provincia de Buenos Aires
Tandil, August 2006
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
intro/motivation
proposed approach
architecture
implementation
conclusions/outlook
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Pub/Sub Systems
intro
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
NS characteristics
intro
Adressing Models
• channel-based
• subject/topic-based
• content-based
Behavior
(performance influenced by…)
• Matching: is the process carried out by event brokers
to check messages against subscriptions.
• Routing: is in charge of forwarding delivering the messages
to the subscribers. Different algorithms (flooding, simple
routing, identity routing, covering, merging)
• Other features: persistence, priorities, transactions, selectors,
caching, expiration, filters, message types, …
Implementations
(different features)
• Research Projects: REBECA, Siena, Hermes, Jedi, Gryphon, Srcibe
• Commercial Products: Tibco Rendezvous, WebSphereMQ,
SonicMQ, Fiorano
• JMS Standard API specification: Open Source Implementations
(OpenJMS, JBossMQ, ActiveMQ, Joram)
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Centralized/Distributed
Centralized
intro
Distributed event broker network
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Traditional Analysis Aproaches
intro
Tracing, log files: add extra source code finding first the right places
and then modifying the code to write the relevant data into a text file.
• enormous ad-hoc text files
• source code abstraction level
Black-box analysis:
data are observed and registered
at the clients, when messages are published and/or received
• thick granularity does not allow
to determine behavior details.
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Motivation/Goal
intro
Motivation
• Today is difficult to analyze notification services
focalizing individuals aspects of the behavior
• Few partials and proprietary solutions
• Should help to improve NS behavior and performance
(research/development/tuning)
Goal
• Create an NS-independent analysis framework
based on observations, metrics definitions and visual
representations of them.
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
intro
proposed approach
architecture
implementation
conclusions/outlook
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Proposed Approach
proposed approach
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Observations
proposed approach
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Observations (2)
proposed approach
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Following messages
proposed approach
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Metrics from observations
proposed approach
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Metrics Analysis
proposed approach
Online analysis
metric values
external visualization
techniques can be
configured to represent
metrics in different ways
Post-mortem analysis
Offline analysis: replay of executions
Static analysis: traditional queries, data mining, etc.
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
intro/motivation
proposed approach
architecture
implementation
conclusions/outlook
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
architecture
Proposed Architecture
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
architecture
Considerations
Visualization
• Reusable and configurable visualization techniques
• Metrics independence
• Multi users (client)
Metrics Composition
•
•
•
•
•
Communication
• Push-based communication
• Many-to-many
• Firewalls, security restrictions
Management
Data in movement
Generic data transformation
High-level metrics definition
Post-mortem analysis
Global Time
• Configuration
• Components orchestration
Behavior Observation
• Notification service independence
• Influence over the NS under observation
• Sending and generation mechanisms performance
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
architecture
Behavior Observation Layer
• Adopting AOP to avoid
intrusive modification of
source code.
• AOP lets separate
and modularize
observation logic.
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
architecture
Behavior Observation Layer (2)
• Enable/disable
observations generation
according metrics
composition needs
• Sending frequency:
(online, offline, time
intervals, fixed amount,
package size)
Implementation Alternatives
-static weaving
-dynamic weaving
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
architecture
Metrics Composition Layer
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
architecture
Compositors
Generic transformation over data streams (configurable)
metric values
time
interval
observations
Implementation alternative:
Stream Process Engines
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Metrics Definition: Data Warehouse Metaphor
•
•
•
•
•
High-level and intuitive modelling (from static analysis)
Generic and abstract data transformation
Different measures from the same observations
Decoupling metric composition output from visualization
Metadata for stored data
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
architecture
architecture
Metrics Composition
Configuring compositors from metrics definition
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
architecture
From Data to Visualization
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
architecture
Visualization
Visualization panel
(web browser)
Visualization components
Implementation Alternatives
•
•
•
•
•
Applets
DHTML/AJAX
SVG
Flash
VRML/X3D
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
architecture
Communication
centralized approach
pub/sub approach
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
architecture
Components Orchestration
Test Run Definition
• load generation scenario
• metrics
• visualization techniques
Test Run Recording
1.
2.
3.
4.
5.
Enable observations
Deploy compositors (subs)
Init visualization components (subs)
Subscribe repository (all)
Start Load generation
Test Run Reproduction
1.
2.
Init visualization panel
Stored data publication
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
intro
proposed approach
architecture
implementation
conclusions/outlook
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Initial decisisons
Communication
HTTP-based
publish/subscribe
framework
HTTP streaming
Instead of closing the HTTP connection
after fetching an HTML page, the
connection remains open while fresh data
is pushed to the client
Pushlets Characteristics
• subject based
• runs in a servlet container
• very thin Java client
• JavaScript client for browsers
• modes stream/polling
implementation
Data representation
• XML data representation for
observations and metric values
Repository
• Adopting eXist XML native database
(XQuery, XUpdate support)
XML repository structure
Pushlets Limitations
• scalability with mode stream
• message type
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Reusing Aspects
implementation
method signature & parameters
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Compositors: Aggregator filter + primitives
implementation
Enhancing Filtering in Notication Services
Primitives based filters
Aggregator Filter
Implementations
•
•
•
•
Pushlet adapters
XPath primitive
Compositors creation (from metrics)
Compositors deploy/undeploy (pub/sub)
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Composing metrics: data transformation
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
implementation
Visualization Panel
implementation
•Metrics and components
from test run definition
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Visualization Components
implementation
• Visualization components description.
• Mapping between metrics and components
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Components Orchestration
Test Run Recording
implementation
Test Run Reproduction
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
intro
proposed approach
architecture
implementation
conclusions/outlook
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Conclusions
conclusions
The utilization of three different kinds of components to observe the behavior
(aspects), compose streams of observations (metrics) and graphically represent them
(visualization techniques) as well as their loosely-coupled communication through
a pub/sub service offer a high flexible framework to enable online
analysis of notification services.
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Conclusions (2)
conclusions
The use of a repository to store the data produced by
aspects and metrics composition supports postmortem (offline and static) analysis using the
same architecture that was developed for online
analysis.
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Conclusions (3)
conclusions
• Not intrusive modification of the source code
(by using AOP to generate observations)
• Independence of notification service implementations
(by associating semantics of observations to behavior
description instead source code)
• Generic and intuitive modeling of computations over data streams
• Metrics decoupled from visualization
• Metadata associated to stored executions
(by data cube description)
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Outlook
conclusions
• Declarative description of observations and automatically generate
the aspects involved letting choose different AOP alternatives.
• By seeking and improving the data warehouse metaphor could be possible
to associate a complete and unique multidimensional
analysis to data streams and historical data.
• Graphical wizards and interfaces upon XML descriptions towards
a higher abstraction level for users, offering a complete
analysis-oriented application (bookmarks, notes, etc..)
• Other domains (alternative data sources): software analysis,
business activity monitoring, applications that rely on pub/sub systems, etc..
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006
Muchas Gracias!
questions / comments / feedback
Enabling Online Analysis of Notification Services
Sebastian Salvucci - Tandil, Aug2006