Transcript Design and Execution of Scientific Workflows using Web

Building a KEPLER Extension using
Ptolemy II
KEPLER Collaboration Staff
Presented by: Ilkay Altintas and Efrat Jeager
@ San Diego Supercomputer Center, UCSD
Outline


Overview of Ptolemy II and Kepler collaboration
Ptolemy II







Ptolemy II Architecture
Download, Make and Install
The Vergil user interface and application
Directors and Actors
Ports and Parameters
Manager
Kepler



Overview of pre-built Kepler actors and demos
Ingredients to build a Kepler extension
Building you first actor

2/20/2004
HelloWorld revisited… 
Building a KEPLER Extension Using Ptolemy II
2
The KEPLER System
for Scientific Workflows …
A framework for design, execution and
deployment of scientific workflows
 Caters specifically to the domain scientist
 Builds on Ptolemy II
(next slide... :-)
 Application pull from
various projects

http://kepler.ecoinformatics.org
2/20/2004
Building a KEPLER Extension Using Ptolemy II
3
… based on Ptolemy II


A set of Java packages for heterogeneous,
concurrent modeling, design and execution.
Strengths include:

Precisely defined models of computation and
component interaction




An intuitive GUI that lets rapid workflow composition
A modular, reusable and extendable object-oriented
environment
An XML based workflow definition – MoML


2/20/2004
e.g. Process Networks (PN) – data-flow oriented
Workflows defined in Ptolemy II MoML XML schema
Easily exchangeable
Building a KEPLER Extension Using Ptolemy II
4
Actor-Oriented Design
Reference: PtolemyII
Design Document-1, Chapter 1
2/20/2004
Building a KEPLER Extension Using Ptolemy II
5
Focus on Actor-Oriented Design

Object orientation:
What flows through
an object is
sequential control
class name
data
methods
call

return
Actor orientation:
What flows through
an object is
streams of data
actor name
data (state)
Input data
2/20/2004
parameters
ports
Output data
Building a KEPLER Extension Using Ptolemy II
Adapted from the *.ppt slides by
Edward A. Lee (See References)
6
Layered Software Architecture
Ptolemy II packages
have carefully
constructed
dependencies and
interfaces
Graph
Kernel
Actor
2/20/2004
Data
Building a KEPLER Extension Using Ptolemy II
Math
Adapted from the *.ppt slides by
Edward A. Lee (See References)
7
Ptolemy II Architecture

Core packages



Domain polymorphic actors
Domain packages

2/20/2004
MoML and visual interface support
Library packages


The data model(--abstract syntax) of the models
Abstract semantics
User Interface (UI) packages


(actor, data, kernel, math, util)
Implementations of different models of computation
Building a KEPLER Extension Using Ptolemy II
8
Overview of the Key Classes
2/20/2004
Building a KEPLER Extension Using Ptolemy II
9
Installing Ptolemy II
http://ptolemy.eecs.berkeley.edu/ptolemyII/
 User version


Version Ptolemy II 3.0.2 – Web Start Installer
Source download


Precompiled (…downloaded before…)
Source-only code – 4 steps to make




2/20/2004
cd “PTII”
rm –f config.*
./configure
make fast install
Building a KEPLER Extension Using Ptolemy II
10
Abstract Syntax of PTII Models
•Hierarchical Entities, Ports, Connections and Attributes
Abstract syntax choices:
connection
Actor
Relation
Link
Port
Link
•
Attributes
•
Port
ec
nn
•
co
n
tio
ec
nn
Link
co
tio
n
Attributes
Actor
Port
Actor
•
Attributes
Abstract syntax defines the structure of a
model, but says little about what it means.
2/20/2004
Building a KEPLER Extension Using Ptolemy II
•
Hierarchy is tree
structured (like XML).
A relation mediates
connections.
Ports can link multiple
relations and relations
can link multiple ports.
Ports mediate
connections across
levels of the hierarchy
(no statecharts-style
level-crossing links)
…
Adapted from the *.ppt slides by
Edward A. Lee (See References)
11
Models of Computation

Semantic interpretations of the abstract syntax
Different models <=> Different semantics <=> Different execution
One Class of Semantic Models: Producer / Consumer
process {
…
write();
…
}
process {
…
channel
read();
port
port
…
}
receiver
Are actors active? passive? reactive?
 Are communications timed? synchronized? buffered?

2/20/2004
Building a KEPLER Extension Using Ptolemy II
Models of Computation:
• continuous-time
• dataflow
• rendezvous
• discrete events
• synchronous
• time-driven
• publish/subscribe
•…
Adapted from the *.ppt slides by
Edward A. Lee (See References)
12
Director

Governs the execution of a composite entity.


Scheduling, dispatching threads, generate code, etc.
A composite entity is called opaque if it doesn’t
have a local director.

2/20/2004
An opaque composite entity inherits the director of its
container as its executive director.
Building a KEPLER Extension Using Ptolemy II
13
Vergil User Interface







The graph editor
Actor library
Director library
Utilities
Ports
Parameters
Quick tour
2/20/2004
Building a KEPLER Extension Using Ptolemy II
14
Using Vergil


Executing a pre-built model from the graph
editor
Building a model






Choosing actors
Making connections
Choosing a director
Building composite actors
Saving and running the model
Start Exercise-0
2/20/2004
Building a KEPLER Extension Using Ptolemy II
15
MoML


Modeling Markup Language
A primary persistent XML file format for
Ptolemy II.




2/20/2004
ptolemy filename.xml
ptexecute filename.xml
vergil filename.xml
moml configuration.xml filename.xml
Building a KEPLER Extension Using Ptolemy II
16
An Example MoML File
<?xml version="1.0" standalone="no"?>
<!DOCTYPE entity PUBLIC "-//UC Berkeley//DTD MoML
1//EN"
"http://ptolemy.eecs.berkeley.edu/xml/dtd/MoML_1.dtd">
<entity name="test"
class="ptolemy.actor.TypedCompositeActor">
<property name="director"
class="ptolemy.domains.sdf.kernel.SDFDirector"/>
<entity name="ramp" class="ptolemy.actor.lib.Ramp"/>
<entity name="plot"
class="ptolemy.actor.lib.gui.SequencePlotter"/>
<relation name="r"
class="ptolemy.actor.TypedIORelation"/>
<link port="ramp.output" relation="r"/>
<link port="plot.input" relation="r"/>
</entity>
2/20/2004
Building a KEPLER Extension Using Ptolemy II
Top-level entity
SDF domain
Reference: PtolemyII
Design Document-1, Chapter 6 17
Type System
2/20/2004
Building a KEPLER Extension Using Ptolemy II
18
Building Kepler Extensions


FOCUS: How to build actors?
Exercises are under:


http://www.sdsc.edu/~altintas/KeplerTutorial/KeplerHandsOn.txt
Please complete the preparation for the
programming exercises…
10 minute break…
2/20/2004
Building a KEPLER Extension Using Ptolemy II
19
Ingredients
We have:



Java 1.4.2
Cygwin for Developers (for Windows users)
Ant 1.5 or higher
Downloaded:

Compiled Ptolemy II source code
http://www.sdsc.edu/~altintas/KeplerTutorial/ptII3.0.2.tar.gz

Kepler extensions (a version from CVS)
http://www.sdsc.edu/~altintas/KeplerTutorial/kepler.zip
Completed the preparation for the hands-on
programming exercises.

Coding-style guidelines are at:
http://users.sdsc.edu/~altintas/codingStyle.pdf
Please use them! 
2/20/2004
Building a KEPLER Extension Using Ptolemy II
20
Adding Actors


Domain and/or data polymorphic actors
Use object-oriented inheritance to avoid code
duplication


To use the actors in Vergil




Add them to one of the actor libraries
Most libraries are under $PTII/ptolemy/actor/lib
Libraries are XML files
The basic structure of an actor:

2/20/2004
3 base classes: Source, Sink, Transformer
See
http://www.sdsc.edu/~altintas/KeplerTutorial/ActorStructure.txt
Building a KEPLER Extension Using Ptolemy II
21
Actor Interfaces:
Ports & Parameters
parameters:
a1 = value
a2 = value
Example:
input ports
output port
p1
p3
p2
input/output
port
2/20/2004
port
Building a KEPLER Extension Using Ptolemy II
22
Anatomy of an Actor: Ports





Used for “message transport”, can be an
input, an output, or both.
Key class: IOPort (Can be connected to other
IOPort instances via IORelations.)
Use TypedIOPort in order to benefit from the
type system! (Domain specific: DEIOPort)
Receiver and Sender interfaces depending on
the usage of the port.
Public members of the actors!!!
2/20/2004
Building a KEPLER Extension Using Ptolemy II
23
Ports: Introduction to the API
public TypedIOPort portName; //Definition
//Create the port
portName = new TypedIOPort(this, “portName”, true, false);
portName.setMultiport(true); //Can support more than one link
int width = portName.getWidth(); //0 or 1 if single port
//Reading and Writing
portName.send(channelNumber, token);
Token token = portName.get(channelnumber);
//Setting the type of the port
portName.setTypeEquals(BaseType…a type in the type system…);
portName.setTypeAtLeast(…must be a port or parameter…);
2/20/2004
Building a KEPLER Extension Using Ptolemy II
24
Anatomy of an Actor: Parameters


Public members of the actors!
Similar API with ports…
public Parameter parameterName; //Definition
//Creation and setting the initial value: 2-ways
parameterName = new Parameter (this, “parameterName”,
new Token(…token value…));
OR
parameterName = new Parameter (this, “parameterName”);
parameterName.setExpression(…tokenValue…);
2/20/2004
Building a KEPLER Extension Using Ptolemy II
25
Anatomy of an Actor: Constructors

The major task:


super(container, name);


To create and configure ports and parameters
Carries the NameDuplication and IllegalAction
exceptions from the super class.
The icon for the actor can be set here.
2/20/2004
Building a KEPLER Extension Using Ptolemy II
26
Exercise-1

Create an actor with





An output port
A parameter
Introduction to the fire method.
Adding your actors into the actor library
Using your actors in a workflow
Break for 10 minutes when finished…
2/20/2004
Building a KEPLER Extension Using Ptolemy II
27
Execution of an actor
initialize()
prefire
Iterations
wrapUp()
2/20/2004
fire()
postfire()
Building a KEPLER Extension Using Ptolemy II
28
Action Methods

initialize(): Initialize the state variables of
an actor.

prefire(): Returns a boolean which
indicates if the actor wants to fire.


Can also be used to perform an operation that
will happen exactly once per iteration.
fire(): The main point of execution.

2/20/2004
For reading inputs, producing outputs, read the
current parameter values.
Building a KEPLER Extension Using Ptolemy II
29
Action Methods (cont.)

postfire(): Has two tasks:



2/20/2004
Updating persistent state
Determining whether the execution of an
actor is complete.
wrapUp(): For displaying final results.
Building a KEPLER Extension Using Ptolemy II
30
Things to remember when
implementing a fire() method




Use the methods of the Token class for
arithmetic whenever possible (to get data
polymorphism)
If data-polymorphism is not necessary, set the
type to a base type then cast the token to that
type.
Cannot assume that there is be data available at
all the input ports (for domain-polymorphism)
Do not update the persistent state in fire() (use
postfire())
2/20/2004
Building a KEPLER Extension Using Ptolemy II
31
Implementing Polymorphism





Class: PortParameterFunction
A PortParameterFunction object will be returned as a
function of two objects.
Set the type of the output equal to the type of this object.
Type system will compute the type of the
PortParameterFunction object and use it as the type of the
output when necessary.
We’ll see this in the example 2! 
2/20/2004
Building a KEPLER Extension Using Ptolemy II
32
Exercise-2




Getting data through an input port.
Understanding the action methods.
Designing a polymorphic actor.
Using inner classes
Break for 10 minutes when finished…
2/20/2004
Building a KEPLER Extension Using Ptolemy II
33
The manager




Controls the overall execution of a model.
Interacts only with the “top-level composite actor”
startRun() -> run() -> execute()
ExecutionListener interface provides the manager
with info on the events generated during execution.
Hand-outs... show how Ptolemy II handles mixing
models of computations hierarchically.
--Using opaque composite actors!
2/20/2004
Building a KEPLER Extension Using Ptolemy II
34
Exceptions



A uniform mechanism for reporting errors
Base class: KernelException
Exception chaining re-implemented since Java
versions < 1.4 doesn’t support it.



The detail message includes the detail message from the
cause argument.
A protected _setCause() method is implemented, but not
the public initCause() method that JDK1.4 has.
Non-severe exceptions: IllegalActionException,
NameDuplicationException, NameDuplicationException.

Severe-exceptions: KernelRuntimeException,
InvalidStateException, InternalErrorException.
2/20/2004
Building a KEPLER Extension Using Ptolemy II
35
Exercises-3&4






Stopping a model using an actor
Utilizing the manager functions
Handling exceptions
Using the postfire method
Creating a customized icon for an actor
Using an existing code/application in
Kepler.
Break for 10 minutes when finished…
2/20/2004
Building a KEPLER Extension Using Ptolemy II
36
Existing Kepler Actors








Actor prototyping tool
Generic Web and Grid Service
RDBMS Connection and Querying
Generic User Interface and Transformation
Biological Service and Data Access
Rock Classification
EML Data Ingestion
GARP Native Species Pipeline (Using JNI to
utilize C++ code)
2/20/2004
Building a KEPLER Extension Using Ptolemy II
37
Actor Prototyping Tool



Allows “place-holder” actors to be defined on the fly by non-programmers
during workflow creation
Scientists can thereby create workflows without programming knowledge
Workflows created with these actors can be executed once their
functionality is implemented by a programmer
2/20/2004
Building a KEPLER Extension Using Ptolemy II
38
Distributed SWFs in KEPLER

Web and Grid Service plug-ins



WS Harvester



2/20/2004
WSDL, GWSDL
ProxyInit, GlobusGridJob, GridFTP, DataAccessWizard
Imports all the operations of a specific WS (or of all
the WSs in a UDDI repository) as Kepler actors
WS-deployment interface (…ongoing work…)
XSLT and XQuery transformers to link non-fitting
services together
Building a KEPLER Extension Using Ptolemy II
39
A Generic Web Service Actor
Given a WSDL and
the name of an
operation of a web
service, dynamically
customizes itself to
implement and
execute that method.
Configure - select service
operation

2/20/2004
Building a KEPLER Extension Using Ptolemy II
40
Set Parameters and Commit
Set parameters
and commit
2/20/2004
Building a KEPLER Extension Using Ptolemy II
41
WS Actor after Instantiation
2/20/2004
Building a KEPLER Extension Using Ptolemy II
42
Web Service Harvester
• Imports the web services in a repository
into the actor library.
• Has the capability to search for web
services based on a keyword.
2/20/2004
Building a KEPLER Extension Using Ptolemy II
43
Composing 3rd-Party WSs
Output of previous
web service
User interaction &
Transformations
2/20/2004
Building a KEPLER Extension Using Ptolemy II
Input of next
web service
44
Current Grid Actors
2/20/2004
Building a KEPLER Extension Using Ptolemy II
45
Providing RDBMS Access

Database connection.


Database Querying.


Opening a database connection and passing it to
all actors accessing this database.
A generic actor that queries a database and
provides its result.
DBConnection type and DBConnectionToken.

2/20/2004
A new IOPort type and a token to distinguish a
database connection from any general type.
Building a KEPLER Extension Using Ptolemy II
46
Database Connection

OpenDBConnection actor:


2/20/2004
Input: database connection information
Output: A DBConnectionToken, a reference to a
database connection instance, through a DBConnection
output port.
Building a KEPLER Extension Using Ptolemy II
47
Querying a Database

DatabaseQuery actor:



2/20/2004
Input: A query string (SQL) and a database connection
reference.
Parameters: output type – XML, Record or String.
Process: Execute query. Produce results according to
parameters.
Building a KEPLER Extension Using Ptolemy II
48
Using Bio-Services Actors: PIW
2/20/2004
Building a KEPLER Extension Using Ptolemy II
49
Sub-workflow:
Gene Sequence Processing
Data transformations
between the tasks
2/20/2004
Building a KEPLER Extension Using Ptolemy II
50
Sub-workflow:
2/20/2004
Transfac Detail
Building a KEPLER Extension Using Ptolemy II
51
Classifying Igneous Rocks
2/20/2004
Building a KEPLER Extension Using Ptolemy II
52
Actors for Modeling The Classifier
Finer granularity
Extracted from the mineral
composition and this level’s
diagram coordinates.
Diagrams information and
transitions between them.
Classifier: Locates
the point’s region.
SVG to polygons.
Displays the point in the
diagram for this level.
2/20/2004
Building a KEPLER Extension Using Ptolemy II
53
EML Data Ingestion Actor

Ingests any data format described by EML metadata

Converts raw data to Ptolemy format

Data can then be operated on with other actors
2/20/2004
Building a KEPLER Extension Using Ptolemy II
54
Using EML ingestion actor
2/20/2004
Building a KEPLER Extension Using Ptolemy II
55
Third-Party Local Applications
2/20/2004
Building a KEPLER Extension Using Ptolemy II
56
What did we learn?



Basic concepts of Ptolemy II
Creating actors within the Ptolemy II framework
The existing Kepler extensions
What we didn’t learn:

Extending the type system


Extending the user interface


Need to define a class for the new type and add it to the type lattice
Effigy, Tableaux (Similar to Model-View-Controller architecture)
Creating our own director

Developing a director is harder!

2/20/2004
Need to inherit the Director class, and implement the semantics of how
the actors will behave. (Semantics==Model of computation)
Building a KEPLER Extension Using Ptolemy II
57
Ptolemy II Design Document
Volume 1:
User-Oriented
2/20/2004
Volume 2:
Developer-Oriented
Building a KEPLER Extension Using Ptolemy II
Volume 3:
Researcher-Oriented
Adapted from the *.ppt slides by
Edward A. Lee (See References)
58
Becoming a Kepler member?

http://kepler.ecoinformatics.org/members.html
Email me:
[email protected]
to get a read-only account.
Will upgrade to become a
contributing member
depending in time.
2/20/2004
Building a KEPLER Extension Using Ptolemy II
59
References







Ptolemy II Design Documentation and Source Code
http://ptolemy.eecs.berkeley.edu/papers/03/ptIIDesignIntro/
Kepler website: http://kepler.ecoinformatics.org
Overview of Ptolemy II (Powerpoint presentation), Edward A. Lee,
October 9, 2003, presented in course EE249 - Design of Embedded
Systems: Models, Validation and Synthesis.
An Overview of the Ptolemy Project a nd Actor-Oriented Design
(powerpoint presentation), Edward A. Lee, February 4, 2004, OMG
Technical Meeting, Anaheim, CA.
Presentations from the Ptolemy website:
http://ptolemy.eecs.berkeley.edu/ptolemyII/
Kepler documentation, javadocs and this tutorial will be made available
at the website soon.
2/20/2004
Building a KEPLER Extension Using Ptolemy II
60