Transcript Simon Cox - National e

Geodise & GeodiseLAB
Simon Cox
University of Southampton
21st April 2005
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
Grid Enabled Optimisation and
Design Search for Engineering
(GEODISE)
Simon Cox- Grid/
W3C Technologies
SYSTEMS- Engineering
Southampton,
OxfordBAE
and Manchester
and High Performance Computing
Global Grid Forum Apps Working Group
Andy Keane- Director of Rolls
Royce/ BAE Systems University
Technology Partnership in Design Search
and Optimisation
Mike Giles- Director of Rolls Royce
University Technology Centre for
Computational Fluid Dynamics
Carole Goble- Ontologies and
DARPA Agent Markup Language (DAML)
/ Ontology Inference Language (OIL)
Nigel Shadbolt- Director of
Advanced Knowledge Technologies
(AKT) IRC
Rolls-Royce- Engineering
Fluent- Computational Fluid Dynamics
Microsoft- Software/ Web Services
Intel- Hardware
Compusys- Systems Integration
Epistemics- Knowledge Technologies
Condor- Grid Middleware
The GEODISE Team ...
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Richard Boardman
Sergio Campobasso
Liming Chen
Mike Chrystall
Trevor Cooper-Chadwick
Simon Cox
Mihai Duta
Clive Emberey
Hakki Eres
Matt Fairman
Mike Giles
Carole Goble
Ian Hartney
Tracey Hunt
Zhuoan Jiao
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Andy Keane
Marc Molinari
Graeme Pound
Colin Puleston
Nicola Reader
Angus Roberts
Mark Scott
Nigel Shadbolt
Wenbin Song
Paul Smart
Barry Tao
Lisa Telfer-Brunton
Jasmin Wason
Fenglian Xu
Gang “Luke” Xue
© Geodise Project, University of Southampton, 2001-2005
http://www.geodise.org/
Distributed Systems 2005
IP
Moore’s Law
ASCII
HTTP
(HTML) XML
(HTML)
Web Services
Network
Compute/ Data
Future Services
Flexible,
Seamless,
Secure Sharing
© Geodise
Project,
Universityand
of Southampton,
2001-2005
http://www.geodise.org/
Software
Applications - Geodise
CFD 3D engine nacelle optimisation
0
1
2
Conventional Inlet
Negative Scarf Inlet
 Goal - reduce ground noise
generated by fan when plane
takes off.
 Optimise aerodynamic
performance when scarf angle
is varied.
© Geodise Project, University of Southampton, 2001-2005
http://www.geodise.org/
Workflow for aerodynamic shape optimisation
using CAD, Gambit, and Fluent
Problem definition
Design of Experiment
ProEngineer CAD
(Condor Pool)
Gambit Meshing
(Globus Compute)
Response surface modelling
Optimisation on Response surface
Validation
Fluent CFD
(Globus Compute)
Applications – GEM
Electromagnetic optimisation
Transmission of light properties through a
photonic crystal
Parameters are radius of holes and light
wavelength
• Large number of
designs, parameters
and solutions
• Query for a particular
data range to postprocess
© Geodise Project, University of Southampton, 2001- 2004.
http://www.geodise.org/
GEODISE
Engineer
GEODISE
PORTAL
Knowledge
repository
Ontology for
Engineering,
Computation, &
Optimisation and
Design Search
Visualization
Session
database
Traceability
OPTIMISATION
OPTIONS
System
APPLICATION
SERVICE
PROVIDER
Intelligent
Application
Manager
Reliability
Security
QoS
CAD System
CADDS
IDEAS
ProE
CATIA, ICAD
Globus, Condor, OGSA
Optimisation
archive
COMPUTATION
Licenses
and code
Analysis
CFD
FEM
CEM
Parallel machines
Clusters
Internet Resource Providers
Pay-per-use
Intelligent
Resource
Provider
Geodise will provide grid-based seamless access to an intelligent knowledge
Design
repository, a state-of-the-art collection of optimisation and search tools,
archive industrial strength analysis codes, and distributed computing & data resources
Collaboration
Coordination
Grid-Enabled Scripting Environment
Applications
Enterprise
• Why use scripting languages?
Grid
– Flexibility
– High-level functionality
– Quick application development
– Extend the user’s existing PSE
– … Acts as execution/ enactment engine too
Management
• Approach
– Matlab initially chosen as the hosting environment
– Now working with Python/ Jython
© Geodise Project, University of Southampton, 2001-2005
http://www.geodise.org/
Hardware
Engineering Applications of the
Grid via Geodise
Problem Solving Environment
Matlab, Python, etc
Bringing underlying infrastructure into
the Problem Solving Environment
Geodise Toolkits
“Useful into useable”
“Dial-tone”
Enterprise
“Heartbeat”
Grid
Hardware
• Grid Compute toolbox



Cycle stealing
High-performance computing
Integration with heterogeneous environments
(Windows, Linux, Solaris, …)
Problem Solving
Environments e.g.
Matlab, iSight
Python,
ModelCenter,…
Optimisation
`Tools
Geodise deliverables
• Grid Data toolbox



Raw files into journaling filesystem
Metadata in relational database (Oracle, DB2, SQL
Server, mySQL…)
XML processing capability
• Intelligent Workflow composition, execution
and monitoring tool




Rule and case-based composition of ‘best’
workflow (semantics and knowledge driven)
Deployment to compute/ data resources
System-level “heartbeat”
User/ Application level “dial-tone” (security,
resource availability, network weather, …)
Grid-enabled Toolboxes for
Engineers
Middleware (Grid)
Hardware
Designed around web-services to inherit and leverage
Technologies (e.g. security, reliability) and Development tools
Collaboration
Geodise Tools
Coordination
Applications
Enterprise
• A number of Grid-enabled design tools have
been developed
Grid
Management
– Compute Toolbox: certificate based authentication, job
Hardware
submission, and file transfer, etc.
– Database Toolbox: archive files and data, custom metadata,
locate and retrieve data, etc.
– XML Toolbox: convert Matlab structures to and from XML
– Workflow Editor: graphical problem solving environment which
allows users to construct and configure workflows for the Grid
– OptionsMatlab: Matlab interface to the Options design
exploration system
– Knowledge Toolbox: Intelligent Design advice
© Geodise Project, University of Southampton, 2001-2005
http://www.geodise.org/
Engineer
Graphical
Workflow Tool
Geodise Architecture
Results
Knowledge Toolbox
Compute Toolbox
API
Results
CAD
Globus Servers
OptionsMatlab
Matlab Script
Pro/Engineer
Globus Server #1
Database Toolbox
Compute Toolbox
ICAD
Grid Job
Submission
Condor Cluster
Geodise Database
Globus Server /
Condor Node
Globus and Condor
Job Managers
XML Toolbox
Results by GridFTP
Catia
Globus Server #N
Metadata by webservice
Database Toolbox
API
Knowledge Toolbox
API
Applications
Computation
Matlab Scripting Environment
Results
IDEAS
Analysis
Gambit/Fluent
Abaqus
Metadata
Webservice
Condor Master
Catia
HYDRA
Globus Server
Knowledge
Repository
Condor Node #1
Physical files,
scripts, and
outputs
Metadata
and
Structure
Repository
Condor Node #N
Java
Matlab
Local
Filesystem
Sponsors
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
GeodiseLab Activities for the OMII
• Harden & deliver Geodise toolboxes
– Geodise Compute Toolbox
– Geodise Database Toolbox
– XML Toolbox for Matlab
•
•
•
•
•
Documentation
Testing
Extend existing functionality
Train and support users
... GGF: Simple API for Grid Applications (SAGA)
Working Group
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
Compute Toolbox: Aims
• Support access to whichever compute resources
users wish to exploit
– Globus GT2
– OMII services
– Condor
• Provide access from alternative environments
– Matlab
– Jython
– cPython
• Leverage existing APIs wherever possible
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
Compute Toolbox: Hit-list
Matlab
Jython
cPython
.m files
.py files
.py files
?
Geodise Java API
.NET Client
Windows
HPC/Grid
Condor
Native
Java Client
Java CoG
OMII API
Condor
Web Service
Globus
GT2
OMII
Services
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
Available
Now
GeodiseLab Compute Toolbox
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
Compute Toolbox Release
Matlab
Jython
cPython
.m files
.py files
.py files
?
Geodise Java API
Condor
Native
Java Client
Java CoG
OMII API
Condor
Web Service
Globus
GT2
OMII
Services
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
Toolbox Architecture
Matlab
Environment
script.m
gd_jobsubmit.m
gd_getfile.m
Java CoG
Globus GT2
Resource
GRAM
job submission
GridFTP
file transfer
GRAM
Service
GridFTP
Service
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
Certificate Management Functions
gd_certinfo
Returns information about the user's certificate
gd_createproxy
Creates a Globus proxy certificate
gd_proxyinfo
Returns information about the user's proxy certificate
gd_proxyquery
Queries whether a valid proxy certificate exists
gd_destroyproxy
Destroys the local copy of the user's Globus proxy
certificate
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
Certificate Management
C:\GEODISE
>> gd_createproxy
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
Certificate Management
C:\GEODISE
>> subject = gd_certinfo
subject
: C=UK,O=eScience,OU=Southampton,L=SeSC,CN=graeme pound
issuer
: C=UK,O=eScience,OU=Authority,CN=CA,[email protected]
start date : Wed Sep 22 16:31:24 BST 2004
end date
: Thu Sep 22 16:31:24 BST 2005
subject =
/C=UK/O=eScience/OU=Southampton/L=SeSC/CN=graeme pound
>> isvalid
subject :
issuer
:
type
:
strength :
timeleft :
= gd_proxyinfo
C=UK,O=eScience,OU=Southampton,L=SeSC,CN=graeme pound,CN=proxy
C=UK,O=eScience,OU=Southampton,L=SeSC,CN=graeme pound
full legacy globus proxy
512 bits
11 h, 59 min, 30 sec
isvalid =
1
>>
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
Job Submission Functions
gd_jobstatus
Gets the status of a Globus GRAM job
gd_jobsubmit
Submits a compute job to a Globus GRAM job manager
gd_jobpoll
Queries the status of a Globus GRAM job until complete
gd_jobkill
Kills a Globus GRAM specified by a job handle
gd_chmod
Changes file permissions of a file on a Globus resource
gd_condorsubmit
Submits a job to a Condor pool via a Globus resource
gd_submitunique
Submits a GRAM job to a unique working directory
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
Job Submission
C:\GEODISE
>> RSL = '&(executable="/bin/date")(stdout="date.out")';
>> host = 'grid-compute.oesc.ox.ac.uk';
>> jobhandle = gd_jobsubmit(RSL, host)
jobhandle =
https://grid-compute.oesc.ox.ac.uk:30001/30705/1098694366/
>> isdone = gd_jobpoll(jobhandle)
isdone =
1
>>
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
GridFTP File Transfer
gd_getfile
Retrieves a remote file using GridFTP
gd_putfile
Puts a remote file using GridFTP
gd_transferfile
Performs a third-party file transfer using GridFTP
gd_makedir
Creates a remote directory using GridFTP
gd_listdir
Lists the contents of a directory on a GridFTP resource
gd_fileexists
Tests the existence of files on a Globus resource
gd_rmdir
Deletes a remote directory using GridFTP
gd_rmfile
Deletes a remote file using GridFTP
gd_rmuniquedir
Deletes a remote directory and its contents
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
File Transfer
C:\GEODISE
>> exists = gd_fileexists(host, 'date.out')
exists =
1
>> gd_getfile(host,'date.out','localfile.txt');
>> type('localfile.txt')
Mon Oct 25 09:52:46 BST 2004
>> gd_rmfile(host, 'date.out');
>> exists = gd_fileexists(host, 'date.out')
exists =
0
>>
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
Resource Testing
gd_servermetrics
Performs tests upon a list of Globus resources
gd_testauthentication Tests authentication with a Globus resource
gd_testfiletransfer
Tests file transfer to a Globus resource
gd_testjobsubmission
Tests the job submission to a Globus resource
gd_timeauthentication Times authentication to a Globus resource
gd_timefiletransfer
Times file transfer to a Globus resource
gd_timejobsubmission
Times a job submission to a Globus resource
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
Resource Testing
C:\GEODISE
>>
>>
>>
>>
servers(1).name = 'grid-compute.leeds.ac.uk';
servers(2).name = 'escience-dept2.sesnet.soton.ac.uk';
results = gd_servermetrics(servers);
disp(results(1))
name: 'grid-compute.leeds.ac.uk'
authentication: 188
jobsubmission: 672
filetransfer: 984
>> disp(results(2))
name:
authentication:
jobsubmission:
filetransfer:
'escience-dept2.sesnet.soton.ac.uk'
125
485
4938
>>
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
Available
Now
GeodiseLab XML Toolbox
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
Web Technology & Services
ID
Name
Value
0001
A
1
0002
B
2
0003
C
3
Databases
Engineer’s familiar scripting
environment
Access
XML Toolbox
Communication
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
Features
Toolbox
<struct xmlns="http://www.geodise.org/matlab.xsd" idx="0" fields="name id param">
<char idx="1" name="name" size="1 14">eng_opt_design</char>
<char idx="1" name="id" size="1 11">1022223-779</char>
<struct idx="1" name="param" size="1 1" fields="width height material">
<double idx="1" name="width" size="1 1"> 100 </double>
<double idx="1" name="height" size="1 1"> 87.9 </double>
<double idx="1" name="material" size="1 1"> 0.07244 </double>
</struct>
</struct>
Filestore
Toolbox
DB
Standard File Formats
(eg. STEP-XML, …)
<root>
<name>eng_opt_design</name>
<id>1022223-779</id>
<param>
<width> 100 </width>
<height> 87.9 </height>
<material> 0.07244 </material>
</param>
</root>
• Powerful Matlab to XML interface
• Allows transparent conversion between proprietary Matlab
data types and standard XML
• Brings flexibility of open XML Standard to Matlab
• Easy-to-use interface (4 simple core functions:
xml_save, xml_load, xml_format, xml_parse)
• Extensive documentation & direct help functionality
• Large range of potential applications:
– Science, Engineering, IT, Web, …
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
XML Toolbox in action
• Jan 2004 v2.0 released
• Nov 2004
– Downloads from soton.ac.uk/~gridem: 227
– Downloads from mathworks.com/matlabcentral: 882
• Apr 2005 v3.0
– > 1500 downloads total
• Large user base
– Academic & Commercial
– Extensive testing & feedback community
– Market pull
• So far very positive feedback from users
• Matlab Central user rating: 5 / 5
• Linked by Mathworks help for XML in Matlab
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
Aalesund College: Read data from database
Aberdeen University: Export analysis results from Matlab to XML
ADVA: to read existing XML-files in a Matlab structure and do some calculations
Advanced Tech Center: configuring xml input into matlab
Alfred-Wegener-Institut: Handle data from echo sounder
AMAT: Evaluation of XML input in matlab
Anna University: We are programmed to develop a new tool for analysing discrete event system using matlab.
Applied Math Labolatory, Ecole Centrale Paris
Aston University: To read date from xml files in matlab
Avantium Technologies BV: Transferring HTS data stored as XML into Matlab
BAE Systems: Data import/export between MATLAB and Delphi; Generation of XML files to control a BAE SYSTEMS Software Defined Radio
Beb Corp
Bosch GmbH
Calypso Medical: looking at xml files in matlab
Carnegie Mellon university: Use XML in our facial expression analysis project
Center for Advanced Spatial Technologies: Writing least squares adjustments for geodetic observations as part of research in new techniques
for parameter estimation combining photogrammetric and terrestrial observations. Data is varied but structured and I am interested in using XML
Chouaib Doukkali Universit: Numericals simulations of the physics phenomenons
Christian-Albrechts-Universitat zu Kiel (CAU): http://busselab.uni-kiel.de
City College of CUNY: Projects using Matlab and XML technology.
CMS: For trial
Cold Spring Harbor Laboratory: Storing and parsing neuronal data
Conestoga college: intro to telecomme
Crystal Engineering
CU: Sending sending commands to matlab via xml, outputting results of calculations from matlab in xml to be used by other products
Daimler Chrysler: test
DeAM: xml reading
DECO
DEIMOS Engenharia: Reading/writing configuration files
DGA Defence, France: For managing data files used in simulation's input
DSTO: input parameters to radar model
EADS CRC: Information
Emyo: For teaching
ENI: Reading & Modifying XML documents
EPFL: Evaluation for data exchange between matlab and other software
Fairchild Dornier: Currently I make a research, how I can efficiently store my matlab data structure in a text editable data format, that's
why I'am looking for a xmlread and a xmlwrite function.
Financial Engines
FIOT
© Geodise Project, University of Southampton, 2001-2005.
Fraunhofer Germany: The last aim of our research is to provide user a platform for maintenance oriented decision via web service.
38 sample institutions
http://www.geodise.org/
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
Available
Now
GeodiseLab Database Toolbox
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
Database Toolbox – Aims
• Make data easier to manage, find and share
• Store data with descriptive information
– Standard and user defined metadata
– Query over metadata to easily locate data
• Familiar interface for engineers
– Wrap toolbox as Matlab/Jython functions
– Easy integration into user scripts
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
Database Toolbox – Features
• Archive files and data structures with
metadata
• Group related data
• Simple query interface
• Retrieve data based on ID rather than
location
• Security
– User grants permission for others to access their data
– Certificate-based authentication and authorisation
– Secure Web service access to central database
© Geodise Project, University of Southampton, 2001-2005.
http://www.geodise.org/
Functions Overview
gd_archive
Stores a file or structure with metadata into an archive
gd_query
Queries over metadata or structures in the archive
gd_retrieve
Retrieves a file or structure from the archive to the local
machine
gd_addusers
Grants users permission to access some data
gd_datagroup
Creates a new datagroup to aggregate files and
structures
gd_datagroupadd
Adds a file or structure to a datagroup
gd_display
Displays the results of a query
© Geodise Project, University of Southampton, 2005.
http://www.geodise.org/
GeodiseLab Database Toolbox
Worked Example
© Geodise Project, University of Southampton, 2005.
http://www.geodise.org/
Client
Generate Data
Grid
Location
Service
Metadata
Service
Authorisation
Service
Query
Service
XML Toolbox
Java CoG
Globus Server
Database
Metadata
Client
Grid
Location
Service
Metadata
Service
Authorisation
Service
Query
Service
XML Toolbox
Java CoG
Globus Server
Database
Archive
Client
Grid
Location
Service
Metadata
Service
Authorisation
Service
Query
Service
XML Toolbox
Java CoG
Globus Server
Database
Client
Generate More Data
Grid
Location
Service
Metadata
Service
Authorisation
Service
Query
Service
XML Toolbox
Java CoG
Globus Server
Database
Group Files
Client
Grid
Location
Service
Metadata
Service
Authorisation
Service
Query
Service
XML Toolbox
Java CoG
Globus Server
Database
Query
Client
Grid
Location
Service
Metadata
Service
Authorisation
Service
Query
Service
XML Toolbox
Java CoG
Globus Server
Database
Retrieve
Client
Grid
Location
Service
Metadata
Service
Authorisation
Service
Query
Service
XML Toolbox
Java CoG
Globus Server
Database
Applications - GENIE
Earth system model tuning studies
 GENIE supports environmental scientists modelling long term climate
change.
 Optimising a parameterised model over a multi-dimensional state
space.
 Database used for monitoring, sharing, post-processing and fault
tolerance.
© Geodise Project, University of Southampton, 2001-2005
http://www.geodise.org/
Distributed Systems: 2005 and beyond?
IP
Moore’s Law
ASCII
HTTP
(HTML) XML
(HTML)
Web Services
Network
Compute/ Data
Software
Services
Platform Strategies
Development Tools
Systems Architecture
Distributed Data Management
Problem Solving Environments
Algorithms, Methods and Libraries
Heterogeneous Distributed Computing
Robustness
Security
Reliability
Visualization
Quality of Service
Knowledge Technologies
Collaboration tools & frameworks
Experiences
© Geodise Project, University of Southampton, 2005.
http://www.geodise.org/