Transcript ppt

ILDG Middleware Status
Presented By: Bálint Joó, Jlab, USA
Working Group Members:
G. Beckett (EPCC, UK)
T. Boku (CCS Tsukuba, Japan)
D. Byrne (EPCC, UK)
P. Coddington (CSSM, Australia)
M. Ernst (DESY, Germany)
M.Sato (CCS Tsukuba, Japan, Convener)
J. Simone (Fermilab, USA),
W. Watson (Jlab, USA, Convener)
Contents
●
Reminder of Our Task
●
Reminder of Our Architecture
●
Summary of Current Discussions
●
Future plans
Our Mission
●
●
We develop standard interfaces that
facilitate the interoperation of a grid of
grids
Local grids
–
Control Local Policies (Eg Security)
–
Control Local Grid Implementations
●
Choice of software components
Our Architecture
●
Simple veneer over local implementations
–
Minimally intrusive (wrapper?)
–
Not too burdensome to implement
●
Language and platform neutral if possible
●
Our choice: SOAP Web Services
Our Architecture: Components
XML ILDG
service
description file
Metadata Catalogue
Web Service
(MDC)
Security Infrastructure
(Identity management
access control
encryption)
Logical File Name (LFN, formerly GFN)
Replica Catalogue
Web Service
(RC)
SRM service
(data management)
ILDG Service File
●
●
●
Publishes Service Endpoints
Each collaboration currently has entries
for MDC and RC
Started filling this out at MW Meeting in
Tsukuba
●
http://www.lqcd.org/ildg/Services.xml
●
http://www.lqcd.org/ildg/Services.xsd
●
Entries for LatFor & CSSM
Logical File Names (LFN)
●
Changed to LFN from GFN to fit better
with MDWG usage
●
Unique and Persistent name for files
●
MWWG mandated URI format
lfn://<ILDG Collaboration>/<Collaboration Specific Path>
Collaboration indexes service
file
● Collaborations control their own
path
●
MDC Web Service
●
Maps metadata to LFNs
●
Provides “Virtual XML Database”
●
Prototype WSDL definition by M. Sato
●
●
Production MDC Web Services deployed
by LatFor & CSSM
Interface definition in flux
–
Raw queries – variable return types
–
Targeted Queries – return LFNs
Replica Catalog Service
●
Maps (data) LFNs to URL
●
Intention was to map to SRM URL
–
●
SRM component deals with transfer
SRM requirement temporarily relaxed –
RC can return any URL
●
No standard download protocol
●
Client must have appropriate tool
●
Prototype implementation at Jlab
SRM Service
●
SRM definition v3 almost ready
–
–
●
●
Fewer backward incompatible changes
than v1->v2 transition.
Migration should be simple(r)
Popular dCache storage management
software still uses v1.x (Fermilab, DESY)
but Jlab uses v2. v1 and v2 incompatible
ILDG uses v2 SRM, but SRM use is not
mandated currently
Security Infrastructure
●
Currently quite nebulous
●
Previously defined 'ILDG Group Files'
–
–
XML Documents containing user
certificates
Not very grid-like, not best match with
LatFor preference/implementation
●
Currently considering VOMS service
●
Security not explicit in current interfaces
WG Organization
●
●
●
Had very nice and productive meeting in
Tsukuba in Oct 2005. Minutes on ILDG
web site.
Instituted more frequent teleconference
calls – first of these was on Dec 5.
Minutes on ILDG web site. Next call Jan
23, 2006.
Middleware BLOG set up on ILDG web
site to aid reporting progress.
Plans
●
●
●
Finish up discussions and make a decision
on using VOMS by end of February
US basic service up with some data by end
of January.
EPCC to convert MDC GUI browser to
work with Web Services
●
Interoperability testing Feb-May 2006.
●
Interoperable system by June 2006.
Current Discussions
●
MDC
–
Need to fix interface
●
●
Small inconsistencies between WSDL from
LatFor and Japan – to be fixed
Clarify semantic meaning of query
functions
–
–
–
Raw queries – return results appropriate to
query
LFN queries – return markovChainUR or
dataLFN
Need document retreival queries ?
Raw Queries:
<?xml version=”1.0” ?>
<gaugeConfiguration ...>
<management/>
<implementation/>
<algorithm/>
<precision>
<markovStep>
<markovChainURI>lfn://myCollab/myEnsemble</markovChainURI>
<series>1</series>
<update>010170</update>
<avePlaquette>0.53380336E+00</avePlaquette>
<dataLFN>lfn://myCollab/myFilename</dataLFN>
</markovStep>
</gaugeConfiguration>
XPath funcion
Raw Query:
Returns:
Raw Query:
Returns:
number(/gaugeConfiguration/markovStep
[dataLFN="lfn://myCollab/myFilename"]/avePlaquette)
0.53380336
XPath Number Type
/gaugeConfiguration/markovStep
[markovChainURI="lfn://myCollab/myEnsemble"]/dataLFN
<dataLFN > lfn://myCollab/myFilename </ dataLFN >
XML Tags
LFN Queries:
<?xml version=”1.0” ?>
<gaugeConfiguration ...>
<management/>
<implementation/>
<algorithm/>
<precision>
<markovStep>
<markovChainURI>lfn://myCollab/myEnsemble</markovChainURI>
<series>1</series>
<update>010170</update>
<avePlaquette>0.53380336E+00</avePlaquette>
<dataLFN>lfn://myCollab/myFilename</dataLFN>
</markovStep>
</gaugeConfiguration>
Single Return Type: Array of LFNs for all queries
LFN Query:
Returns:
LFN Query:
Returns:
/gaugeConfiguration/markovStep[update="010170"]
{ lfn://myCollab/myFilename }
/gaugeConfiguration/markovStep
[markovChainURI=”lfn://myCollab/myEnsemble”]
{ lfn://myCollab/myFilename, lfn://myCollab/myFilename2,... }
Comparison
●
Raw Queries
–
Powerful: Need Only 1 Query function
●
–
Can use XPath functions eg: to count
matches
Query returns different types
●
Eg: numbers, strings, XML Snippets
–
Can use to implement LFN Queries
–
Also good for Web interfaces, testing
Comparison
●
Domain Specific (eg LFN) Queries
–
Less Flexible
●
Can't execute arbitrary XPath functions
–
–
More complex
●
Need more functions:
–
–
Query/Get Config XML , Query/Get Ensemble
XML
Tailored to Particular Use Cases
●
–
More logic in clients
Eg: Command Line Tools (find & download)
Don't need XML database necessarily
Security Infrastructure
●
●
●
●
Can use SSL to encrypt communications
with services
Can have certificate based authorization
in Web servers, Application servers, and
even some databases
Use firewalls to secure back ends
Credential forwarding for service
aggregator portals may need thought
Certificate Exchange
Service (MDC/RC)
Clear
Encrypted Connection
Client
Certificate Exchange
Client Encrypted Connection
Aggregate Service
(MDC/RC)
Thought
Needed
Certificate forwarding? Proxying? How to maintain secure connection??
UK Service
(MDC/RC)
US Service
(MDC/RC)
LatFor Service
(MDC/RC)
Japanese
Service
(MDC/RC)
Australian
Service
(MDC/RC)
Firewall
Actual
Database
Actual
Database
Actual
Database
Actual
Database
Actual
Database
VOMS: Virtual Organisation
Management Service
●
Technology for identity and role
management
●
Distributed and decentralized
●
Part of LCG, OSG and gLite
●
LatFor moving to production use
●
To be trialled in UK and Japan
●
In US, Fermilab will use infrastructure on
collaborative project with DESY
VOMS
●
●
Interest in VOMS expressed by Australia
WG to come to a decision on standardising
use of VOMS by end of February 2006
–
–
Personal worry: This is choosing
software rather than defining an
interface.
Pragmatism: LatFor requires users'
certificate in their VOMS server. Will
eventually hold everyone's certificate
LCG Software Components
●
Michael recommended use of LCG
software:
–
–
–
Rich in features, presented as RPMs
through software repositories, with
installation tools (apt-get, yaim)
Can install just components we need
Can use any Red Hat Enterpise Linux
like system (Scientific Linux (SL),
RHEL, Fedora Core) CERN supports SL.
LCG Components
–
–
–
●
Large User Community (LHC, OSG)
Used by LatFor (File Catalog – LFC, and
VOMS)
In US Fermilab will have infrastructure
BUT
–
Don't want to force particular
technology/software. That's not our
mandate !
Conclusions
●
Components coming along – individual
collaborations will present own status
reports
●
MDC interface solidifying currently
●
Most groups have received funding/effort
●
●
Most groups have local infrastructures
and are now working on ILDG issues
I have a LOT of work ahead of me.
US Middleware status
●
●
●
Effort recently revitalised with arrival of
Bálint at the Jlab.
General Web Service Infrastructure:
–
Apache Web Server
–
Tomcat Application Server
–
Axis Web Services Framework
–
X509 Certificates, SSL encryption
Prototype Data: MILC Datasets from
NERSC Gauge Connection
US Middleware Status
●
RC Prototype ready for some time
–
●
Need secure connection and certificate
based authentication for production
use.
RC needs data in it to be useful.
–
In discussion with MILC to choose
suitable LFNs for data in NERSC
archive. Recent helpful changes in
metadata schema facilitate things.
US Middleware Status
●
In analysis stage for MDC
–
Plan: use eXist native XML database
●
Pure Java
●
Integrates well with Tomcat and Axis
●
Comes with Web Service API
●
Sophisticated XML capabilities:
–
●
Validation against Schemata, XQuery, XUpdate
Expect very thin ILDG layer on top
US Middleware Status
●
Progress with ILDG File Format
–
–
–
Conversion routine from NERSC format
to ILDG format exists within MILC
code and QDP++ using QIO library.
MILC Code can read QDP++ files and
those provided by Dirk Pleiter (LatFor)
Dirk could read files produced by
QDP++ and MILC
US Middleware Status
●
File format (cont'd)
–
Only minor discrepancy
●
QIO uses LIME type: ildg-data-lfn
●
Dirk's code & spec give: ildg-data-LFN
●
Dirk suggests case insensitivity in codes
–
–
Backward compatible solution for all sides.
Carleton willing to change QIO to use case
insensitive checking of LIME types.