Transcript Grid Interoperability Update on GridFTP tests

Grid Interoperability
Update on GridFTP tests
Gregor von Laszewski
[email protected]
http://www.cogkit.org
Outline
 Proposal: Documentation consolidation
 Proposal: GIN GridFTP Report
– Proposal for Survey
– Proposal for User Identification
– Proposal for identification of what tests to execute
 Update GridFTP
– Java CoG Kit Workflow (Testing) Framework
– Testing GridFTP on the TeraGrid
– Testing GridFTP on the GIN Resources
 Summary
Proposal Documentation: Making Interoperability possible
 One aspect of interoperability is based on an effective mechanism to
document and distribute the information.
 Proposal: Consolidate Documentation
– Today: information is spread across several web sites and Wikis
– Build a single entry point for the working group documentation
 Proposal: Improve the contents
– Work on extended report, publishable article
– Most important from my point of view:
• Improve the documentation and description on how to get
accounts on the other “Grids”
 Proposal: Test documentation effort
– Have interoperability tests executed by third parties to test
effectiveness of documentation and approaches
– Engage in an iterative improvement process of the documentation
GIN Today: Documentation
 Documentation links (We need your help to add to this page):
– This should be done during the workshop or as response to the mail
– http://wiki.nesc.ac.uk/read/gin-jobs?HomePage
– https://forge.gridforum.org/sf/wiki/do/viewPage/projects.gin/wiki/GINData
 Problem with OGF/GGF infrastructure
– Gridforge is not used (is it too complicated? Is it too much overhead)
• Only charter is published
– Wikis are the tool of choice for the working group as much of the information
may change quickly. Proposal: use a single Wiki
• If wiki is used a feature is needed to archive the Wiki contents in a single
document.
– OGF schedules do not have pointers to the working group
– Google search in OGF Web page is ineffective
• searching for GIN must give as first link the working group page
– Search in Gridforge is non operational
• Example: search for GIN in wikis on Gridforge returns no results
– Links in the mailing list archives are no longer operational
• Example links to documentation that was send by Erwin in May 06
– Transition to OGF has not been completed.
Proposal: GIN GridFTP Report
 Goal: provide an extended document that includes a formal way of reporting on the
activities of this particular subtask as part of the GIN working group in a single
report.
 Assure that the goals overlap with the working group goals
 Proposed Features that the report may address (in the area of GridFTP)
– Survey of different implementations of GridFTP servers and clients
– Identification of the users for interoperability tests and their goals
• Administrators
• Developers
• Users
– Survey of different tools that monitor GridFTP behavior in production Grids
– Identification of which tests need to be performed to satisfy
• Administrators
• Developers
• Users
– Links to programs and documentation that can perform the tests.
Administrators - Developers - Users
 There are different dimensions to be considered when addressing interoperability
issues
– While administrators may care about setting up tests between virtual
organizations, individual user needs are not completely addressed by this test
as they are not conducted by the user itself.
– Lessons learned over the past 5 years have shown that users MUST have the
ability to perform a set of interoperability test themselves. This includes in
most case the answer to the questions:
• Can I copy a file form here to there?
• What is the performance I get when I do the default?
• …
– Administrator may be more interested to obtain the performance and
throughput of the service.
• Test conducted for each individual user are typically not conducted.
– Developers may be more concerned about functional or protocol
interoperability.
– GIN must address not only administrators and developers, but more
importantly the users.
Update GridFTP: Testing with the Java CoG Kit Workflow
 Java CoG Kit Workflow
– Java CoG Kit workflow can be
run through a remote service, an
API, or a command line tool.
– Java CoG Kit contains easy to
use workflow language that in the
past has been able to bridge
between different Globus
middleware (GT2, GT3, GT4,
SSH, …)
– Language features include
• Functional programming
language expressible in XML
like syntax
• Variable Futures
• Explicit parallelism
• HTML form generation (XUL
like)
 Two concepts:
– Abstractions: copy a file from
here to there regardless of the
implementation / protocol
– Providers: Mapping into Grid
middleware or commodity
technologies
… cont . Java CoG Kit workflow
 Transfers are real simple,
specify source, destination,
and provider
 <project>
 <include file="cogkit.xml"/>
 <execute executable="/bin/date"

stdout="thedate"

host="hot.mcs.anl.gov" provider="GT2"/>
 <echo message="Job completed. Transferring the output"/>
 <transfer srchost="hot.mcs.anl.gov" srcfile="thedate"

desthost="localhost" provider="gridftp"/>
 <echo message="Transfer complete"/>
 <set name="date">

<readFile file="thedate"/>
 </set>
 <echo message="The date is {date}"/>
 </project>
 Technically we could
add other providers
such as
nordu-GridFTP,
RFT,
SRM, …
 It is obvious that this can be used to test not only the file transfers but also
job submissions and other things.
 See Also:
– http://wiki.cogkit.org/index.php/Java_CoG_Kit_Documentation
What could be tested
 File transfer
– Directory creation
– Permissions
– 2nd party transfers
– 3rd party transfers
– …
– All the features that are available
through the Java Cog Kit
abstractions
– We can integrate more features if
necessary
 Job execution
– Similar …
 Other things
– …
 The CoG workflow and its embedded
HTML capabilities make it possible to
create custom tests.
What did we test?
 First we wanted a simple test
– Transfer a file form here to there
– Perform the test by someone
outside of the CoG Kit group to
identify if the tool is
• Easy to install
• Functional
• Robust
– Compare if the tests are similar
to the once obtained from Globus
native GridFTP clients
 Result:
– In all aspects successful
– Through a single shell script the
testing framework can be
installed and run
– The results have been confirmed
with Raj, who is a member of the
GridFTP development team.
 To Do:
– We moved to an SVN archive
this month and need to update
that information so others can
replicate it if they wish)
GridFTPTest.sh
 Installing and Running
– 1. Requirements: Apache Ant and a Java SDK
– 2. Mkdir gin
– 3. Get cog-ftptest-install.sh script from Java CoG Kit SVN and place in gin
– 4. Run cog-ftptest-install.sh
– 5. Edit the hosts file and customize the set of sites to include all hosts you like to
test
• host("<name>”
•
service(provider="gsiftp", type="file", url="gsiftp://<host>[:<port>]")
• )
– 6. Run the tests:
• cog-workflow runtests.k
– 7. The output files should appear in the "output" directory
 Note: We just moved to an SVN directory and some issue with sourceforge SVN
causes a problem upon download. Sourcforge is working with us to resolve it.
 In future we intend to host this in dev.globus.org
GridFTP tests on on TeraGrid (Conducted by Raj K.)
 Verification
– Download
– Install
– Configure
– Run
– No problems
– Tool works
as promised
– Raj is a GridFTP developer and confirmed he gets the same results
with the GridFTP tools
GridFTP test on GIN resources (Conducted by Raj K.)
 Just changing the hosts to the GIN resources results in
 Most Issues are related to the user not being able to
authenticate/authorize
Next Steps
 Complete the documentation of the tool
 Complete the SVN transition
 Have a user apply to all GIN Grid resources and make sure we do not
have authentication issues.
– I thought this is trivial, but in fact it is not.
– Even on TG this is still a challenge today as it is not as simple as just
applying for an account.
Summary: GIN gridFTP from a users perspective
– It is possible to use existing tools such as the CoG Kit to write interoperability
tests that can be executed by the user.
– Tools such as the Java CoG Kit even help briding interoperability issues
– The issue is at this time not interoperability of gridftp servers, we can simply
use the appropriate client. The Java CoG kit has demonstrated that multiclient abstractions are a feasible approach. Hence through a higher level of
abstraction we can provide a considerable notion of interoperability.
– Surprisingly we found that the today’s real issue is that a user has to spend
days/weeks to get onto the Grid due to post account bootstraping of the users
environment to use the Grid.
– Proposal: We should as community make it possible that users can get on the
Grid “fast”
• If it takes a user more than one day to get access to the about 30 Grid
resources we tested with this effort need to do something about it.
 Testing GridFTP on the TeraGrid
 Testing GridFTP on the GIN Resources
Summary