Transcript PRAGMA19, Sep. 15 Resources breakout Migration from Globus

PRAGMA19, Sep. 15
Resources breakout
Migration from Globus-based Grid to Cloud
Mason Katz, Yoshio Tanaka
Let’s review the past 8 years
• Good things
– It was exciting to build a real Grid infrastructure across
pacific rim.
– Experiences on building PRAGMA Grid gave us a lot of
insights.
– Experiments on PRAGMA Grid gave us a lot of valuable
input.
• Bad things
– Who used, who are using PRAGMA Grid?
– Globus-based Grid is still troublesome to use
• Need to check application run on every site.
– Resources (e.g. network, computing resources) are not
enough to motivate users to use PRAGMA Grid.
Proposed direction
- migration to Cloud • Build once, run everywhere!
• What a user should do?
– Build and test applications on his local machine.
– Create a VM image which is able to run on any
cloud resource.
• What a resource provider should do?
– Build VM hosting servers.
VM Replication Experiment
http://goc.pragma-grid.net/wiki/index.php/VC-replication-2
• VM hosting server:
•Rocks 5.3 Xen roll
• Avian Flu Grid VM
• Rocks VM
• Globus/SGE
• Autodock
• Replication updates
• hostname and IP
• Compute nodes
• Network configurations
• Globus configuration
• SGE configuration
User Interface
AutoDock
NAMD
Service Manager
OPAL2
Resource Manager
(edu.sdsc.nbcr.opal.manager.CSFJobManager)
generate RSL files
Scheduling:
Workflow Job
CSF4
Array Job
Input/Ouput Files:
StageIn and StageOut
MetaScheduler
Grid Resources
Grid Sites
Biomedical CLOUD
AFG VM
(original)
SDSC VM hosting server
AFG VM
(copy)
NBCR VM hosting server
AFG VM
(copy)
AIST VM hosting server
GEO Grid System Zones
Google Service map
Internet
Network SW
Frontend service
portal, workflow,
Cloud Service
WPS
zone
CS-W?
applications, libraries
DEM, GRASS, GDAL,
WMS, WCS, WFS, CS-W?
Data Service
zone
archive
Laboratory zone
software develop
application develop
home
5
Agenda of breakouts
• Have consensus of migration
• Figure out possible problems on migrating to Cloud
– Does anybody need Globus-based Grid?, i.e. is Globus still
mandatory?
• Figure out missing pieces. What do we need to make
PRAGMA Cloud a practical infrastructure?
– How make it easy to create VM images?
• Start with creating baseline VM images
–
–
–
–
–
Interaction with data cloud
Meta scheduling / resource brokering
Security
Interoperation between VM hosting servers
…
• Rough schedule and milestones
• Detailed discussions will be in this afternoon
Steps
1.
Identify sites
1.
2.
Identify what baseline images should be published
1.
2.
3.
3.
Publish a baseline Condor image (Phil UCSD by PRAGMA20)
Gfarm client in baseline image
Independent test is necessary
Automatic replication (Solve re-configuration / re-forming
problems) (Mason by PRAGMA 20)
1.
4.
AIST, UCSD, NCHC, JILIN, Osaka, Indiana
Hopefully by PRAGMA20@HK
Consider how to share VM images
1.
2.
Build a stable Gfarm filesystem. (UCSD, AIST, Tsukuba)
Build and use Gfarm image on Virtual clusters at SDSC and AIST. (by
PRAGMA20)
1.
5.
Interoperability testing
1.
6.
physical clusters provided by SDSC/AIST. Tatebe will lead deployment)
Future issues
Other research issues to be done (by each institution)
1.
Security, meta-scheduling, data, …