Transcript Introduction to High Performance Computing

Introduction to
Grid Computing with
High Performance Computing
Outline
•
•
•
•
•
Objectives
Training Units
Introduction to Grid Computing
Registration with the White Rose Grid
Information Sources
Learning Outcomes
•
•
•
•
•
Develop a High Performance computing (HPC) application.
Develop and manage applications for Computational grids
Manage the execution of HPC applications over a grid
computing system.
Discover resources, applications and data on distributed
systems.
Demonstrate an appreciation of evolving Grid Technologies.
Sections
• Introduction to the White Rose Grid and grid
operating systems.
• Application development for computational grids.
• Development tools and techniques for distributed
high performance computing applications.
• Grid technologies and example grid projects
Assessment
• 3 Sets of assessment problems
• A mini project on grid computing with a 3000 word
report
• Marks for the course are broken down as follows
– 40% from assessed problems
– 60% from mini project
Grid Computing Mini Project
• Grid Applications for Specific Disciplines
– Design of a grid application
– Implementation of a prototype
– An essay on the grid technologies that will influence
different disciplines and how this will be achieved.
Introduction to the White Rose Grid and
Grid Operating Systems
• Introduction to Grids, Registration and Access
• Application management and development using the
UNIX operating system.
• Review of White Rose Grid architecture and
applications.
• Job Management and Scheduling.
• Middleware for grid computing.
Application Development for
Computational Grids
• Using and developing matlab applications for the grid.
• Development of HPC applications using the C and Fortran
• Review of object orientation techniques in the development of
HPC applications.
• Application Development using C++
• Application development environments for the White Rose Grid
Development tools and techniques for
distributed high performance computing
applications.
• Libraries for HPC Application Development on the
WRG.
• Performance Assessment Tools
• Distributed Application Development Using MPI
• Distributed Application Development Using Open MP
Grid technologies and example grid
projects
• Introduction to Grid Services and the Open Grid
Services Architecture.
• Development of Grid Portals
• Implementing a project as a grid service, The DAME
project.
• Cluster building and Integration with Grids
• From semantic web to semantic grids
– Distributed searching of textual databases
Grid Computing is New Technology
• Grid computing is in the
pioneering age!
• As grids and grid technology
evolves it is to be expected
that the contents of course
material will change.
Introduction to Grid Computing
•
•
•
•
Grid Technologies
Grid Definition
Types of Grid
Grid Architecture
Grid Technologies
Grid Technologies
•
•
•
•
Large scale multi site data mining and data fusion
Shared virtual reality
Interactive collaboration
Real-time access to remote resources.
Access Grid
Definition of the Grid Problem
The “Grid Problem” is defined as the flexible,
secure, coordinated resource sharing among
dynamic collections of individuals, institutions
and resources.
From “Anatomy of the Grid (Enabling Scalable Virtual Organisations)”
I.Foster et al. Intl. J. Supercomputer Applications 2001.
Characteristics of Grids
•Global collections of resources connected with high speed
networks
•Supercomputers, databases, storage, instruments, immersive environments
•Next generation internet
•Emerging technology
•Cost effective management of high end compute and data
resources
Grid Architecture
Networks – High Bandwidth
The
Grid
Computing
Tflops
Data storage
Peta byte
Virtual
Organisations
Evolution of Networked Computing
Architecture
Client
Server
Distributed
Web
Applications
Web
Services
Grid
Applications
Grid Services
Description
Network is
computer
Objects
The web
Computer
is Network
Virtual
Organisations
Globally
Extended
Virtual
Organisations
Peak
Late 80’s
Early 90’s
Late 90’s
Early 2k
Late 2k??
??
Protocol
X
X
HTTP, JVM
XML,
LDAP
SOAP
WSDL
GRIP, GRAM,
GFTP,
LDAP
MDS,UDDI
The Internet and Dynamic Web
Technology
The Web Application Model
Web Services
Web service are OK but…
They can only run on the computer
Where they are put.
The computer or data
availability may not be fit for the task.
Web Services Model
Types of Grid
Computational Grid
Distributed Supercomputing
Grid aggregates computational
resources to compute large
complex problems
High throughput computing
Grid Schedules large numbers
of loosely coupled or
independent tasks
On Demand Computing
Meet short term requirements
for projects that cannot be
cost-effectively or
conveniently computed locally
Data Grid
Data Intensive Computing
Grid Synthesises from data
managed on geographically
distributed repositories
Access Grid
Collaborative Computing
Enhancing human-to-human
interactions, e.g. collaborative
design and virtual worlds
Grid Services
Web services on
Steroids!
They go anywhere!
Take on any task!
Grid Services
Grid Architecture
APPLICATION
Molecular Modelling, Aerospace, Graphics and
Visualisation
APPLICATION TOOLKITS
Data Grid, Remote Computation, Portals, Collaborative
Visualisation, Sensors
GRID SERVICES
Protocols, Authentication, Policy, Resource
Management, Instrumentation
GRID FABRIC
Storage, Computers, Networks, Sensors, Devices,
Experiments
Grid Examples
•NASA’s Information Power Grid
•computing infrastructure connecting the
resources of several of its R&D Laboratories
forming the Information Power Grid or IPG
•High Energy Physics E Data Grid
•The DataGrid Project is providing the
solution for storing and processing data
generated by the LHC at CERN.
Grid and Web Service Demonstrations
• SETI@home
• The Dame Portal
– Requires valid e-Science Certificate
•
•
•
•
Virtual Observatory
European Data Grid Demonstrators
OpenGIS Web Service
Matlab data Mining Service
Statistics for SETI at Home
(16/10/2003)
Total
Last 24 Hours
Users
4710399
1534
Results received
1073252223
Total CPU time
1655366.998 years
1347.953 years
Floating Point
Operations
3.769869e+21
5.372156e+18
(62.18
TeraFLOPs/sec)
Average CPU time
per work unit
13 hr 30 min 40.6
sec
8 hr 34 min 20.0 sec
1377476
SETI@home’s Most Promising Candidates
The DAME Portal
• Distributed Aircraft Maintenance Environment
• Portal runs engine vibration analysis tools on
selected sets of engine performance data.
• http://iri02.leeds.ac.uk:9080/damexto/damexto
Data Grid
• Portal Demonstrators
– Medical Imaging
– Bioinformatics
• http://edg-wp10.healthgrid.org
Visit the Virtual Sky Portal
http://virtualsky.org
OpenGIS Web Service
• http://www.opengis.org/resources/?page=demos
• Standards based
• Data Connections
– Forestry information
– Satellite data Landsat 7, ESA Global imagery data
– Atlas data, Roads, boundaries, populations, water areas..
• Based on Web services enables easy integration with
OGSA based Grid Services
e-Science Projects
Lots of projects Here are a few!!
•Distributed Aircraft Maintenance Environment
•Distributed Collaborative Visualisation and
problem solving environments
•Clinical and Biomedical projects
• A co-operative clinical e-Science Framework
•Biosim GRID
•Decision making in the health sector
White Rose Grid Registration
• Conditions of Use
• Becoming a White Rose Grid user
• Obtaining an e-Science certificate
Conditions of Use
• White Rose Grid users must abide by the conditions
pertaining to the individual systems at York, Leeds
and Sheffield.
– http://www.shef.ac.uk/calendar/compregs.html
– http://www.leeds.ac.uk/iss/rules/ISShtml
– http://www.wrg.york.ac.uk
Becoming a White Rose Grid User
1. Register for an account on your local system
2. Apply for White Rose Grid resources
3. Obtain an e-Science certificate
Why obtain an x509v3 Certificate
• Enables secure single sign on to the White Rose Grid
• Use portals e.g. the DAME portal
• Use Globus to access WRG compute nodes
Register for an Account on Your Local System
Institution
Registration Details
The University of
Sheffield,
Titania
Email: [email protected]
Web:
http://www.shef.ac.uk/cics/support/userreg/r
egistration.html
The University of http://www.leeds.ac.uk/iss/helpdesk/
Leeds,
usernames.html
Maxima,Snowdon
The University of
York, Pascali
http://www.wrg.york.ac.uk/access.html
Application for White Rose Grid Resources
• Details at http://www.shef.ac.uk/wrgrid/access
• Complete the Application form “Application for a
Username and Resources on the White Rose Grid”
• Forward application to local member of the White
Rose Grid executive and obtain Authorisation.
• Await confirmation of registration
Accessing the White Rose Grid
• From a UNIX account node access using ssh or
XServer application (e.g. Exceed
• Using Globus with e-Science Certificate
• White Rose Grid Portal (requires e-Science
certificate)
Accessing the White Rose Grid
• Resources
–
–
–
–
titan00.shef.ac.uk (10 Sun V880 Sun Fire Servers)
pascali.york.ac.uk (1 Sun V880 Sun Fire Server)
maxima.leeds.ac.uk (5 Sun V880 Sun Fire Servers)
snowdon.leeds.ac.uk (256 node Intel Beowulf cluster)
• From a UNIX account node access using ssh
– ssh –l wrsmyname –X [node name].ac.uk
• Allows X applications from maxima and pascali
References
•
•
•
•
•
•
•
The White Rose Grid http://www.wrgrid.org.uk
http://www.shef.ac.uk/wrgrid/trainingresources
http://www.shef.ac.uk/wrgrid/access
Grid Support Centre http://www.grid-support.ac.uk
National e-Science Centre http://umbriel.dcs.gla.ac.uk/NeSC
UK DTI e-Science programme http://www.escience-grid.org
Global grid forum http://www.ggf.org
Registration
• Register with the White Rose Grid
• Obtain an e-Science Certificate
Go to the link
http://www.shef.ac.uk/wrgrid/access/index.html
Grid Computing References
• The Grid: Computing Without Bounds
– Ian Foster, Scientific American, April 2003.
• “The Anatomy of the Grid”
– http://www.globus.org/research/papers/anatomy.pdf
• Grid Services – “The Physiology of the Grid”
– http://www.gridforum.org/ogsi-wg/drafts/ogsa_draft2.9_2002-0622.pdf
• Research Agenda for the Semantic Grid
– http://www.semanticgrid.org/v1.9/semgrid.pdf
Demonstrators To Try
•
•
•
•
•
•
http://setiathome.ssl.berkeley.edu/
http://virtualsky.org
http://edg-wp10.healthgrid.org
http://iri02.leeds.ac.uk:9080/damexto/damexto
http://www.opengis.org/resources/?page=demos
http://eepc269.eng.ohio-state.edu/matlab/