Transcript Clouds in education

Virtual Appliances and Education
using Clouds
Dr. Renato Figueiredo
ACIS Lab - University of Florida
Advanced Computing and Information Systems laboratory
Background

Traditional ways of delivering hands-on
training and education in parallel/distributed
computing have non-trivial dependences on
the environment
• Difficult to replicate same environment on different
•

resources (e.g. HPC clusters, desktops)
Difficult to cope with changes in the environment (e.g.
software upgrades)
Virtualization technologies
•
•
•
Remove key software dependences
Allow packaging and replication of hands-on,
executable educational environments
Can be deployed, managed with cloud technologies
Advanced Computing and Information Systems laboratory
2
Overview




Virtual appliances
Virtual networking
•
Grid appliances and FutureGrid
Support for classes in FutureGrid
Demo
Advanced Computing and Information Systems laboratory
3
Guiding principles

Fidelity: activities should use full-fledged,
executable software: education/training
modules
• Learn using the proper tools

Reproducibility: Creators of content should
be able to install, configure, and test their
modules once, and be assured of the same
functional behavior regardless of where the
module is deployed
• Incentive to invest effort in developing, testing and
documenting new modules
Advanced Computing and Information Systems laboratory
4
Guiding principles

Deployability: Students and users should
be able to deploy modules in a simple
manner, and in a variety of resources
• Reduce barriers to entry; avoid dependences
•

upon a particular infrastructure
Interactive use
Community-oriented: Modules should be
simple to share, discover, reuse, and
expand
• Create conditions for “viral” growth
Advanced Computing and Information Systems laboratory
5
Role of clouds and portal

Executable modules – virtual appliances
• Deployable on FutureGrid resources
• Deployable on other cloud platforms, as well
as virtualized desktops

Community sharing – Web 2.0 portal,
appliance image repositories
• An aggregation hub for executable modules
and documentation
Advanced Computing and Information Systems laboratory
6
What is an appliance?

Hardware/software appliances
• TV receiver + computer + hard disk + Linux +
user interface
• Computer + network interfaces + FreeBSD +
user interface
Advanced Computing and Information Systems laboratory
7
Virtual appliance example

Linux + Apache + MySQL + PHP
A web server
Another Web server
LAMP
image
instantiate
copy
Virtualization
Layer
Repeat…
Advanced Computing and Information Systems laboratory
8
Educational appliances


A flexible, extensible platform for hands-on,
lab-oriented education on FutureGrid
Support clustering of resources
•
Virtual machines + virtual networking to create
sandboxed modules
• Virtual “Grid” appliances: self-contained, pre-packaged
•
execution environments
Group VPNs: simple management of virtual clusters by
students and educators
Advanced Computing and Information Systems laboratory
9
Virtual Networking


A single appliance encapsulates
software and configuration
Cluster/Grid/Cloud computing
• Middleware expects a collection of machines,
•
•
typically on a LAN (Local Area Network)
Appliances need to communicate and
coordinate with each other
Each worker needs an IP address, uses
TCP/IP sockets
Advanced Computing and Information Systems laboratory
10
Virtual cluster appliances

Virtual appliance + virtual network
Virtual
network
Hadoop
+
Virtual
Network
A Hadoop worker
Another Hadoop worker
instantiate
copy
Virtual
machine
Repeat…
Advanced Computing and Information Systems laboratory
11
Grid appliance in a nutshell

Plug-and-play clusters with a preconfigured software environment
• Linux + (Hadoop, Condor, MPI, …)
• Scripts for zero-configuration

Hands-on examples, bootstrap
infrastructure, and zero-configuration
software – you’re off to a quick start
Advanced Computing and Information Systems laboratory
12
Virtual Network - GroupVPN

Setting up and managing typical VPNs
can be daunting
• VPN server(s), key distribution, NAT traversal


GroupVPN makes it simple for users to
create and manage virtual cluster VPNs
Key insights:
• Web 2.0 interface: create/manage user groups
• All the complexity of setting up and managing
VPN links is automated
Advanced Computing and Information Systems laboratory
13
Deploying virtual clusters

Same image, different VPNs
Group
VPN
Hadoop
+
Virtual
Network
A Hadoop worker
Another Hadoop worker
instantiate
copy
GroupVPN
Credentials
Virtual
machine
Repeat…
(from
Web site)
Virtual IP - DHCP
10.10.1.1
Virtual IP - DHCP
10.10.1.2
Advanced Computing and Information Systems laboratory
14
Support for classes on FutureGrid


Classes are setup and managed using
the FutureGrid portal
Project proposal: can be a class,
workshop, short course, tutorial
• Needs to be approved by FutureGrid project
to become active

Users can be added to a project
• Users create accounts using the portal
• Project leaders can authorize them to gain
•
access to resources
Students can then interactively use FG
resources (e.g. to start VMs)
Advanced Computing and Information Systems laboratory
15
Use of FutureGrid in classes

Cloud computing/distributed systems
classes
• U.of Florida, U. Central Florida, U. of Puerto
Rico, Univ. of Piemonte Orientale (Italy), Univ.
of Mostar (Croatia)

Distributed scientific computing

Tutorials, workshops:
• Louisiana State University
• Big Data for Science summer school
• A cloudy view on computing
• SC’11 tutorial – Clouds for science
• Science Cloud Summer School (this month)
Advanced Computing and Information Systems laboratory
16
Cloud computing classes


Massimo Canonico, U. Piemonte Orientale
Difficulties to overcome:
• Hardware issues: find enough free physical
•
•

machines able to host virtual machines
Software issues: time to install/configure as many
as possible different cloud platforms
University was not able to provide me the
necessary hardware and software support
Students started to play with FutureGrid
• After attending few lessons, they were able to
start/stop virtual instance with several Cloud
Computing platforms
Advanced Computing and Information Systems laboratory
17
Cloud computing classes

Students used Eucalyptus, OpenStack
and Nimbus
• Half were not computer scientists.

As FutureGrid freely shares their
physical machines and their cloud
platforms, decided to freely share all
materials of my class.
• Hands-out, configuration files and link to
•
useful documentation are available
https://portal.futuregrid.org/contrib/cloudcomputing-class
Advanced Computing and Information Systems laboratory
18
Cloud computing classes

Graduate-level “Cloud computing for
Data-Intensive Sciences” (Judy Qiu, Fall
2010)
• Virtualization technologies and tools
• Infrastructure as a service
• Parallel programming (MPI, Hadoop)
• FutureGrid provided a set of software options
that made it possible for students to work on
different projects along the system stack.
Advanced Computing and Information Systems laboratory
19
Term Projects
Higher
Level
Languages
Cloud
Platform
Cloud
Infrastructu
re
Hypervisor/
Virtualizatio
n
Dryad/DryadLINQ
#1 Matrix Multiplication (Swapnil,Amit,Pradnay)
#2 PhyloD (Ratul,Adrija,Chengming)
Iterative MapReduce
#3 LDA (Changsi, Yang)
#4 MemCache (Saliya, Yiming ,Jerome)
#5 Avro (Yuduo, Yuan, patanachai)
#6 PageRank (Shuo-Huan,Parag)
Cloud Infrastructure
#7 Nimbus, Eucalyptus (Stephen, Sonali, Shakeela)
Cloud Storage
#8 Cloud Storage Survey (Xiaoming, Nixiaogang)
Virtualization
#9 Hypervisor Performance Analysis Project (James ,
Andrew)
Advanced Computing and Information Systems laboratory
(Slide courtesy of Judy Qiu)
Big Data for Science
300+ Students (200 on sites from 10 institutes; 100 online) July 26-30, 2010 NCSA Summer School Workshop
IU MapReduce and UF Virtual Appliance technologies are supported by FutureGrid. http://salsahpc.indiana.edu/tutorial
Washington
University
University of
Minnesota
Iowa
State
IBM Almaden
Research Center
Univ.Illinois
at Chicago
Notre
Dame
University of
California at
Los Angeles
San Diego
Supercomputer
Center
Michigan
State
Johns
Hopkins
Penn
State
Indiana
University
University of
Texas at El Paso
University of
Arkansas
University
of Florida
(Slide courtesy of Judy Qiu)
Advanced Computing and Information Systems laboratory
Demonstration




Deploying virtual appliance node on
FutureGrid (Nimbus @ Alamo)
Connecting to virtual machine
Virtual networking
Running sample job
Advanced Computing and Information Systems laboratory
22
Demonstration

Pre-instantiated VM to save us time:
• cloud-client.sh --conf alamo.conf --run --name
grid-appliance-2.05.03.gz --hours 24

Connect to VM

Check virtual network interface

Ping other VMs in the virtual cluster
Submit Condor job

• ssh root@VMip
• ifconfig
Advanced Computing and Information Systems laboratory
23
Uploading and sharing images


APIs available to upload images,
customize, save, and share images
Community education pages are
available
• FutureGrid Web portal allows users to publish
•
their own content
Tutorials, presentations on Web portal; VMs
on image repositories
Advanced Computing and Information Systems laboratory
24
Where to go from here?

Tutorials on FutureGrid and Grid
appliance Web sites for various
middleware stacks
• Condor, MPI, Hadoop

A community resource for educational
virtual appliances
• Success hinges on users effectively getting
•
•
involved
If you are happy with the system, let others
know!
Contribute with your own content – virtual
appliance images, tutorials, etc
Advanced Computing and Information Systems laboratory
25
Questions?

More information:

This document was developed with support from the National
Science Foundation (NSF) under Grant No. 0910812 to Indiana
University for "FutureGrid: An Experimental, High-Performance
Grid Test-bed." Any opinions, findings, and conclusions or
recommendations expressed in this material are those of the
author(s) and do not necessarily reflect the views of the NSF
• BOF: tomorrow 4:45pm, Burnham (8th floor)
• http://www.futuregrid.org
• http://grid-appliance.org
Advanced Computing and Information Systems laboratory
26