Transcript Virtualisation+Wkshop+071206+-+QUT

VMware as implemented by
the ITS department, QUT
Scott Brewster
7 December 2006
CRICOS No. 00213J
Queensland University of Technology
Note
• IT services at QUT are provided primarily by the
central ITS department and additionally by the IT
departments of various faculties and divisions.
• This presentation focuses on the VMware
implementation managed by the central ITS
department.
(There are other VMware implementations at QUT managed by
faculty IT departments.)
a university for the
real world
R
CRICOS No. 00213J
Overview
• Why VMware?
• VMware software
• Physical hardware
– Host hardware
– Network hardware
– Storage hardware
•
•
•
•
•
Virtual machine configuration
Guest operating-systems
Backup of virtual machines
VirtualCenter
Future directions
a university for the
real world
R
CRICOS No. 00213J
Why VMware?
•
Server consolidation through server
virtualisation
– Relocating instances of operating-systems on
multiple under-utilised physical servers to multiple
virtual machines on a single physical server
– Test and development environments are key targets
for virtualisation
a university for the
real world
R
CRICOS No. 00213J
VMware software
Timeframe:
• Late-2005: Initial deployment: 6 hosts running ESX Server 2.5.2
• Mid-2006:
– Installed ESX Server 3.0 on 8 new hosts
– Migrated virtual machines from 6 original hosts:
• Manually shutdown and migrated existing virtual machines one at a time
from the ESX Server 2.5.2 hosts to the new ESX Server 3.0 hosts, leaving
all ESX Server 2.5.2 hosts empty of virtual machines. Unfortunately required
virtual machine downtime!
– Re-installed ESX Server 3.0 on the original 6 hosts
• Late-2006: Upgraded all hosts to ESX Server 3.0.1
– Used VMotion to migrate all virtual machines from a given host prior to
its updating to ESX Server 3.0.1. No virtual machine downtime required!
• Now: Another 8 new hosts awaiting installation of ESX Server 3.0.1
a university for the
real world
R
CRICOS No. 00213J
Physical hardware
• VMware implementation requires three key types of
physical hardware: hosts, a network, and shared storage
– Hosts:
• 22 Hewlett-Packard (HP) ProLiant-series servers
– Network:
• 1000 Mb/s Ethernet
• Cisco and Nortel network infrastructure
– Storage:
• Local boot disks
• Shared storage provided by SAN
• SAN consists of HP storage arrays and fibre channel switches
a university for the
real world
R
CRICOS No. 00213J
Host hardware
•
22 physical hosts dedicated to VMware implementation:
– 4  HP ProLiant DL380 G4
•
•
•
•
2  3.4 GHz Intel Xeon CPU’s
5 GiB memory
2  200 MiB/s Fibre channel (200-M5-SN-I) ports
4  1000 Mb/s Ethernet (1000BASE-T) ports
•
•
•
•
2  2.2 GHz AMD Opteron (dual core) CPU’s
9 GiB memory
2  400 MiB/s Fibre channel (400-M5-SN-I) ports
4  1000 Mb/s Ethernet (1000BASE-T) ports
•
•
•
•
2  2.6 GHz AMD Opteron (dual core) CPU’s
14 GiB memory
2  400 MiB/s Fibre channel (400-M5-SN-I) ports
4  1000 Mb/s Ethernet (1000BASE-T) ports
– 10  HP ProLiant DL385 G1
– 8  HP ProLiant BL465c G1
a university for the
real world
R
CRICOS No. 00213J
Network hardware
•
Each host has 4  1000 Mb/s network connections:
1. IP subnet 131.181.117.128/25 for the service console
2. IP subnet 10.0.0.0/8 on a dedicated VLAN for VMotion
3. IP subnet 131.181.118.0/24 or 131.181.117.0/25 for use by
virtual machines
4. Additional connection identical to (3) above, for redundancy.
•
Some hosts have an extra 2  1000 Mb/s network
connections:
5. IP subnet 131.181.108.0/24 or 131.181.107.0/24 for use by
virtual machines
6. Additional connection identical to (5) above, for redundancy.
a university for the
real world
R
CRICOS No. 00213J
Network hardware
• Now: External switch tagging (EST) mode
Service
console
Vmotion
module
Vswitch
Vswitch
131.181.117.128/25
10.0.0.0/8
Virtual machines
Vswitch
Vswitch
131.181.108.0/24 131.181.118.0/24
Physical network connections
a university for the
real world
R
CRICOS No. 00213J
Network hardware
• Currently need access to four IP subnets just for
virtual machines with desired access to even
more subnets.
• Intention is to use virtual switch tagging (VST)
mode
– Allows virtual machines to access any subnet
– Provides redundancy for all connections (including
Service Console and Vmotion)
– Allows Vmotion between more ESX Server hosts
a university for the
real world
R
CRICOS No. 00213J
Network hardware
• Desired: Virtual switch tagging (VST) mode
Service
console
Vmotion
module
Virtual machines
Vswitch
Physical trunk connections
131.181.117.128/25
a university for the
10.0.0.0/8 131.181.108.0/24 131.181.118.0/24
real world
R
CRICOS No. 00213J
Storage hardware
• Hosts boot from local disks:
– Local disks (all SCSI) are configured into a RAID-1 logical disk.
– Our non-blade servers use an extra local disk as a hot spare.
• All other storage is shared and presented from a SAN:
– Hosts have dual 200 MiB/s (or 400 MiB/s for newer hosts) fibre
channel connections to the SAN one to each SAN fabric.
(QUT has two identical SAN fabrics for redundancy.)
– HP Storage arrays (EVA8000 in this case) provide shared SAN
LUN’s to the hosts.
– SAN LUN’s for use by VMware are 500 GiB RAID-5 LUN’s.
a university for the
real world
R
CRICOS No. 00213J
Storage
• Each SAN LUN provides the backing for a single ESX
datastore.
• Datastores can span SAN LUN’s but we haven’t tried
this.
• In turn, a datastore can be formatted with the VMFS3
filesystem.
• Virtual machine’s virtual disks are backed by files in
VMFS3 filesystems.
• We keep all of a virtual machine’s virtual disks on the
same datastore.
a university for the
real world
R
CRICOS No. 00213J
Virtual machine configuration
• Currently hosting 64 virtual machines
• CPU:
– Majority of virtual machines configured with a single “virtual” CPU
– Some are configured with dual “virtual” CPU’s
• Memory:
– Majority are configured with 512 MiB or less
– Some use 1 GiB or more
• Network:
– All currently use a single virtual network interface
• Storage:
– Most have a relatively small boot virtual disk with one or more large data
virtual disks
– Some have a larger combined boot/data virtual disk
a university for the
real world
R
CRICOS No. 00213J
Guest operating-systems
• Red Hat Enterprise Linux 4
– 29 virtual machines running this OS
– Even physical host hardware cannot always keep up with the
default system timer rate of 1000 clock interrupts/s. A custom
kernel is therefore required to reduce this rate to 100 interrupts/s
for virtual machines.
– Virtual machine is created manually by system-administrator.
– Operating-system is then installed using network-based Kickstart
process from the university’s Red Hat Satellite. Custom scripts
install additional QUT-specific software and customisation.
– The host is automatically registered for updates as part of the
Kickstart process.
a university for the
real world
R
CRICOS No. 00213J
Installation of guest operating-systems
• Microsoft Windows 2003 Server
– 35 virtual machines running this OS
– Clock interrupts already occur at less than 100 interrupts/s, so no
customisation of the system timer is required.
– Virtual machine is created by cloning a virtual machine template
which has previously been manually installed from a Windows
installation CD. The template is configured to both run Sysprep
and add the instance to the WSUS server for updates.
– The system-administrator then modifies the newly created virtual
machine if extra disks, memory, etc. are required.
a university for the
real world
R
CRICOS No. 00213J
Backup of virtual machines
• No backup of ESX Server hosts is made:
– Virtual machines are stored on the shared SAN LUN’s and can
be restarted from a different ESX Server host if an ESX host is
lost.
• Each virtual machine is backed-up traditionally using a
network-backup agent:
– If a virtual machine is lost is must be recreated and restored from
tape.
• The shared SAN LUN’s are not backed-up:
– If a shared SAN LUN is lost, all virtual machines it contained
must be recreated and restored from tape.
a university for the
real world
R
CRICOS No. 00213J
VirtualCenter
•
VirtualCenter version
–
–
–
•
Client: Only supported on Windows
–
–
•
Dedicated license server running on the same physical host as the VirtualCenter server
VirtualCenter database
–
•
Installed on a physical host
License server
–
•
Linux users have to use Terminal Services client to first connect to Windows host
Virtual consoles become unreliable when this is done – key-press and key-release events
are delayed causing unwanted repetition on virtual consoles
Server: Only supported on Windows
–
•
Late-2005: Initial deployment used VirtualCenter 1.3.1
Mid-2006: Fresh installation of VirtualCenter 2.0
Late-2006: Upgrade to VirtualCenter 2.0.1
Oracle database running under Linux on a physical host
Vmotion
–
–
Separately licensed and additional cost, but essential tool in our experience
Allows on-line migration of virtual machines between physical hosts
a university for the
real world
R
CRICOS No. 00213J
Future directions
• Review virtual machine backup
– Current backup strategy does nothing to reduce the
number of costly network backup licenses required
– Network backups generate a lot of extra network
traffic, which is undesirable on virtual machines
• Configuration of resource pools
– Currently little consideration is being given to
guaranteeing resources for virtual machines
– Appropriately configured resource pools should help
a university for the
real world
R
CRICOS No. 00213J