Transcript Windows NT Performance Notebook

Virtual Systems
Monitoring and Capacity Planning –
An Update
Demand Technology Windows Symposium CMG – 12/05/2005
Phil Henninge
Demand Technology Software
1020 Eighth Avenue South, Suite 6, Naples, FL 34102
phone: (239) 261-8945 fax: (239) 261-5456
e-mail: [email protected]
http://www.demandtech.com
-5Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
•Charley
August 13, 2004
•Cape
Coral
•Naples
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
2
•Wilma
- October 24, 2005
•Cape Coral
•Naples
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
3
Agenda
Review
 What’s New
 Virtual Machine Timing
 What Lies Ahead
 Sessions Available at CMG
 Questions

© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
4
Introduction

What is a virtual machine?

An abstract machine for which an interpreter exists. Virtual machines
are often used in the implementation of portable executors for highlevel languages.
- Melinda Varian, Princeton University




Java VM
SAS
Visual Basic for Applications – VBA
A software emulation of a physical computing environment


See http://computing-dictionary.thefreedictionary.com
e.g., VM/CMS
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
5
Virtual System Overview
Virtual Machine Operating System
and Applications
Virtual Machine Operating System
and Applications
Virtual Hardware
Virtual Hardware
Guest Virtual Machine
Guest Virtual Machine
Virtual System Software
Host Operating System
Physical Hardware
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
6
Introduction
Who Are the Major Players?

VMWare (wholly owned subsidiary of EMC)




Workstation - powerful virtual machine software for developers and system
administrators
GSX Server -enterprise-class virtual infrastructure for departmental server
consolidation and streamlining development and testing operations
ESX Server -virtual infrastructure software for partitioning, consolidating and
managing systems in mission-critical environments
Microsoft (formerly Connectix).



Virtual PC - a software virtualization solution that allows you to run multiple PCbased operating systems simultaneously on one workstation.
Virtual Server Standard Edition – run on one server with up to 4 processors.
Virtual Server Enterprise Edition – run on one server with up to 32 processors.
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
7
Introduction – VMWare Virtualization
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
8
Introduction - Microsoft Virtualization
From the bottom of the stack:
 The host operating system — Windows
Server 2003— manages the host system.
 Virtual Server 2005 provides a VM
virtualization layer that manages virtual
machines, providing the software
infrastructure for hardware emulation.
 Each virtual machine consists of a set of
virtualized devices, the virtual hardware
for each virtual machine.
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
9
Roles of Virtual Systems in an
Enterprise

Consolidate multiple server workloads.




Re-host legacy applications on newer
hardware.




NT 4.0
W2K
Linux under Windows/Windows under Linux
Enterprise software test and development.


Underutilized Servers
Disaster Recovery
Environmental and TCO (total cost of ownership)
Technology arose for ISV test and development.
Technology demos.
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
10
Performance Monitoring of VSs

At the system level we look at the system resources





CPU Utilization
Memory Utilization (memory consumption and paging)
Disk Utilization
Network Utilization (NIC traffic and topology)
At the software level we look at specific objects.



Process (what are the VMWare and Microsoft specific processes)
Network Interface (what virtual network adapters are defined)
Other Performance Objects
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
11
Monitoring VMware

VMWare object

One instance for each Virtual Machine

Virtual Disk (8 Counters)
–

Guest Locked Memory Bytes
–

The number of bytes of simulated physical memory that is locked by the
guest OS
Guest Virtual Physical Memory Bytes
–

Disk operations (R-W-Total) performed by the guest OS
The number of bytes of simulated physical memory in the virtual
machine
Percent Guest Physical Memory Touched
–
The percentage of simulated physical memory recently used by the
guest OS
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
12
Monitoring VMware

VMWare object (Continuted)

Network Counters (9 Counters)
Network Transfers/sec
Network Bytes Transferred/sec
Network Transfer Errors/sec
Network Packets Sent/sec
Network Bytes Sent/sec
Network Send Errors/sec
Network Packets Received/sec
Network Bytes Received/sec
Network Receive Errors/sec
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
13
Monitoring VMware

Host OS

Processes


Network Interface


Vmware virtual Ethernet adapter VMNet1, .. Adapter VMNet8
VMWare object


Vmnat, vmnetdhcp,vmware,vmware-authd,vmware-vmx
One instance for each Virtual Machine
Guest OS

Processes


VMWareService, VMWareTray, VMWareUser
Network Interface

AMD PCNET Family Ethernet Adapter
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
14
Monitoring Virtual Server

Virtual Processors Object (Virtual PC)

One instance for each Virtual Machine
 Guest External Interrupts
Number of virtual interrupts delivered to guest OS.
 Host-to-VMM Context Switches
Number of context switches between Windows and the guest
(VMM) context.
 Cumulative Guest Run Time
The guest run time represents the number of microseconds
the guest processor has run on a host processor. With
the default scaling, the graph represents guest run time
percentage.
 VMM Exceptions
Number of processor exceptions handled by the VMM.
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
15
Monitoring Virtual Server

The Virtual Server WMI Class contains
two objects



VirtualMachine - CPU, disk, and network usage
counters – an instance for each virtual machine
VirtualNetwork - monitor the usage of each virtual
network (must be attached to a physical NIC – an
instance for each virtual network
For detailed information on these objects:
http://www.microsoft.com/technet/prodtechnol/virtuals
erver/2005/proddocs/vs_tr_tools_WMI.mspx
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
16
Monitoring Virtual Server

Host OS

Processes


Virtual Processors Object (Virtual PC)


Vssrvc (one for each guest machine)
One instance for each Virtual Machine
WMI Objects (Virtual Server)


VirtualMachine
One instance for each virtual machine
VirtualNetwork
One instance for each virtual network
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
17
Monitoring Virtual Server

Guest OS

Processes (after Virtual Machine Additions)


VMSrvc, VMUSrvc, VMPCMap, Interface
Network Interface


Virtual Server - Intel 21140-Based PCI Fast Ethernet
Adapter (Generic) Packet Scheduler Miniport
Virtual PC – Intel DC21140 PCI Fast Ethernet Adapter
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
18
Performance Monitoring of VSs

HALT/Idle Loop Measurement Anomaly

When a machine is idle, its operating system will either issue a HALT
instruction or repeatedly execute an idle loop of NOP instructions




Idle loop is the default for most server machines
Idle loop is a function contained in hal.dll
When a virtual machine executes an idle loop, it is actively executing
instructions which run on the host machine’s physical processor. Thus
performance tools in the guest machine will show inactivity, while the host
machine will appear fully utilized.
Virtual machines running Windows operating systems having the wrong HAL
(Hardware Abstraction Layer) installed will make the guest operating system
spin in its idle loop, instead of HALTing when there is nothing else to do.
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
19
Performance Monitoring of VSs

Halt/Idle Anomaly

Even when the correct HAL is installed, some guest
operating systems HALT more aggressively than others.
 The multiprocessing HAL favors using the Idle loop,
instead of HALTing a processor.
 VMWare reports that W2K frequently spins, whereas
Windows 2003 HALTs whenever it is idle. See AnswerID
1077 in WMWare’s KB:
http://www.vmware.com/support/kb/enduser/std_adp.php?
p_faqid=1077
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
20
Virtual Systems Sizing

VMWare Planning Tools

VMware P2V Assistant
 http://www.vmware.com/products/vtools/p2v_fe
atures.html

VMWare Virtual Infrastructure Methodology
 http://www.vmware.com/pdf/vim_datasheet.pdf
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
21
Virtual Systems Sizing

Microsoft Planning Tools

Microsoft Virtual Server Migration Toolkit
 http://www.microsoft.com/windowsserversystem/vi
rtualserver/overview/vsmtdatasheet.mspx

Solution Accelerator for Consolidating and Migrating
LOB Applications

http://www.microsoft.com/technet/itsolutions/techguide/msa/
solacc/lobsa/default.mspx
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
22
Virtual Systems Sizing


Sizing destination servers requires first understanding
the performance of the applications running on the
source servers.
The VM Host machine must contain sufficient capacity
(Processor, Memory, Disk and Network) to handle the
peak loads of guest machines



accumulate measurement data over long term periods that include
seasonal peaks
compute Peak:Average ratios and understand when peak periods
occur to ensure they do not overlap on the same host
compute 90-95th percentiles
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
23
Virtual Systems Sizing
Metric
Average or Median, Peak
Notes
Processor
Total percent of processor time.
Required to calculate CPU resource allocation
on destination server.
Memory
Available bytes of memory.
This includes the total standby, free, and zero
page lists. Monitor this counter over time and
use the lowest number (minimum value in
Windows Performance Monitor) to
appropriately represent memory consumption
under a load. To express this number in MBs,
divide it by 1,024. Subtract this number from
the installed memory.
Network I/O
Total bytes per second for the network
To determine the need for dedicated or
interface (all instances).
shared network adapter cards on the
destination server.
Disk I/O
Physical disk reads per second (all instances). Include each physical drive used by the
operating system.
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
24
CPU Capacity

The processor requirements of a source server should
not exceed the processor capacity available to a virtual
machine on the destination server.

Normalize based on MHz
CPU requirements = number of CPUs x CPU speed x CPU
utilization

The  % Processor Time for all virtual machines running
on a destination server should be < 90 % of the
available CPU capacity

10% reserved for the host OS and I/O for virtual machine threads.
CPU capacity = number of processors x CPU speed
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
25
Memory Capacity

The total amount configured for all virtual machines
cannot exceed the size of physical RAM
Guest Memory = sizeof(RAM) – Available Bytes (95th percentile)


Every virtual machine requires an additional 32 MB
of physical memory
The host operating system requires exclusive use of
at least 384 MB of memory.
Host Memory Capacity >
384 +
(SizeofVM1+32MB)+(SizeofVM2+32MB)+…+(SizeofVMn+32MB)
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
26
Disk Capacity
The disk must be sized to support 
Physical Disk\Transfers/sec for all guests
I/Os.
 A single drive can sustain 100-200 random
I/Os per second.



Faster disks with 15,000 RPMs and 6 ms seek time
may be able to do better.
See Friedman’s “A simplified approach to Windows
disk tuning” on Tuesday.
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
27
Disk Capacity
The following are best practices for performance optimization on
virtual hard disks:




Use a hard disk solution that allows fast access, such as a locally-attached
SCSI hard disk, RAID, or SAN.
Put each virtual hard disk on a dedicated volume, SCSI hard disk, RAID, or
SAN disk. It is easiest to put virtual hard disks together with their associated
virtual machine configuration files on a RAID or SAN because this keeps
everything in one place.
Reduce disk fragmentation. As a dynamically expanding virtual hard disk
increases in size, it becomes increasingly fragmented. You can defragment the
host operating system to make the virtual hard disk more contiguous. If disk
performance is important, consider doing this. Fixed size virtual hard disks are
allocated a contiguous block of reserved space on the physical hard disk.
Therefore, there is no overhead created by the growing disk.
Compact the virtual hard disks to create more physical disk space.
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
28
Network Capacity
Provide a dedicated network adapter in
the destination server for each network
adapter that existed in the source server.
 Configure at least one additional network
adapter for managing Virtual Server itself
and remote access to virtual machine
consoles.

© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
29
Network Capacity

Load balance.


Add network adapters.


You can load-balance virtual machines for networking. To do this, run a mix of
network-intensive and non-network-intensive applications on a single physical
computer.
For best performance, you should allocate a physical network adapter to each
virtual machine.
Note: Virtual machines cannot take advantage of software-based
network load balancing (NLB)


The Virtual Server network driver runs below the network load balancing driver
in the host operating system network stack.
This isolates each host & guest operating systems.
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
30
Questions?
Resources
 “VM and the VM Community: Past, Present, and Future”
Melinda Varian, Princeton University:
http://pucc.princeton.edu/~melinda/25paper.pdf

Microsoft Virtual Server
http://www.microsoft.com/windowsserversystem/virtualserver/default.mspx
http://www.microsoft.com/virtualserver

VMWare
http://www.vmware.com

Planning Guide for the Virtual Server 2005 Solution
http://www.microsoft.com/technet/itsolutions/techguide/msa/solacc/lobsa/lobsaplg.mspx

VMWare Capacity Planning
http://www.askewview.net/~lxy/VMware/Capacity_Planning.html
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
31
Agenda
Review
 What’s New
 Virtual Machine Timing
 What Lies Ahead
 Sessions Available at CMG
 Questions

© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
32
What’s New with VMWare

ESX Server 3 and VirtualCenter 2
Distributed Availability Services
Distributed Resource Scheduling



VMware Workstation 5.5



64-bit Guest Support for AMD64 and EM64T Systems with VT
Support
Two-way Virtual SMP (experimental)
VMware Player

Enables Anyone to Easily Run, Share or Evaluate Software in a
Virtual Machine on a Windows or Linux PC
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
33
What’s New with Microsoft

Virtual Server 2005 R2

High Availability



Greater Scalability


Up to 50% drop in CPU utilization
Improved Manageability


64-bit (x64) host support
Improved Performance


Clustering virtual machines across hosts
Host clustering support
PXE network boot support
Better Interoperability

Linux guest support
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
34
Agenda
Review
 What’s New
 Virtual Machine Timing
 What Lies Ahead
 Sessions Available at CMG
 Questions

© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
35
Virtual Machine Timing
Whitepaper that discusses timing
mechanisms between operating systems
and hardware.
 Processes within a virtual machine
interact with the virtual machine monitor
instead of the hardware:

http://www.vmware.com/pdf/vmware_timekeeping.pdf
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
36
Virtual Machine Timing



Windows keeps track of time by counting timer
interrupts or timer ticks. When the operating system
starts up, it reads the current time to the nearest
second from the computer's battery-backed (CMOS)
real time clock or queries a network time server to
obtain a more precise time.
To update the time from that point on, the operating
system sets up one of the computer's hardware
timekeeping devices to interrupt periodically at a
known rate (say, 100-200 times per second).
This is timekeeping mechanism is known either as the
periodic interrupt or the quantum in Windows.
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
37
Virtual Machine Timing

Types of Hardware Timers






PIT - Programmable Interval Timer
CMOS RTC - Real Time Clock
Local APIC - Advanced Programmable Interrupt
Controller
ACPI - Advanced Configuration and Power Interface
or Chipset
TSC - Time Stamp Counter
HPET - High Precision Event Timer
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
38
Virtual Machine Timing

Using a hardware interrupt to track time leads to
problems in the guest virtual machine:
 At the moment a virtual machine should generate a
timer interrupt, it may not actually be running. In
fact, the virtual machine may not get a chance to
run again until it has accumulated a backlog of
many timer interrupts.
 Timer interrupts queued up for a single timer device
cause a scalability issue as more and more virtual
machines are run on the same physical machine.
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
39
Virtual Machine Timing

What does this mean for Windows analysts?
“Microsoft Windows has an additional time
measurement feature accessed through the
QueryPerformanceCounter system call. This name
is a misnomer, since the call never accesses the
CPU's performance counter registers. Instead, it
reads one of the timer devices that have a counter,
allowing time measurement with a finer granularity
than the interrupt-counting system time of day
clock.”
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
40
Virtual Machine Timing
Let’s look at the counter PERF_100NSEC_TIMER

Description - This counter type shows the active time of a component as
a percentage of the total elapsed time of the sample interval. It measures
time in units of 100 nanoseconds. Counters of this type are designed to
measure the activity of one component at a time.

Formula - (N1 - N0) / (D1 - D0) x 100, where the denominator (D)
represents the total elapsed time of the sample interval, and the
numerator (N) represents the portions of the sample interval during which
the monitored components were active.

Example
- Processor\ % User Time
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
41
Virtual Machine Timing
Let’s Look at another counter PERF_PRECISION_100NS_TIMER

Description - This counter type shows a value that consists of two counter values:
the count of the elapsed time of the event being monitored, and the "clock" time
from a private timer in the same units. It measures time in 100 nanosecond units.
This counter type differs from other counter timers in that the clock tick value
accompanies the counter value eliminating any possible difference due to latency
from the function call. Precision counter types are used when standard system
timers are not precise enough for accurate readings.

Formula - Nx - N0 / D1 - D0, where the numerator (N) represents the counter
value, and the denominator (D) is the value of the private timer. The private timer
has the same frequency as the 100 nanosecond timer.

Example - PhysicalDisk\% Disk Time
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
42
Virtual Machine Timing
Counter types and example counters in Windows
Server 2003 using timers:





PERF_100NSEC_TIMER – Processor\ % User Time
PERF_100NSEC_TIMER_INV – Processor\ % Processor Time
PERF_AVERAGE_TIMER - PhysicalDisk\ Avg. Disk sec/Transfer
PERF_ELAPSED_TIME - System\System Up Time
PERF_PRECISION_100NS_TIMER - PhysicalDisk\% Disk Time
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
43
What Lies Ahead?

Hardware enabled virtualization

Intel - VT: Virtualization Technology formerly ‘Vanderpool’



Now shipping in desktops – server support in 2006
AMD - Pacifica – first half of 2006
Software




VMWare – ESX will most likely be modified to support hardware
virtualization
Microsoft – “Longhorn Server” will have extensions after it is released
XenSource – Open source consortium targeting VT and Pacifica
Virtuozzo – from SWSoft – uses a customized HAL to virtualize OS
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
44
Agenda
Review
 What’s New
 Virtual Machine Timing
 What Lies Ahead
 Virtual Machine Sessions at CMG 2005
 Questions

© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
45
Virtual Machine Sessions at CMG





314 Virtualization: Concepts, Applications, and Performance
Modeling
 TUESDAY 9:15 AM - 10:15 AM OSCEOLA 3&4
324 To V or not to V: A Practical Guide to Virtualization
 TUESDAY 10:30 AM - 12:00 PM OSCEOLA 3&4
334 Measuring Up for Server Virtualization
 TUESDAY 2:00 PM - 3:00 PM OSCEOLA 3&4
344 Virtual Performance Won’t Do: Capacity Planning for Virtual
Systems
 TUESDAY 3:45 PM - 4:45 PM OSCEOLA 3&4
546 Modeling VMware ESX Performance
 THURSDAY 3:45 PM - 4:45 PM OSCEOLA 1&2
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
46
Questions?
Resources
 Timekeeping in VMware Virtual Machines:
http://www.vmware.com/pdf/vmware_timekeeping.pdf

Microsoft Virtual Server:
http://www.microsoft.com/virtualserver

VMWare:
http://www.vmware.com

Others:
http://www.xensource.com
http://www.swsoft.com/en/products/virtuozzo
http://www.run-virtual.com
http://about-virtualization.com
© 2001-2005 Demand Technology Software, Inc.
Virtual Systems Monitoring and Capacity Planning – An Update
47