Transcript Support

TA03 - VMware Infrastructure 3.5:
New Networking Concepts and
Advanced Troubleshooting
Emiliano Turra
VMware Technical Support
Escalation Engineer
Agenda
New Networking Concepts in 3.5
Advanced Troubleshooting
CDP
Common issues, troubleshooting hints and
IPv6
recommendations:
Enhanced VMXNET
PortGroups
Jumbo Frames
Nic-Teaming
TCP Segment Offloading (TSO)
NIC Failover and Failback
10GigE
VLAN
NetQueue
Duplicated IP address
InfiniBand (Community Support)
Domain Name System (DNS)
NetFlow (Experimental)
Multihomed ESX
VMKernel Routing table
VirtualCenter and Network Address Translation (NAT)
Virtual/Physical NICs/Switches
Service
Console
Virtual
Machine
Virtual
Machine
VMKernel
VMKernel PortGroup
Virtual NICs
Virtual Switch 0
Virtual Switch 1
Virtual Switch 2
Virtual Switches
Physical NICs
Physical Switches
New Networking Concepts in 3.5
CDP
IPv6
Enhanced VMXNET
Jumbo Frames
TCP Segment Offloading (TSO)
10GigE
NetQueue
InfiniBand (Community Support)
NetFlow (Experimental)
VMKernel Routing table
CDP - Cisco Discovery Protocol
Ease of cooperation between Network
Admin and VI Admin
Exchange of Information about:
which vmnic is connected to which physical
switch
Which Physical switch port is connected to a
specific Virtual Switch
Speed and Duplex setting
CDP - Cisco Discovery Protocol
For ESX Server 3i and ESX Server
3.5, CDP is enabled by default in
listening mode.
Additionally, on ESX Server 3.5, it is
possible to configure CDP also in
advertising mode.
CDP – Configuring ESX 3.5
ESX 3.5 configuration options for CDP:
4 possible choices for virtual Switches configuration:
down – CDP is disabled
listen – (Default and the only option for ESX 3i) the vSwitch will receive
CDP information from physical switch
advertise – the vSwitch will provide information to the physical switch
both – both listen and advertise is enabled
Enabled/disabled only via command line with
esxcfg-vswitch –B <status> vSwitch0
Verify the setting with:
esxcfg-vswitch –b vSwitch0
IPv6
Number of IPv4 addresses is becoming insufficient for modern
needs
IPv6 uses 128 bit addresses, as opposed to 32bit in IPv4, thus
allowing 3.4E+38 possible combinations
With ESX 3.5.0
IPv6 is supported inside the Virtual Machine if the GuestOs is IPv6
capable.
Each application running inside the Guest needs to support IPv6
Other components of Virtual Infrastructure don’t support IPv6 yet
VMware Tools
ESX Service Console
Virtual Center
VMKernel
Enhanced VMXNET
Only available:
In the Add Hardware Wizard
For the following Guests:
Microsoft Windows 2003 Enterprise
32 and 64 Bit
Microsoft Windows 2003 Datacenter
32 and 64 Bit
Red Hat Linux 4
64 Bit
Red Hat Linux 5
32 and 64 bit
SuSE Linux Enterprise 10
32 and 64 bit
Advanced features:
Improved performance
64bit support for VMXNET
Jumbo Frames
TCP Segmentation Offload
Jumbo Frames
Modern switches support Layer 2 frames with MTU
(Maximum Transmit Unit) up to 9000 bytes
Improves network throughput performance
less CPU overhead
fewer CPU interrupts
reduced segmentation
Jumbo Frames in ESX
Support for up to 9000 bytes MTU
Two different areas:
Virtual
Machine
VMKernel
VMkernel
Virtual Machines
Physical Switches need to support
Jumbo Frames end to end
Virtual Switch
Jumbo Frames in ESX for VMkernel
Not supported
for tg3 (some Broadcom NICs) driver
VMKernel
VMKernel iSCSI
VMotion:
Performance increase
Be careful to have consistency across all your
VMKernel ports
Enabled at VMKernel port creation time
with the option “-m 9000” added to the
esxcfg-vmknic CLI command
Not available for 3i
Virtual Switch
Jumbo Frames in ESX for Virtual Machines
Virtual NIC Adapter needs to be “Enhanced
VMXNET”
Virtual
Machine
VMKernel
Virtual Switch needs to be configured to the
required MTU size, for example:
esxcfg-vswitch –m 9000 <vSwitchX>
Available for 3i via RCLI
Virtual Switch
TSO - TCP Segmentation Offload
Segmentation: arbitrary sized packet is divided into MTU sized frames
Offloading: Hardware acceleration
NIC hardware performs the task instead of CPU
Without TSO
Application
TCP
IP
Ethernet
With TSO
TSO - VMKernel
Automatically detected and enabled if the
Virtual
Machine
VMKernel
physical NIC supports it
To verify, check whether the TSO MSS field
is 40960 in the output of the following
command:
esxcfg-vmknic –l
To enable back, re-create the VMKernel port
Virtual Switch
TSO – Virtual Machines
Requires Enhanced VMXNET adapter
Virtual
Machine
VMKernel
Offloads the task to the Virtual
Hardware
Can improve performance of inter-VM
traffic even if there is no TSO enabled
for the physical NIC
TSO with IPv6 not supported
Virtual Switch
TSO – Virtual Machines vs VMKernel
Case scenario: TSO is
Virtual
Machine
VMKernel
enabled in the Virtual Machine
enabled for the Vmkernel
Virtual Switch
Performance increased since:
Virtual Machine will offload to the Virtual
Hardware
VMKernel will offload to the Physical NIC
10GigE
Benefits: Link consolidation
For ex. using VLAN trunking, 2 10 GigE NICs can theoretically replace up
to 20 1Gb NICs
Fewer IRQ Sharing
Less cabling
Increased bandwidth available for VMs
Support:
As of today iSCSI is supported on Intel E1000 and Broadcom based NICs running
on a gigabit link only
NFS is not supported on 10GigE NICs
Supported Hardware (HCL):
IBM 10 GbE Fiber SR Server Adapter (NetQueue, s2iO)
IBM 10 GbE PCIe SR Server Adapter (unm_nic) HW Rev. 25 or newer
Neterion 10GbE NIC (NetQueue, s2iO)
NetXen 10GbE NIC (unm_nic) HW Rev. 25 or newer
NetQueue
Without NetQueue, the load of receiving (RX) 10Gbit/s traffic
is all routed to one single CPU/CORE
With NetQueue, RX Performance is improved by handing off
different RX queues to different CPUs
Disabled by default in ESX 3.5, not available in 3i
Available only on systems running a Neterion s2io driver and
MSI-X
Systems Compatibility Guide provides a list of systems that
support it
http://www.vmware.com/pdf/vi35_systems_guide.pdf
NetQueue
Steps to Enable/Disable NetQueue:
VI3 Client
Under Configuration > Advanced Settings > VMKernel,
Check/Uncheck VMkernel.Boot.netNetqueueEnabled
Service Console
Enable:
esxcfg-module -s "intr_type=2 rx_ring_num=8" s2io
Disable:
esxcfg-module -s "" s2io
Reboot ESX Server to make the changes effective
InfiniBand
High Performance and high scalability
Allows the ESX to integrate into an existing InfiniBand infrastructure,
or to consolidate into one single high performance fabric
Network, via IPoIB
Storage, via SRP (SCSI RDMA Protocol)
Support:
Mellanox Technologies provides the drivers via VMware Community
Source Project, download them from
http://www.mellanox.com/products/ciov_ib_drivers_vi3-1.php
VMware will cover First Level of Support
Not available for 3i
NetFlow
Client/Server framework to collect valuable information about
network users and applications, peak usage times, and traffic
routing.
Multiple purposes, for example:
Network monitoring,
traffic accounting
network planning,
Intrusion detection system
NetFlow enabled vSwitch
Data Collector (Server)
NetFlow
Experimental support in ESX 3.5.0, not available for 3i
Limited set of features, commonly found on physical switches
NetFlow Version 5, with UDP protocol
NetFlow activation should not create stability issues
Can affect overall ESX performance
To enable:
vmkload_mod netflow
/usr/lib/vmware/bin/vmkload_app –i vmktcp /usr/lib/vmware/bin/net-netflow –e
vSwitch0,vSwitch1 <IP address of collector>:<port>
To Stop:
Ctrl-C if the process is running in the foreground
use ps and kill otherwise
vmkload_mod and vmkload_app commands are not supported
For more Information, Enabling NetFlow on Virtual Switches
http://www.vmware.com/pdf/vi3_35_25_netflow.pdf
NetFlow Appliance
http://www.vmware.com/appliances/directory/293
VMKernel Routing table
With ESX 3.5, the VMKernel PortGroup can have a
routing table that includes more than just the default
routing.
Useful for complex routing scenarios
For example VMotion and iSCSI requiring a gateway each
VMKernel Routing table
Command line examples:
Add a route to 192.168.100.0 network through 192.168.0.1
esxcfg-route -a 192.168.100.0/24 192.168.0.1
or
esxcfg-route -a 192.168.100.0 255.255.255.0 192.168.0.1
Set the VMkernel default gateway of 192.168.0.1
esxcfg-route 192.168.0.1
or
esxcfg-route -a default 192.168.0.1
Delete a 192.168.100.0 route from the VMkernel:
esxcfg-route -d 192.168.100.0/24 192.168.0.1
Advanced Troubleshooting
Common issues, troubleshooting hints and
recommendations:
PortGroups
Nic-Teaming
NIC Failover and Failback
VLAN
Duplicated IP address
Domain Name System (DNS)
Multihomed ESX
Virtual Center and Network Address Translation (NAT)
PortGroups!
Within a Virtual Switch, PortGroups are an easy way to group logically similar
VMs…
…but PortGroups don’t implement segmentation within a virtual Switch (unless
VST is implemented).
PortGroups can override virtual Switch settings
Security
NIC Teaming
Traffic Shaping
Make sure that the override makes sense ! For Example:
Out-PortID in the vSwitch and Out-IP in the PortGroup
Beaconing in the vSwitch and Out-IP in the PortGroup
PortGroup “Production” overrides
security setting inherited from the switch
NicTeaming Policies
1
A
2
B
3
4
5
6
7
8
9
10 11 12
C
Teaming Policies:
Out-PortID Route based on the originating virtual port ID (no special configuration on
physical switch)
Out-MAC Route based on source MAC hash (no special configuration on physical
switch)
Out-IP Route based on ip hash (etherchannel / 802.3ad configured on physical switch
side, only one physical switch for the team )
Port-Id / Mac Hash Policies
Difference: Mac-Hash won’t change the uplink
used for a VM when the VM is rebooted or replugged, PortID-Hash will
Both can be used with a NIC Team that spans
across more than one physical switch in the same
broadcast domain
Both policies require the physical switch to always
use the same port to speak to the same Mac
Address (no link aggregation configured).
The maximum bandwidth available for each VM is
still limited to the bandwidth a single uplink can
provide
IP Hash Policy
Will decide which uplink to use depending
on a hash of source and destination IP
address
Can provide a VM with a theoretical
maximum bandwidth of the sum of each
uplink bandwidth
Physical switch has to be configured to be
etherchannel/link aggregation/802.3ad
aware, but no LACP or any other protocol
negotiation should be enabled
Only one physical switch can be used
Broadcast domain has to be the same
Failover and Failback
Rolling Failover -> Yes in ESX 3.0
Failback -> No in ESX 3.5
Standard failover policy is to fail back to the active NIC as
soon as its link is detected to be “up”
Rolling Failover will instead keep using the current link until it
fails
Benefit:
STP can cause a condition where the link on the previous uplink is
detected to be up again, but the port Is still in
blocking/learning/listening status, leading the failback to cause
packet loss
“flapping” nics
Example case scenario:
Service Console PortGroup
HA
VMKernel PortGroup
Switch Rebooting
Fail Back
Failover
Standby
iSCSI/NAS
STP enables Uplink
Active
VLAN misconfiguration
Symptoms:
Ping does not work
ARP resolution fails, for example:
1
2
3
0.000000 Vmware_93:79:fa -> Broadcast
1.000084 Vmware_93:79:fa -> Broadcast
1.999931 Vmware_93:79:fa -> Broadcast
ARP Who has 10.16.158.254?
ARP Who has 10.16.158.254?
ARP Who has 10.16.158.254?
network captures show other ranges of IPs on the VM
Network Hint might contain an indication of the problem
Troubleshooting techniques:
Collect a trace from inside the VM
Enable promiscuous mode
Collect a Trace from the Virtual Switch
Enable promiscuous mode
Use another VM or a Service Console interface
If you are using VST, create and use an additional PortGroup with VLAN 4095 (ALL)
In the trace:
check whether there is traffic in a network range you would not expect
If you are using a PortGroup in VLAN 4095, check for the presence of
packets tagged in a different VLAN ID
packets related to the VM in question with no tag at all
Tell 10.16.158.178
Tell 10.16.158.178
Tell 10.16.158.178
Duplicated IP addresses
Symptoms
Network connectivity works intermittently
Detecting
Duplicated ARP replies, for example:
No.
Time
Source
Destination
Prot.
Info
10278
869.8113
HewlettP_c9:a9:ee
Broadcast
ARP
Who has 10.250.206.2?
(00:19:bb:c9:a9:ee)
(ff:ff:ff:ff:ff:ff)
Vmware_44:3f:3c
HewlettP_c9:a9:ee
(00:50:56:44:3f:3c)
(00:19:bb:c9:a9:ee)
HewlettP_50:55:be
HewlettP_c9:a9:ee
(00:19:bb:50:55:be)
(00:19:bb:c9:a9:ee)
10279
10280
869.8114
869.8117
Tell
10.250.206.3
ARP
10.250.206.2 is at
00:50:56:44:3f:3c
ARP
10.250.206.2 is at
00:19:bb:50:55:be
arp cache shows different MAC addresses for the same IP over time
DNS – Domain Name System
Main cause for misconfigurations/malfunctions for:
HA (beware also of “vmware” hostname)
VC
VCB
Required forward and reverse lookup, either for FQDN and short
name (therefore /etc/resolv.conf should specify also the “search”
option)
Alternative (not recommended) solution: /etc/hosts
Does not scale up. DNS was developed to address /etc/hosts limitations back
when internet counted tens of hosts
error prone (errors you might realise after months from the change!)
Primary and Secondary DNS recommended for HA
ESX Multihomed / Source Routing
Multihomed/Source Routing
Routin
g
Up to ESX 3.0, the VMKernel had one only
gateway, so if there were two vmkernel
nics (for example iSCSI and VMotion), one
of the two wouldn’t be able to access hosts
outside of it’s netmask range. You can
resolve this issue with “esxcfg-route” in
ESX 3.5
X
Route:
Both with the VMKernel and the Service
Console interface, if you have two network
interfaces in the same network range, the
kernel will chose the first route that
matches the destination IP, and the source
IP address used to respond will be the
same of the chosen NIC.
Network 10.0.0.0/24 dev nic1 src 10.0.0.1
Network 10.0.0.0/24 dev nic0 src 10.0.0.32
???
nic1
nic0
VirtualCenter and ESX behind N.A.T.
NAT
NAT
Typical scenario: VI Management performed as a Service
Communication between Virtual Center and ESX servers is two ways
If There is a Network Address Translation between Virtual Center and ESX, the ESX will attempt to reach
Virtual Center via the IP address Virtual Center “believes” to have.
Starting from VirtualCenter 2.0.2, it is possible to specify what IP address to use
http://www.vmware.com/support/vi3/doc/releasenotes_vc202.html#resolvedissues
Modify the vpxd.cfg file (usually in C:\Documents and Settings\All Users\Application
Data\VMware\VMware VirtualCenter\) so that it contains the following line within <vpxd> tag:
<managedip>ipAddress</managedip>
Before VirtualCenter 2.0.2, or with a mixed environment (some ESX on LAN and some behind the NAT)
Double way N.A.T. required, in order to present VirtualCenter to the remote ESX via the same IP address
Questions
Summary - 1
New Features in ESX 3.5
CDP
IPv6 support
Enhanced VMXNET, for selected GuestOS, first supported VMXNET adapter
for 64Bit GuestOS
Jumbo Frames, Requires Enhanced VMXNET virtual Adapter
TCP Segment Offloading (TSO), Requires Enhanced VMXNET virtual
Adapter
10GigE, for selected NICs
NetQueue for improved performance with 10GigE NICs
InfiniBand, by Mellanox Technologies
NetFlow, Experimental support
Advanced VMKernel Routing table, via esxcfg-route
Summary - 2
Advanced Troubleshooting
PortGroups group logically related VMs, and can override vSwitch
settings to adjust to custom VM needs
Nic-Teaming, vSwitch and Physical Switch configuration caveats
NIC Failover and Failback, Use Link state Tracking or Rolling Failover
How to troubleshoot misconfigured VLANs and duplicated IP addresses
Domain Name System (DNS)
Caveats with ESX and VMKernel having multiple IPs
Caveats with Virtual Center and Network Address Translation (NAT)