Transcript Neutron hybrid mode

NEUTRON HYBRID MODE
Vinay Bannai
SDN Architect, Nov 8 2013
ABOUT PAYPAL
PayPal offers flexible and innovative
payment solutions for consumers and
merchants of all sizes.
• 137 Million Active Users
• $300,000 Payments processed by PayPal
each minute
• 193 markets / 26 currencies
• PayPal is the World’s Most Widely Used
Digital Wallet
2
Confidential and Proprietary
INTRODUCTION
• Data Center Architecture
• Neutron Basics
• Overlays vs Physical Networks
• Use Cases
• Problem Definition
• Hybrid Solution
• Performance Data
• Analysis
• Q&A
3
Confidential and Proprietary
DATA CENTER ARCHITECTURE
Internet
Core
Data Center
Layer-3 router
Bisection BW
Aggregation
Layer-3 switch
Bisection BW
Access
Layer-3 switch
Bisection BW
Racks
4
Confidential and Proprietary
NEW DATACENTER ARCHITECTURE
Internet
Data Center
Layer-3 router
Core
Bisection BW
Aggregation
Layer-3 switch
Bisection BW
Layer-3 switch
Access
Bisection BW
Edge
Layer
VM
5
vswitches
VM
Confidential and Proprietary
VM
VM
VM
VM
VM
VM
VM
VM
DATACENTER WITH VSWITCHES
Data Center
Layer-3 switch
Access
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
Racks
6
Confidential and Proprietary
Racks
Racks
NEUTRON BASICS
7
Confidential and Proprietary
OVERLAY NETWORKS
• Overlays provide connectivity between VMs and Network
Devices using tunnels
• The physical core network does not need to be re-provisioned
constantly
• The tunneling encap/decap is done at the edge in the virtual
switch
• Decouples the tenant network address from the physical Data
Center network address
• Easy to support overlapping address
• Tunneling techniques in vogue
− VXLAN
− STT
− NVGRE
8
Confidential and Proprietary
PHYSICAL NETWORKS
• Physical Networks connect VM’s and Network Devices using
provider network
• VM’s are first class citizens with the hypervisor and the
networking devices
• No tunneling protocols used
• Tenant separation is achieved by using VLANs or IP
subnetting
• Hard to achieve overlapping address spaces
• Underlying network needs to be provisioned with VLANs
9
Confidential and Proprietary
PHYSICAL VS OVERLAY
Tenant on
Physical
Network
VM
VM
VM
VM
VM
VM
VM
VM
VM
L2
L3
L2
L2
Network Virtualization Layer
VM
Tenant on
Overlay
Network
PROS & CONS
11
Function
Hypervisor
Bridged VMs
(VLAN)
Tunneled
VM’s
Throughput
Best
Better
Worse
Latency
Best
Better
Worse
Flexibility
Worse
Better
Best
Overlapping IP
addresses
Worse
Worse
Best
Operational
Dependency
Worse
Better
Best
Confidential and Proprietary
USE CASES
• Production Environment
− Production website across multiple data centers
− Low latency and high throughput
− Bridged Mode
• Mergers & Acquisitions Private Community Cloud
− Private Community Cloud
− Needs address isolation and overlapping
− Address isolation, Flexibility, low latency and high throughput
− Overlay Mode
• Development & QA Environment
− Production development, QA & Staging
− Flexibility, high throughput but can tolerate higher latency
− Bridged and Overlay Mode
12
Confidential and Proprietary
PROBLEM STATEMENT
• Support flexibility, low latency, high throughput and
overlapping address space all at the same time
• Support both bridged and overlay networks
• VM’s on a hypervisor should be able to choose networks
• Need a consistent deployment pattern
• Configurable by automation tools (puppet, chef, salt etc)
13
Confidential and Proprietary
TYPICAL VSWITCH
VM
Ta
VM
Tb
VM
Tc
 HYBRID VSWITCH
VLAN 200
Hypervisor
br-int
Bridged
Traffic
Overlay
Traffic
br-tun
br-bond
Bond Intf
Prod
Interface
Mgmt
Interface
14
Confidential and Proprietary
IP
Interface
CONFIGURATION OF HYBRID MODE
• Create the neutron networks
− Flat Network
− neutron net-create bridged-flat --provider:network_type=flat --provider:
physical_network=<Physnet>
− neutron subnet-create --allocation-pool start=10.x.x.100, end=10.x.x.200
bridged-flat --gateway 10.x.x.1 10.0.0.0/23 --name bridged-flat-subnet -enable_dhcp=False
− VLAN Network
− neutron net-create bridged-vlan --provider:network_type=vlan --provider:
physical_network=<Physnet> --provider:segmentation_id=<vlan-id>
− neutron subnet-create --allocation-pool start=10.x.x.100, end=10.x.x.200
bridged-vlan 10.x.x.1 10.0.0.0/23 --name bridged-vlan-subnet
15
Confidential and Proprietary
CONTD.
• Neutron networks (contd.)
− Overlay Network
− neutron net-create overylay-net
− neutron subnet-create --allocation-pool start=10.x.x.100, end=10.x.x.200
overlay-net --gateway 10.x.x.1 10.0.0.0/23 --name overlay-net-subnet
• On the compute node
− Configure the bond
− ovs-vsctl add-br br-bond0
− Configure the OVS
− ovs-vsctl br-set-external-id br-bond0 bridgeid br-bond0
− ovs-vsctl set Bridge br-bond0 fail-mode=standalone
− ovs-vsctl add-port br-bond0 eth0 eth1
16
Confidential and Proprietary
PERFORMANCE DATA
• To measure latency and throughput, we ran following tests
• Within a rack (L2 switching)
− Bare metal to Bare metal
− Bridged VM to Bridged VM
− Tunneled VM to Tunneled VM
• Across racks (L3 switching)
− Bare metal to Bare metal
− Bridged VM to Bridged VM
− tunneled VM to tunneled VM
• Across the Network Gateway
− Bare metal to Bare metal (outside the cloud)
− Bridged VM to Bare metal (outside the cloud)
17
− tunneled VM to Bare metal (outside the cloud)
Confidential and Proprietary
HYPERVISOR, VM AND OS DETAILS
• Compute Hypervisors
− 2 sockets, 16 cores/socket SandyBridge @ 2.6GHz (32 Hyper Threaded)
− 2 x 10G ports (Intel PCIe)
− RAM : 256GB
− Disk: 4 x 600GB in RAID-10
− RHEL 6.4 running OVS
• VM
− vCPUs: 2
− RAM: 8GB
− Disk: 20GB
− RHEL 6.4
18
Confidential and Proprietary
TEST SETUP
Half rack with
Two Fault Zones
X.X.X.X/23
19
L3 Gateways
For Overlays
Y.Y.Y.Y/23
Confidential and Proprietary
X.X.X.X/23
Y.Y.Y.Y/23
X.X.X.X/23
Y.Y.Y.Y/23
TESTING METHODOLOGY
• Tunneling VM uses STT (OVS)
• Bridged VM uses Flat Network (OVS)
• Used nttcp 1.47 for throughput
• Bi-directional TCP with varying buffer size
• Buffer size in bytes : [64,… 65536]
• MTU size : 1500 Bytes (on both bare metal and VM’s)
• Used ping for latency measurement (60 samples)
• Used python scripts and paramiko to run the tests
• Tests done with other traffic (Dev/QA)
− Around 470+ active VM’s
− Around 100 Hypervisors
− Multiple half racks
20
Confidential and Proprietary
TEST SETUP FOR SAME RACK
21
Confidential and Proprietary
WITHIN A RACK (L2 SWITCHING)
THROUGHPUT
22
Confidential and Proprietary
WITHIN A RACK (L2 SWITCHING)
PING LATENCY
23
Confidential and Proprietary
ANALYSIS
• Observations
• Results for buffer size < MTU size
− Tunneled VM’s tend to have best overall throughput
− Bridged VM’s tend to better than bare metal
− OVS and tunnel optimizations at play
• Results for buffer size > MTU size
− Tunneled VM’s and bare metal performance about the same
− Bridged VM’s bests both bare-metal and tunneled VMs (??)
• OVS and tunnel optimizations apply for buffer sizes smaller
than MTU
• OVS optimization apply for buffer sizes greater than MTU
• Tunneled and Bridged VM’s have a slightly higher latency
than bare metal
24
Confidential and Proprietary
TEST SETUP ACROSS RACKS
25
Confidential and Proprietary
ACROSS RACKS (L3 SWITCHING)
THROUGHPUT
26
Confidential and Proprietary
ACROSS R3ACKS (L SWITCHING)
PING LATENCY
27
Confidential and Proprietary
ANALYSIS
• No bridged VM’s in the tests (setup problem)
• Results for buffer size < MTU size
− tunneled VM’s tend to have best overall throughput
− OVS and tunnel optimizations at play
• Results for buffer size > MTU size
− tunneled VM’s and bare metal performance about the same
• OVS and tunnel optimizations apply for buffer sizes smaller
than MTU
• tunneled and Bridged VM’s have a slightly higher latency than
bare metal
28
Confidential and Proprietary
TEST SETUP ACROSS L3 GATEWAY
29
Confidential and Proprietary
ACROSS NETWORK GATEWAY
THROUGHPUT
30
Confidential and Proprietary
ACROSS NETWORK GATEWAY
PING LATENCY
31
Confidential and Proprietary
ANALYSIS
• tunneled VM’s tend to have similar if not better throughput as
bare metal or bridged VM
• tunneled VM’s have a slightly higher latency
• Bridged VM’s tend to have same overall throughput as the
hypervisor
• Bridged VM’s tend to have same latency as the hypervisor
• Latency from a tunneled VM across L3 gateway is higher than
Physical VMs due to extra hops, but need to re-run the tests
32
Confidential and Proprietary
CONCLUSION & FUTURE WORK
• Understand your network requirements
− Latency, bandwidth throughput, flexibility
• Overlay Vs Physical
• Hybrid Mode
• Performance Analysis
• Make your deployment patterns simple and repeatable
• Future work
− Additional performance tests
− VXLAN, NVGRE
− Varying MTU size
− Setup without background traffic
• Let me know if you are interested to collaborate
33
Confidential and Proprietary
THANK YOU
[email protected]