Transcript SR-IOV HandsOn_v1.0x

SR-IOV Hands-on
Lab
Rahul Shah
Clayne Robison
NFV Models – DPDK SRIOV usage
Intel® Ethernet XL710 Family
(I/O Virtualization)
Hands-On
Session (SRIOV)
SR-IOV Hands-on: System
setup
Compute Node—Phase 1
Compute Node—Phase 2
VM with SR-IOV Virtual Function Passthrough
VM with Physical Function Passthrough
DPDK
testpmd
DPDK
pktgen
Pass-through
Port
VF0
PF0
I40e PFO
I40e PF0
Port 0
(SRIOVON)
VF1
2x10G
Host OS
loopback
DPDK
testpmd
Port 1
(SRIOV-ON)
DPDK
pktgen
Pass-through
Port
pci-stub/
vfio
Port 0
(SRIOV-OFF)
2x10G
Host OS
loopback
PF1
pci-stub/
vfio
Port 1
(SRIOV-OFF)
SR-IOV-Lab: Connect to Virtual
Machine
SSH from your laptop1 in to Cluster Jump Server2
•
•
•
•
•
1. Your
Laptop
IP Address: 207.108.8.161
SSH v2 preferred
Username: student<1-50> (e.g. [email protected])
Password: same as username (e.g. student9)
Repeat so that you have multiple connections to the Jump Server2
2.
Cluster
Jump
Server
SSH from Cluster Jump Server2 in to assigned HostVM3
• $ssh HostVM-____
• Username: user; Password: user
Only Even
Numbered HostVMs
can do the SR-IOV
lab.
3. HostVM
Note: You need two 2
ssh sessions into the
jump server.
6
Prepare Compute Node for I/O
Pass-through
IOMMU support is required for VF to function properly when assigned to VM. The following boot parameter is
required to enable IOMMU support for Linux kernels
1. (Done) Before booting compute node OS, enable Intel VT features in BIOS
2. (Done) Append “intel_iommu=on” to the GRUB_CMDLINE_LINUX entry in /etc/default/grub
3. (Done) Update the compute node grub configuration using grub2-mkconfig command
$sudo grub2-mkconfig –o /boot/grub2/grub.cfg
3. (Done) Reboot the compute node for the iommu change to take effect
4. Check the compute node kernel command line in Linux and look for “iommu=on”
$cat /proc/cmdline.
Create Virtual Functions
Linux does not create VFs by default. The X710 server adapter supports upto 32 VFs per port. The
XL710 server adapters supports up to 64 VFs per port.
1. On the compute node, create the Virtual Functions:
# echo 4 > /sys/class/net/[INTERFACE NAME]/device/sriov_numvfs (for
kernel versions 3.8.x and above)
(For Kernel versions 3.7.x and below, to get 4 VFs per port) #modprobe i40e
max_vfs=4, 4
2. On the compute node, verify that the Virtual Functions were created:
# lspci | grep ‘X710 Virtual Function’
3. On the compute node, bring up the link on the virtual functions
# ip l set dev [INTERFACE NAME] up
4. You can assign a MAC address to each VF on the compute node.
# ip l set dev enp6s0f0 vf 0 mac aa:bb:cc:dd:ee:00
Upon successful VF creation, the Linux operating system automatically loads the i40 vf driver.
Prepare Hypervisor –
KVM/libvirt Method
To simplify integration with VMs, SR-IOV Virtual Functions can be deployed as a pool of NICs in a libvirt network.
1. Compute node: Create an XML fragment/file that describes an SR-IOV network:
<network>
<name>sr-iov-enp6s0f0</name>
<forward mode='hostdev' managed='yes'>
<pf dev=’[INTERFACE NAME]'/>
</forward>
</network>
2. Compute node: Use virsh to create a network based on this XML fragment
# virsh net-define <sr-iov-network-description.xml>
3. Compute node: Activate the network.
# virsh net-start sr-iov-enp6s0f0
Prepare and Launch the VM
image – KVM/libvirt Method
On the compute node, once you have a libvirt network based on the SR-IOV virtual functions, Add a NIC from that network
1. Create an XML fragment/file that describes the NIC, and optionally add a MAC address
<interface type='network'>
<mac address=’aa:bb:cc:dd:ee:ff'/>
<source network='sr-iov-enp6s0f2'/>
</interface>
2. Use #virsh edit [VM NAME] to insert the XML fragment into the VM Domain definition
<controller type='virtio-serial' index='0'>
<address …/>
</controller>
<interface type='network'>
<mac address=’aa:bb:cc:dd:ee:ff’/>
<source network='sr-iov-enp6s0f2'/>
</interface>
3. Launch the VM
#virsh start [VM NAME]
Prepare the Hypervisor--QEMu
Method
Using qemu directly is not as elegant, but it works as well
1. Get the PCI Domain:Bus:Slot.Function information for the VF
# lshw –c network -businfo
2. Load the pci-stub driver if necessary
# modprobe pci-stub
3. Unbind the NIC PCI device from the i40e driverFollow the below steps to passthrough each VF
port in VM
# echo "8086 154c" > /sys/bus/pci/drivers/pci-stub/new_id
# echo [PCI Domain:Bus:Slot.Function] > /sys/bus/pci/devices/[PCI
Domain:Bus:Slot.Function]/driver/unbind
# echo [PCI Domain:Bus:Slot.Function] > /sys/bus/pci/drivers/pcistub/bind
Launch the image – Qemu
Method
1. Start the virtual machine by running the following command
# qemu-system-x86_64 –enable-kvm
\
–smp 4 –cpu host –m 4096 –boot c
–hda [your image]
-nographic –no-reboot
\
\
\
-device pci-assign, host=[VF PCI Bus:Slot.Function]
\
Install DPDK on the SR-IOV VF
on the Virtual Machine
Now that the compute node has been set up, get the virtual machine ready
1. From the Jump Server, ssh into your assigned Virtual Machine ($ssh HostVM-____)
2. Scripts for the lab are located in /home/user/training/sr-iov-lab.
3. View the Virtual Functions that have already been loaded into the Virtual Machine.
$ ./00_show_net_info.sh
1. Steps 01-03 are only necessary if the DPDK lab was done before the SR-IOV lab
2. Compile DPDK and load it onto the Virtual Functions
$ ./04_build_load_dpdk_on_vf.sh
1. Write down the MAC addresses that were displayed in the previous step
TESTPMD_MAC=___:___:___:___:___:___
PKTGEN_MAC =___:___:___:___:___:___
Build and Run Testpmd and
PKTGEN
1. On the virtual machine, build and run testpmd. Look at the parameters in the build script. Running testpmd requires
that you know the MAC address of the port on which pktgen is going to run. This was output to the console in step 04.
# 05_build_start_testpmd_on_vf.sh [PKTGEN MAC]
1. Look at the command line to see what parameters we are using.
2. Open another SSH session into your assigned virtual machine (HostVM-____)
3. Build and launch pktgen.
# 06_build_start_pktgen_on_vf.sh
4. You need to know the MAC address of the port where pktgen is going to send packets, which is the port on which
testpmd is waiting. You can find the testpmd port when you launch testpmd. You’ll see lines that look like this:
Configuring Port 0 (socket 0)
Port 0: 52:54:WW:XX:YY:ZZ (This is the testpmd MAC address)
Checking link statuses...Port 0 Link Up - speed 10000 Mbps - full-duplex
Note: You can also get the testpmd MAC address from step 04
5. Allow CRC stripping. In a VM, using a VF, we can’t disable CRC stripping. Edit pktgen-port-cfg.c and change line 94 to
“.hw_strip_crc
= 1,”
#vi /usr/src/pktgen-3.0.14/pktgen-port-cfg.c
Note: testpmd also has this problem, but we take care of it on the command line: --crc-strip
Generate and measure Traffic
Flow with SR-IOV
Now that we have pktgen and testpmd launched, start the traffic
1. In pktgen, set the mac 0 address to point to the testpmd SR-IOV port
> set mac 0 [TESTPMD MAC]
2. Start generating traffic
> start 0
3. View stats in testpmd
> show port stats 0
4. Record the RX and TX info to compare with Physical PCI passthrough
Mbit/s RX:___ TX:___
PPS RX:___ TX:___
Prepare for Physical PCI
Passthrough
1. Close pktgen
> quit
2. Close testpmd
> quit
3. Unload all drivers from the SR-IOV virtual functions
# 07_unload_all_x710_drivers.sh
1. Watch the PCI pass-through NICs appear
$ 08_wait_for_pf.sh
Build and Run Testpmd and
PKTGEN on Physical Functions
1. Build and load DPDK on the Physical Function NICs. Note the MAC addresses that are displayed at the end
#09_build_load_dpdk_on_pf.sh
2. Build and run testpmd on the PF. Running testpmd requires that you know the MAC address of the port on
which pktgen is going to run. This was output to the console in step 09. Note that they are different addresses
than the SR-IOV VF. Why?
# 10_build_start_testpmd_on_pf.sh [PKTGEN MAC]
TESTPMD_MAC=___:___:___:___:___:___
PKTGEN_MAC =___:___:___:___:___:___
3. Build and launch pktgen in your other ssh session.
# 11_build_start_pktgen_on_pf.sh
4. In pktgen, set the mac 0 address to point to the testpmd SR-IOV port
> set mac 0 [TESTPMD MAC]
5. Start generating traffic
> start 0
6. View stats in testpmd
> show port stats 0
Mbit/s RX:___ TX:___
PPS RX:___ TX:___
7. Record the RX and TX info. How does it compare with SR-IOV Virtual Function passthrough?
8. Unload all XL710 drivers: # 12_unload_all_x710_drivers.sh
Backup
information
•
•
•
•
SRIOV Pool (VF) x Queue
PF-VF Mailbox Interface Initialization
Mailbox Message support – DPDK IXGBE PMD vs Linux ixgbe Driver
SRIOV L2 Filters and Offloads
Intel® X520 (10G) vs XL710
(40G) SRIOV Internals
Intel® Ethernet XL710 Filters
•
Filters available to select
PF/VF for packet delivery
within a LAN port based on
Ether Type, MAC/VLAN and
S/E-Tag
•
Filters available to select a
VSI for packet delivery within
a PF/VF (PCI function)
•
Filters available to select an
Rx/Tx Queue for packet
delivery within a VSI (Pool of
Queues)
•
Explicit flow configuration
using Flow Director for packet
steering to a VSI/RxQueue
Reference
•
•
•
•
•
•
•
Intel® Network Builders University - https://networkbuilders.intel.com/university
Basic Training – Course 3: NFV Technologies
Basic Training – Course 5: The Road to Network Virtualization
Basic Training – Course 6: Single Root I/O Virtualization
DPDK – Course 1-5: DPDK API and PMDs
Data Plane Development Kit – dpdk.org
Intel Open Source Packet Processing - https://01.org/packet-processing
• Intel® Resource Director Technology (Intel® RDT)
• Intel® QAT Drivers
Questions..!?
Sessions coming up…
1. DPDK Performance Analysis with VTune Amplifier
2. DPDK Performance Benchmarking
3. DPDK Hashing Algo. Support
Creating VF – using DPDK pf
driver - Backup
1. To use the port for DPDK, you need to bind it to igb_uio driver. You can check all the ports using following
# ./tools/dpdk_nic_bind.py --status
2. To bind the ports to igb_uio driver
# ./dpdk_nic_bind.py –b igb_uio 06:00.0
# echo 4 > /sys/bus/pci/devices/0000\:bb\:dd.ff/max_vfs
4. You can test the VFs created using the same lspci command before.
DPDK PF driver – HoST CP/DP
- Backup
1. Run the DPDK PF driver in the host using testpmd application on a single PF port
# ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xF -n 4 -- -i --portmask=0x1 --nb-cores=2
<testpmd> set fwd mac
<testpmd> start
<testpmd> show port stats 0 – [optional]