Transcript ppt - UiO

INF5062:
Programming Asymmetric Multi-Core Processors
IXP:
The Bump in the Wire
21 March 2017
Background and Motivation:
Network traffic increase
Software-Based Network System
 Uses conventional, shared hardware (e.g., a PC)
 Software
− runs the entire system
− allocates memory
− controls I/O devices
− performs all protocol processing
 First generation
network systems:
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
Review of General Data Path on
Conventional Computer Hardware Architectures
sending:
receiving:
forwarding:
application
application
application
communication
system
communication
system
communication
system
user space
kernel space
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
Question:
 Which is growing faster?
− network bandwidth
− processing power
 Note: if network bandwidth is growing faster
− CPU may be the bottleneck
− need special-purpose hardware
 Note: if processing power is growing faster
− no problems with processing
− network/busses will be bottlenecks
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
Growth Of Technologies
Mbps
Engineering rule:
1GHz general purpose CPU = 1Gbps network data rate
Thus, software running on a general-purpose processor is
insufficient to handle high-speed networks because the
aggregate packet rate exceeds the capabilities of the CPU
year
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
Network Processors: The Idea in a Nutshell
 Many designs through many generations
(varying amount of HW & SW)
 Include support for protocol processing and I/O on one chip
− General-purpose processor(s) for control tasks
− Special-purpose processor(s) for packet processing and table lookup
− Include functional units for tasks such as checksum computation,
hashing, …
 Call the result a network processor
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
Network Processors: Main Idea
Traditional system:
- slow
- resource demanding
- shared with other operations
Network processors:
- a computer within the computer
- special, programmable hardware
- offloads host resources
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
Explosion of Commercial Products
 1990  2000: network processors transformed from
interesting curiosity to mainstream product
− reduction in both overall costs and time to market
− 2002: over 30 vendors with a vide range of architectures
− e.g.,
•
•
•
•
•
•
•
•
•
•
Multi-Chip Pipeline (Agere)
Augmented RISC Processor (Alchemy)
Embedded Processor Plus Coprocessors (Applied Micro Circuit Corporation)
Pipeline of Homogeneous Processors (Cisco)
Pipeline of Heterogeneous Processors (EZchip)
Configurable Instruction Set Processors (Cognigine)
Extensive And Diverse Processors (IBM)
Flexible RISC Plus Coprocessors (Motorola)
Internet Exchange Processor (Intel)
…
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
Intel IXP1200 / 2400:
A Short Overview
IXA: Internet Exchange Architecture
 IXA is a broad term to
describe the Intel network
architecture (HW & SW,
control- & data plane)
 IXP: Internet Exchange
Processor
− processor that implements IXA
− IXP1200 is the first IXP chip
(4 versions)
− IXP2xxx has now replaced the
first version
University of Oslo
 IXP1200 basic features
−
−
−
−
−
−
−
1 embedded 232 MHz StrongARM
6 packet 232 MHz µengines
onboard memory
4 x 100 Mbps Ethernet ports
multiple, independent busses
low-speed serial interface
interfaces for external memory
and I/O busses
−…
 IXP2400 basic features
−
−
−
−
1 embedded 600 MHz XScale
8 packet 600 MHz µengines
3 x 1 Gbps Ethernet ports
…
INF5062, Carsten Griwodz & Pål Halvorsen
IXP1200 Architecture
PCI bus:
- allow IXP to connect to I/O devices
- enable use of host CPU
- rate 2.2 Gbps
SRAM bus:
- shared bus (several external units)
- usually control rather than data
- rate 3.71 Gbps
Serial line:
- connects to the RISC
- intended for control and management
- rate 38 Kbps
SDRAM bus:
- provide access to external SDRAM memory
used to store packets
- can also pass addresses, control/store operations, etc.
- rate 7.42 Gbps
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
IX (Intel eXchange) bus:
- enable higher rates compared to PCI
- form fast path (IXP and high-speed interfaces)
- interface to other IXP cards
- 4.4 Gbps
IXP1200 Architecture
RISC processor:
- StrongARM running Linux
- control, higher layer protocols and exceptions
- 232 MHz
Access units:
- coordinate access to external units
Scratchpad:
- on-chip memory
- used for IPC and synchronization
Microengines:
- low-level devices with limited set of instructions
- transfers between memory devices
- packet processing
- 232 MHz
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
IXP1200 Processor Hierarchy
General-Purpose Processor:
- used for control and management
- running general applications
I/O processors (microengines):
- transfers between memory devices
- packet processing
University of Oslo
RISC processor:
- chip configuration interface (serial line)
- control, higher layer protocols and exceptions
Coprocessors:
- real-time clock and timers
- IX bus controller
- hashing unit
- ...
INF5062, Carsten Griwodz & Pål Halvorsen
Physical interface processors:
- implement layer 1 & 2 processing
IXP1200 Memory Hierarchy
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
IXP1200 Memory Hierarchy
Different memory types…
…are organized into different addressable data units (words or longwords)
…have different access times
…connected to different busses
Therefore, to achieve optimal performance, programmers must understand the
organization and allocate items from the appropriate type
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
IXP1200  IXP2400
IXP1200
PCI bus
SRAM
bus
SRAM
access
SRAM
FLASH
SCRATCH
memory
MEMORY
MAPPED
I/O
PCI
access
multiple
independent
internal
buses
Embedded
RISK CPU
(StrongARM)
microengine 1
microengine 2
microengine 3
microengine 4
microengine 5
SDRAM
access
DRAM
IX
access
DRAM
bus
IX bus
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
microengine 6
IXP2400 Architecture
Coprocessors
- hash unit
- 4 timers
SRAM
- general purpose I/O pins
bus
- external JTAG connections (in-circuit tests)
- several bulk cyphers (IXP2850 only)
SRAM
- checksum (IXP2850 only)
-…
PCI bus
IXP2400
RISC processor:
- StrongArm  XScale
- 233 MHz  600 MHz
SRAM
access
coprocessor
SCRATCH
memory
SlowportFLASH
- shared inteface to external units
- used for FlashRom during bootstrap
slowport
access
SDRAM
access
DRAM
PCI
access
Embedded
RISK CPU
(XScale)
multiple
independent
internal
Mediabuses
Switch Fabric
microengine 1
microengine 2
microengine 3
microengine 4
- forms fast path for transfers
Microengines
- interconnect for severalmicroengine
IXP2xxx
-5
68
MSF
access
…
microengine 8
DRAM
bus
Receive/transmit buses
- shared bus  separate busses
University of Oslo
receive bus
INF5062, Carsten Griwodz & Pål Halvorsen
- 233 MHz  600 MHz
transmit bus
IXP2400 Architecture
 Memory
− generally more of everything
− generally larger gap between CPUs and memory access in
terms of cycles
− local memory on each microengine
•
•
•
•
•
saving temporary results
private per packet processor
small (2560 bytes)
low latency (one cycle)
accessed through special registers
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
IXP2400 Basic Packet Processing
PCI bus
SRAM
bus
SRAM
access
SRAM
coprocessor
SCRATCH
memory
FLASH
slowport
access
SDRAM
access
DRAM
PCI
access
multiple
independent
internal
buses
Embedded
RISK CPU
(XScale)
microengine 1
microengine 2
microengine 3
microengine 4
microengine 5
MSF
access
…
microengine 8
DRAM
bus
receive bus
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
transmit bus
Using IXP2400
Intel IXP2400 Hardware Reference Manual
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
Programming Model
head
tail
logical mapping
linked list
metadata
queue
Scratch rings
data buffers
core
component
XScale
microengines
input
ports
RX
microblock
University of Oslo
process
microblock
INF5062, Carsten Griwodz & Pål Halvorsen
TX
microblock
output
ports
Programming Model
Threads
core
component
XScale
microengines
input
ports
RX
microblock
process
microblock
Hardware contexts
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
TX
microblock
output
ports
Framework
 uclo
− Microengine loader
− Necessary to load your microengine code into the microengines at runtime
 hal
− Hardware abstraction layer
− Mapping of physical memory into XScale processes’ virtual address space
− Functions starting with hal
 ossl
− Operating system service layer
− Limited abstraction from hardware specifics
− Functions starting with ix_
 rm
− Resource manager
− Layered on top of uclo and ossl
− Memory and resource management
• all memory types and their features
• IPC, counters, hash
− Functions starting with ix_
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
Bump in the Wire
Bump in the Wire
Count web packets, count ICMP packets
Internet
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
Packet Headers and
Encapsulation
Ethernet
48 bit address configured to an
interface on the NIC on the receiver
0
1
2
3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
Destination Address
|
+
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+
|
Source address
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
Frame type
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
describes content of ethernet frame,
e.g., 0x0800 indicates an IP datagram,
0x0806 indicates an ARP packet
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
48 bit address configured to an
interface on the NIC on the sender
Internet Protocol version 4 (IPv4)
indication of the abstract parameters of the
quality of service desired – somehow treat
high precedence traffic as more important –
indicates the format of the internet header, i.e., version 4
tradeoff between low-delay, high-reliability, and
length of the internet header in 32 bit words, and thus
high-throughput – NOT used, bits now reused
points to the beginning of the data (minimum value of 5) for differential services code point
first zero, fragments allowed
and last fragment
1
identifying
value to aid
assembly of
fragments
disable a packet
to circulate
forever,decrease
value by at least
1 in each node –
discarded if 0
0
2
3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version| IHL |Type of Service|
Total Length
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
Identification
|Flags|
Fragment Offset
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Time to Live |
Protocol
|
Header Checksum
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
Source Address
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
Destination Address
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
Options
|
Padding
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
indicates used transport
layer protocol
32-bit address fields. May be configured
differently from small to large networks
options may extend the header – indicated by IHL. If the options do
not end on a 32-bit boundary, the remaining fields are padded in the
padding field (0’s)
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
datagram length
(octets) including
header and data allows the length
of 65,535 octets
indicate where
this fragment
belongs in
datagram
checksum on the
header only – TCP,
UDP over payload.
Since some header
fields change(TTL),
this is recomputed
and verified at
each point
Internet Control Message Protocol (ICMPv4)
Type of the control msg, including
echo request (8) and echo reply (0)
Refinement
Checksum for the
ICMP header only
0
1
2
3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
Type
|
Code
|
header checksum
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
data
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
type-specific arbitrary length data
ICMP Echo Request
0
1
2
3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
8
|
0
|
header checksum
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
identifier
|
sequence number
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
data
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Optional identifier,
chosen by sender, echoed by receiver
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
Optional sequence number,
chosen by sender, echoed by receiver
UDP
port to identify the
sending application
port to identify
receiving application
0
1
2
3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
Source Port
|
Destination Port
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
Length
|
Checksum
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
specifies the total length of the UDP
datagram in octets
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
contains a 1’s complement checksum over
UDP packet and an IP pseudo header with
source and destination address
TCP
code bits: urgent, ack, push, reset, syn, fin
sequence number
for data in payload
port to identify the
sending application
port to identify
receiving application
acknowledgement
for data received
0
1
2
3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
Source Port
|
Destination Port
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
Sequence Number
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ receiver’s buffer
header length in |
Acknowledgment Number
| size for
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
32 bit units
additional data
| Header|
|U|A|P|R|S|F|
|
| length| Reserved |R|C|S|S|Y|I|
Window
|
|
|
|G|K|H|T|N|N|
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
Checksum
|
Urgent Pointer
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
Options
|
Padding
|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
contains a 1’s complement checksum over
UDP packet and an IP pseudo header with
source and destination address
pointer to urgent data in segment
options may extend the header. If the options do not end on a 32-bit
boundary, the remaining fields are padded in the padding field (0’s)
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
Encapsulation
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
Identifying Web Packets
These are the header
fields you need for
the web bumper:
 Ethernet type 0x800
 IP type 6
 TCP port 80
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
Lab Setup
Lab Setup
IXP lab
switch
switch
Internet
…
Student lab
University of Oslo
switch
INF5062, Carsten Griwodz & Pål Halvorsen
Lab Setup - Addresses
IXP lab
switch
switch
…
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
Lab Setup - Addresses
IXP lab
129.240.66.50
switch
switch
129.240.66.51
192.168.2.51
129.240.66.52
192.168.2.52
…
…
…
129.240.66.55
University of Oslo
192.168.2.50
192.168.2.55
INF5062, Carsten Griwodz & Pål Halvorsen
192.168.2.11
Lab Setup – Data Path
SSH connection
to IFI: 129.240.66.50
IXP lab
PCI
IO
switch
hub
switch
hub interface
…
memory
hub
IXP2400
system 192.168.1.1
bus
RAM CPU
interface
memory
web bumper
(counting web packets and forwarding
all packets from one interface to another)
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
192.168.1.5
Lab Setup – Data Path
SSH connection
to IFI: 129.240.66.50
IXP lab
IO
switch
hub
192.168.2.50
IXP2400
CPU
192.168.2.11
…
memory
hub
switch
192.168.1.1
192.168.1.5
memory
web bumper
(counting web packets and forwarding
all packets from one interface to another)
SSH connection to IFI: 129.240.66.48
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
The Web Bumper
web bumper
wwbump
(core)
XScale
microengines
input
port
rx
microblock
the wwbump microblock checks all packets from rx
block: if it is a ping or web packet:
 if web packet, add 1 to web counter and
forward to tx block
 if ping packet, forward to wwbump core component
 if neither, forward to tx block
The wwbump microblock forwards all packets from the
wwbump core component to the tx block
IXP 2400
the wwbump core components
checks a packet forwarded by the
wwbump microblock
 count ping packet - add 1 to icmp
counter
 send back to wwbump microblock
wwbump
(microblock)
tx
microblock
web bumper
(counting web packets and forwarding
all packets
to another)
rx block processing encompasses
all from one interface
tx block
processing encompasses all
operations performed as packets arrive
operations applied as packets depart
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen
output
port
Starting and Stopping
 On the host machine
− Location of the example: /root/ixa/wwpingbump
− Rebooting the IXP card: make reset
− Installing the example: make install
− Telnet to the card:
• telnet 192.168.1.5
• minicom 
 On the card
− To start the example: ./wwbump
− To stop the example: CTRL-C
 Let’s look at an example...
University of Oslo
INF5062, Carsten Griwodz & Pål Halvorsen