Transcript Fred Kuhns - Washington University in St. Louis

Endsystem Support for
Network Virtualization
Fred Kuhns
[email protected]
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
Overview
• Network Diversification:
– Virtual network (vNet): distinct vNets coexist within a common physical network
– Diversification layer: common substrate to share physical resources and provide isolation
– vNet is composed of one or more virtual routers (VR) interconnected by virtual links. Virtual routers
and links are direct corollaries to their physical counterparts … Network resources are virtualized.
– An end-system implements vNet protocols and provides connectivity services within a virtualized
network protocol environment (virtual end-system). The virtual end-system provides mechanisms for
protocol implementation, resource control and isolation.
• Diversification layer provides two levels of abstraction (i.e. two core services):
– Substrate: encapsulate existing layer 1 and layer 2 technologies and provide a single, consistent
framework for implementing virtualized links and routers.
substrate link: abstraction to provide similar behavior as a point-to-point connection between
communicating end points. Provides isolation services to different virtual networks using a common
substrate link.
substrate router: A physical device which forwards network traffic based on its vNet membership.
Provides sharing and isolation services to disparate vNets and hosts virtual routers.
– Virtual: framework providing a simple model and set of interfaces for implementing virtual
networks. The model defines virtual routers, end-systems and links. The goal is for virtual inks to
and routers to behave similar to their physical counterparts.
virtual link: simulates the behavior of a dedicted point-to-point link interconnecting virtual end
points (virtual routers and/or virtual end systems). A virtual link is implemented by one or more
substrate links.
virtual router: implements a particular vNet’s routing logic. The underlying substrate router
provides the necessary isolation and resource management functions.
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
2
vNet Discussion
• Develop examples/scenarios
– intranet (no routing) use existing model
– internet (routing) use diversified networking model
– use Ethernet and virtualized IP as running example
• Model: Simple
– network devices interconnected through simplex, point-to-point links.
– common link layer protocol used for delivering packetized data to
neighbor (not end-to-end but hop to hop)
• Achieving this model
– context: shared heterogeneous physical network, links and packet switches
(aka packet routers)
– objectives:
• partition physical resources into virtual links and routers
• isolation mechanisms for virtualized resources
• bind virtualized resources to network instances
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
3
Context: Network Diversification (vNets)
substrate
link
substrate
router
virtual
router
virtual
link
virtual
end-system
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
4
Simulates Star Topology for Substrate Links
…
VLANX1
Internetworking over a diversified network
Substrate function with Ethernet:
• Substrate links: use VLANs to provide the equivalent
of a virtualized “wire” connecting an endsystem to a
specific substrate router.
• Sharing and Isolation:
- All vNet traffic use assigned VLANs
- Use priority queuing (802.1P/Q)
- All intranet traffic uses lower priority queues.
• Resource management:
- LAN: Use admission control (static or dynamic) to
provide bandwidth guarantees to vNet traffic.
- End system: Substrate layer on end-system enforce
per VLAN and per vNet bandwidth constraints
• Virtual links: In this simple example there is exactly
one virtual link for each substrate link.
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
VLANX2
VLANXN
switched LAN
vNetX
VR1
• Each host to substrate router connection is
assigned a distinct VLAN. So N hosts implies
N VLANs on Ethernet.
• Alternative is to define one VLAN tree for
each protocol suite (i.e. vnet).
5
Traffic isolation with priority aware substrate
…
Ethernet Hub
with High and Low
Priority TX queues
Low
High
Low
Low
High
Low
vNet traffic to High
otherwise Low
High
Local control/management;
Legacy internet traffic
vNet traffic (internet)
all vNet traffic
vNetX
VR1
Local traffic (intranet)
Fred Kuhns - 4/8/2016
High
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
6
Substrate Link as a VLAN Tree
…
Internetworking over a diversified network
Substrate function with Ethernet:
• Substrate links: The VLAN creates a tree
interconnecting all end-systems to the substrate router.
Substrate end-point then uses the VLAN tag and
source/destination address to realize the logical pointto-point substrate link.
• Sharing and Isolation:
- no change from substrate star topology. The only
difference is the shared VLAN domain. Scheme
provides traffic isolation.
• Resource management:
- Same
• Virtual links: Same.
Fred Kuhns - 4/8/2016
ethernet switched LAN
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
VLANX
7
Multiple Substrate Links
…
Internetworking over a diversified network
Substrate function with Ethernet:
• Substrate links: Three VLAN trees are used for all
virtual net traffic to/from a substrate router:
- Low priority: default for best-effort traffic
- Medium priority for virtual nets with soft
performance requirements (average bandwidth)
- High priority for isochronous or low-delay,
interactive applications
• Sharing and Isolation: See above.
• Resource management: See above
• Virtual links: Same.
Fred Kuhns - 4/8/2016
ethernet switched LAN
VLANdgram
VLANhigh
VLANmed
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
8
Multiple vNets per Host
…
virtual interface
substrate interface
ether
addr/vlan
ether
addr/vlan
VLAN1
The full model:
• Substrate link: connects end-system to substrate router.
Virtualization of a physical cable or wire. A packet
enters one end, exists the other and is opaque within.
- Simplex or Duplex?
• Substrate interface: end-system abstraction
- Ethernet: <interface, VLAN, dst_addr>
- tunnel: MPLS, IP, IPsec, L2TPv3, GRE, AToM
- Layer 2: ATM, others?
• Virtual link: Logical interconnection (virtual wire) of
adjacent vNet nodes.
- Point-to-point, Simplex or Duplex?
• Virtual interface: end-system abstraction representing
one end of a virtual link. Substrate defines mechanism
for multiplexing onto common substrate link. For
example a virtual link identifier (VLI) in a substrate
header
- Simplex or Duplex?
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
ether
addr/vlan
VLAN2
VLAN3
ethernet LAN
VLAN tag and dst addr
identify substrate
router. VLI tag
used to router pkt
substrate interfaces
VLI
VLI
VLI
VR1
virtual interface
VR1
VLI
VLI
9
Multiple next hop VRs
Host A
member of
vNetX and vNetY
substrate
router 2
vNetX
VR2
VLI1
enetAddrSR2
enetAddrA
VLANA2
Multiple Next Hop Virtual Routers:
• Substrate link: per end-system, substrate router pair.
• Substrate interface: three substrate interfaces:
ethernet switched LAN
SI1 = <eth0, VLANXA1, enetAddrSR1>
SI2 = <eth0, VLANXA2, enetAddrSR2>
VLANA1
SI3 = <eth0, VLANXA3, enetAddrSR3>
• Virtual link: Logical point-to-point connection between
virtual end-system and access virtual router. Since we model
enetAddrSR1
a point-to-point link there is no need for link addresses.
VLI1
VLI2
• Virtual interface: Representation of virtual link on the endsystem. The substrate assigns a per substrate link, virtual link
vNetX
vNetY
identifier (VLI) for each virtual link.
VR1
VR1
VI1 = <SI1, VLI1>
substrate
VI2 = <SI1, VLI2>
router 1
VI3 = <SI2, VLI1>
VI4 = <SI3, VLI1>
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
substrate
router 3
vNetX
VR3
VLI1
enetAddrSR3
VLANA3
10
TCP/IP as an Example Protocol
destination
prefix
gateway
(router address)
192.168.12.0/24
*
(default)
virtual interface
substrate
interface
ll_info
0.0.0.0
eth0
ARP
192.168.12.254
vint0
(eth0,VLAN,ethDst)
VLI
vNet Protocl = IP
vNet
framework
vint0
VLANX
eth0
standard ethernet
Interface
…
eth0
direct connect
ethernet device
VLANX
Substrate Interface:
Directly connected: destination IP address + ARP = enet addr
Gateway: (Gateway’s IP + ARP = enet addr) + VLAN
Virtual Interface:
Directly connected: Not used, model only for internetworking
Gateway: VLI assigned by substrate.
How is this integrated into the current ARP/route interface?
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
ethernet LAN
ethernet
dest. addr
VLAN
VLI
VLI
Substrate Router
SR1
VLI
IP
11
Using Tunnels for the substrate layer
• Need to look into the various tunneling approaches/protocols.
How can we leverage these?
–
–
–
–
–
–
–
–
MPLS and MPLS VPNs
Generic Routing Encapsulation (GRE): RFC 2784
Point-to-point tunneling protocol (PPTP)
Secure VPN
Any transport over MPLS (AToM)
IP tunnel
IPsec VPNs
Layer 2 Tunneling Protocol version 3 (L2TPv3)
• version3 is a draft standard
• RFC 2661: Layer 2 tunneling protocol
– 802.1Q Tunneling: Cisco 802.1Q-in-Q VLAN Extension Services
• What about MPLS over IP tunnels: what was done there?
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
12
OS Kernel Block Diagram
User Space (Applications)
AST Processing
File Interface
ops
FS management
open
files
…
TCPn
UDP
RAW IP
1
callback
task management
util
tasks
Interrupt Processing
TCP module
TCP
TCP2
Basic I/O Interface
buffer
cache
op
s
Socket Interface
hardware independent layer
Device independent I/O
hardware dependent layer
uart
Hardware
timer
Fred Kuhns - 4/8/2016
scheduler
SW int
(AST)
TCP
poll
callout Q
IP
TC/
AST
routes
qdisc
clock handler
process accounting
scheduling
time management
device driver
ethernet
configuration: registers, MMU (TLB, cache, VM) bus
and peripherals
System Exception handlers
OS ISR demux
Washington
HW interrupt/Exception
WASHINGTON UNIVERSITY IN ST LOUIS
eth0
txqueue
rxqueue
13
User or kernel Space protocols?
• Each has pros and cons
• User space protocols:
– easier to implement and debug
– easier to introduce new protocols (not tightly dependent on socket layer
knowing about the new protocol)
– easier to isolate and protect protocols and apps from each other (leverage
process model)
• kernel level protocols
– easier to integrate into existing framework (simplifies support for system
interface functions like select/poll)
– simplifies intra-protocol security and protection (since protocol runs within
trusted kernel)
– simplifies (well, more direct) kernel demultiplexing to correct protocol
context (endpoint)
– increased efficiency
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
14
User Space Protocol Implementation
• Uncommon outside of high-performance community, they want
zero-copy and specialized demux keys.
• Problems: asynchronous processing, life cycle, authentication and
demultiplexing to endpoints
– latency in delivering packets (i.e. acks) to user space
– increased overhead in per packet processing before a drop/keep decision is
made
– processing received acks
– timeouts and retransmissions
– establishing connections and security: snooping, masquerading
– supporting select and poll
– protocols where connection may outlive process (TCP’s TIMED_WAIT)
– global routing and address resolution tables
– global connection tables
• need to know what other ports are being used (locally)
• accepting/rejecting new connections
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
15
Assumptions
• Assumptions:
– Applications using different VNs (or no VN) will need
to communicate using the various IPC mechanisms
– We want to manage all aspects of Network I/O but not
the use of other traditional resources (memory, files etc)
– CPU, memory and interface bandwidth controlled at
the virtual net granularity
– intra-VN, implementers should have the mechanisms to
support QoS and Security
– simple mechanism for adding new protocols/VNs
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
16
User Space Protocols
 Chandramohan A. Thekkath , Thu D. Nguyen , Evelyn Moy , Edward D.
Lazowska, Implementing network protocols at user level, IEEE/ACM
Transactions on Networking (TON), v.1 n.5, p.554-565, Oct. 1993
 Chris Maeda, Brian Bershad, Protocol Service Decomposition for HighPerformance Networking, Proceedings of the 14th ACM Symposium on
Operating Systems Principles. December 1993, pp. 244-255.
• Aled Edwards , Steve Muir, Experiences implementing a high
performance TCP in user-space, Proceedings of the conference on
Applications, technologies, architectures, and protocols for computer
communication, p.196-205, 1995
• Kieran Mansley, Engineering a User-Level TCP for the CLAN
Network, Proceedings of the ACM SIGCOMM workshop on Network-I/O
convergence: experience, lessons, implications, Pages: 228 – 236, 2003
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
17
user-space protocols: Global Issues
•
Routing: Direct packets to/from correct endpoint/interface
– How is traffic demultiplexed and sent to the correct endpoint/process?
• In-kernel filters
– Where are the routing tables and how are they maintained?
• route fixed when connection established or located in shared memory
•
Control: I use IPv4 as an example
– Address resolution protocols/tables?
– Other control protocols. For example ICMP, IGRP, others?
– Where are the routing protocols implemented?
•
Management:
– Must manage a protocols namespace (for example, port numbers in IPv4).
– Common programming technique, allow protocol instance to select local address part
• specify port = 0 and addr = 0 then implementation will assign correct values
– Passive connect model?
• In IPv4 a server listens on a port (host:port:proto) for a connection request. To establish a connection a
unique (to the endsystem) port number is assigned and new socket allocated.
– socket-oriented system calls must be supported. On UNIX must support non-blocking I/O with
select and poll.
– Connection lifetime may outlast process.
• For example TCP TIME_WAIT or simply waiting for a final ack or resending if no ack received.
•
Security: we must provide sufficient mechanisms for protocol developers
–
implementations must be able to guard against masquerading and eavesdropping
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
18
User Space: Configurations
• Given these global issues there are two likely
configurations:
– all traffic passes through common protocol daemon in user
space
– control daemon implements basic set of control functions while
user library implements majority of data path functions
– prior work has shown the latter approach to be superior.
• Having all traffic pass through a common protocol
daemon => at least one extra copy operation (kernel ->
daemon -> user process)
• A better solution is for a daemon to insert relatively
simple packet filters in kernel for established connections
which directs packets to/filters packets from endpoints.
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
19
User-Space: Passive Open
0. listen/accept
(passive open)
vnetX: application
protocol
library
vnetX
control daemon:
(namespace, lifecycle, connections)
4. new connection
data copy
socket layer
3. insert incoming and
outgoing filters for
vnetX connection
1. connection
request (in)
5. data, established
connections
compare against connection
specific outgoing filter
2. ack (out)
vnet demux
connection filters
ethernet
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
use VLI to access incoming filters
and use to demux to filter set and/or
socket.
20
User-Space: Active Open
0. connect
vnetX: application
protocol
library
vnetX
control daemon:
(namespace, lifecycle, connections)
4. new connection
data copy
socket layer
1. connection
request (out)
3. insert incoming and
outgoing filters for
vnetX connection
5. data, established
connections
compare against connection
specific outgoing filter
2. ack (in)
vnet demux
connection filters
ethernet
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
use VLI to access incoming filters
and use to demux to filter set and/or
socket.
21
User-Space: Datagram (Connectionless)
daemon fills in local address and binds to
socket. No restrictions on destination
0. open(any)
vnetX: application
protocol
library
vnetX
control daemon:
(namespace, lifecycle, connections)
2. new connection
socket layer
(local address)
1. insert incoming and
outgoing filters for
vnetX connection
data copy
3. data established
connections
compare against “connection”
specific outgoing filter
vnet demux
connection filters
ethernet
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
use VLI to access incoming filters
and use to demux to socket. In this
case only the local part is used.
22
User-Space: Datagram (Connectionless)
daemon fills in both local and destination
addresses. Destination restricted
0. open(local and remote addr)
vnetX: application
protocol
library
vnetX
control daemon:
(namespace, lifecycle, connections)
2. new connection(local and remote) data copy
socket layer
1. insert incoming and
outgoing filters for
vnetX connection
3. data established
connections
compare against “connection”
specific outgoing filter
vnet demux
connection filters
ethernet
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
use VLI to access incoming filters and
use to demux to socket.
23
User-Space: App exits
TCP enters TIME_WAIT after close
vnetX: application
protocol
library
vnetX
control daemon:
(namespace, lifecycle, connections)
socket layer
3. remove filters
1. connection
close (out)
2. ack (in/out)
vnet demux
connection filters
ethernet
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
drop
24
Extensible protocol frameworks in the kernel
• Herbert Bos, Bart Samwel, Safe Kernel
Programming in the OKE, Proceedings of the
fifth IEEE Conference on Open Architectures and
Network Programming, June 2002
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
25
OKE
•
•
•
Context: For performance reasons it is useful to permit third parties to load optimized
modules into the kernel
Problem: Third party code is untrusted so loading into kernel will compromise system
security and reliability. Could use safe execution environment like java but incurs
expensive runtime checks.
Solution: create set of mechanisms and policies to permit non-root users to safely load
untrusted application modules into kernel space with minimal impact on runtime
performance.
– Safety: use a trusted compile to enforce policies (constraints). The constraints are designed to
ensure the untrusted module will not adversely affect the kernel (core and loadable modules) or
unrelated processes.
– User privileges: Vary enforced constraints based on user privileges (customizable language)
– Termination: well defined termination boundaries to protect system state
– Enforcement: Static and dynamic checks; language extensions
– Ease of use: Familiar development environment using Cyclone (type safe, C extension) and
kernel module.
•
Contribution: definition of safe kernel programming environment that meets competing
needs:
–
–
–
–
performance
safety
ease of use
hosted in a commodity OS
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
26
Considerations
• Identified areas where modules may impact system
behavior
1. program correctness: language restrictions for safety and
enforce coding conventions
2. Memory access: static and dynamic enforcement of
memory access rules
3. Kernel module access: static and dynamic enforcement
of kernel module (interface) access restrictions
4. Resource usage: Bounded (deterministic or limited)
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
27
Pushing protocols into the Kernel
• Positives:
– All the issues associated with user-space protocol simply go
away. Global tables and lifetime of the kernel
– Performance, efficiency, existing code base
– Enhances intra-Protocol security
– Simplifies integration with existing network I/O subsystems and
interfaces
• Negatives:
– Isolation: More difficult to isolate system from protocol
instances. Inter-protocol isolation difficult.
– Security: Proving trust/security more difficult
– Implementation and debugging more difficult in kernel
Fred Kuhns - 4/8/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
28
Kernel-Space Protocols
Application(s)
Rework!
/dev/protoX
User Space (Applications) /dev/vnet
vnet:ep
File Interface
tcp:port
PF_VNET
Socket Interface
I/O Interface
buffer
cache
vnet
vnet:ep
…
udp:port rawIP
ops
FS management
open
files
…
TCP
TCP1
Socket I/O Interface
vnet ops
vnet Proto
state tables
…
vnet Proto
state tables
TCP/IP
IP
TCP2
PF_INET
…
TCPn
route to interface
UDP RAW IP
routes
SW Interrupt
HW Interrupt
ethetnet
vnet Demux
VLAN
Hardware
Fred Kuhns - 4/8/2016
eth device driver
eth0
HW interrupt/Exception
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
29