Transcript Phil DeMar

La
ta
aS
d
mb
n
tio
LambdaStation
Phil DeMar
Don Petravick
NeSC
Oct. 7, 2004
n
tio
Overview
f
La
ta
aS
d
mb
Fermilab
• The Problem Space
• Functional Description of LambdaStation
• Components of LambdaStation
• Technical Approach
• Case Study
• Test Environment
• Project Outline
• Current Status
• Summary
10.07/2004
Demar, Petravick: Lambda Station
2
n
tio
The Problem
f
La
ta
aS
d
mb
Fermilab
• Very high (1-10 Gb/s) bandwidth research
networks becoming available
 Typically not intended for general network traffic
 Alternate paths for high impact data movement
 Vision of the future: optical networks & lots of λ’s
• Not integrated with facility production LANs
 Production systems & storage facilities not directly
accessible
 Customized local access usually required…
 WAN paths difficult to effectively use
Petravick: Lambda Station
 Used mostly withDemar,
prototypes
or in demonstrations
10.07/2004
3
n
tio
The Problem (II)
f
La
ta
aS
d
mb
Fermilab
• Proposed implementations don’t scale
 Multi-homed end system model breaks down with
multiple alternate paths
 Graceful failover capability missing or very complex
 Virtual interface model difficult to administer & adds
complexity on host systems
• Applications may need customization for each
new technology or high-performance path
• A last mile problem between advanced WAN &
facility production LANs…
10.07/2004
Demar, Petravick: Lambda Station
4
n
tio
f
La
ta
aS
d
mb
Proposed Solution:
LambdaStation
Fermilab
• LambdaStation: an alternate path selection
service
 Agent to coordinate use of alternate WAN paths:
 Dynamic configuration adjuster of local network devices
 Goal is per-flow granularity for alternate path selection
• LambdaStation is a research project
 A 2-3 yr US Dept of Energy, Office of Science
 Fermilab and Caltech
 In very early stages of development
10.07/2004
 Still in design phase
 Production network use is a ways off
Demar, Petravick: Lambda Station
5
n
tio
• Advanced research networks
Fermilab
Storage systems
Advanced
Research Network
Internet
Local Network
Infrastructure
Local Network
Infrastructure
• LambdaStation intended to:
Storage systems
 Offer alternate, high bandwidth
WAN paths
 May be a scheduled resource
 Typically demands customized
local access
f
La
ta
aS
d
mb
How LambdaStation
Would Help
LambdaStation
LambdaStation
Storage systems
Advanced
Research Network
Internet
Local Network
Infrastructure
Local Network
Infrastructure
6
Storage systems
 Facilitates application request
for high bandwidth WAN path
 Coordinates alternate WAN path
establishment
 Dynamically reconfigures LAN
forwarding of application’s
data
10.07/2004
Demar, Petravick: Lambda Station
n
tio
f
La
ta
aS
d
mb
Components of
LambdaStation
Fermilab
• A network path controller
with modules interfacing to:
Advanced WANs
Local network infrastructure
Peer LambdaStations
Local applications &
middleware
 AA infrastructure




• Software on host systems
to coordinate with
LambdaStation
10.07/2004
Advanced
network
scheduling
&
provisioning
Local network
equipment
Resource
Predictive
Service
Peer
lambdastation
WAN
module
VO
module
local steering
service
local demand
module(s)
Virtual
Organization(s)
WAN
Space
Local
network
operating
environment
Local
Network
Space
LambdaStation
Layer 4: TCP, UDP, advanced stacks
Layer 5:
Layer 5: Layer 5:
management
direct file file
transport (SRM)
access
Storage System(s)
Demar, Petravick: Lambda Station
Network
awareness
module
Network
awareness
module
Middleware &
Applications
Storage &
Application Space
7
n
tio
f
La
ta
aS
d
mb
Host Interactions
Fermilab
• Path request primitives:
 Flow identification:
 Source/dest address/port
 DSCP code point
 Possibly IP version
 Transfer characteristics
 Bytes, peak rate, duration
 Schedule (range)
• Path status primitives:
 Allocated bandwidth
 Path setup status
 Path termination status
10.07/2004
 Normal or earlyDemar,
abort…
Petravick:
Advanced
network
scheduling
&
provisioning
Local network
equipment
Resource
Predictive
Service
Peer
lambdastation
WAN
module
VO
module
local steering
service
local demand
module(s)
Virtual
Organization(s)
WAN
Space
Local
network
operating
environment
Local
Network
Space
LambdaStation
Layer 4: TCP, UDP, advanced stacks
Layer 5:
Layer 5: Layer 5:
management
direct file file
transport (SRM)
access
Storage System(s)
Lambda Station
Network
awareness
module
Network
awareness
module
Middleware &
Applications
Storage &
Application Space
8
n
tio
f
La
ta
aS
d
mb
LAN Interactions
Fermilab
• LAN device forwarding
reconfigurations
 Graceful alternate path
setup & teardown
• ACL modifications at
site egress point
• Customizable interface
to fit site specifics
Advanced
network
scheduling
&
provisioning
Local network
equipment
Resource
Predictive
Service
10.07/2004
WAN
module
VO
module
local steering
service
local demand
module(s)
Virtual
Organization(s)
WAN
Space
Local
network
operating
environment
Local
Network
Space
LambdaStation
Layer 4: TCP, UDP, advanced stacks
Layer 5:
Layer 5: Layer 5:
management
direct file file
transport (SRM)
access
Storage System(s)
• Non-objectives:
 QoS
 Traffic policing
Peer
lambdastation
Demar, Petravick: Lambda Station
Network
awareness
module
Network
awareness
module
Middleware &
Applications
Storage &
Application Space
9
n
tio
f
La
ta
aS
d
mb
WAN Interactions
Fermilab
• Capability for multiple
alternate WANs
 Customizable interface to
fit each WAN
 Provide abstract view for
applications
• Path reservation
Advanced
network
scheduling
&
provisioning
Local network
equipment
Resource
Predictive
Service
 Resource availability
 Scheduling
• Path setup/teardown
Peer
lambdastation
WAN
module
VO
module
local steering
service
local demand
module(s)
Virtual
Organization(s)
WAN
Space
Local
network
operating
environment
Local
Network
Space
LambdaStation
Layer 4: TCP, UDP, advanced stacks
Layer 5:
Layer 5: Layer 5:
management
direct file file
transport (SRM)
access
Storage System(s)
 Graceful…
Network
awareness
module
Network
awareness
module
Middleware &
Applications
Storage &
Application Space
• Non-objectives:
 QoS
 Link efficiency
10.07/2004
Demar, Petravick: Lambda Station
10
n
tio
f
La
ta
aS
d
mb
Remote Site
Interactions
Fermilab
• Establish path symmetry
 Exchange traffic selectors
 Coordinate WAN path
setup/teardown
 Ack LAN path
reconfiguration
• Verify path continuity
 Implies concurrent
connectivity via reserved &
commodity network paths
 Inform of:
Advanced
network
scheduling
&
provisioning
Local network
equipment
Resource
Predictive
Service
Peer
lambdastation
WAN
module
VO
module
local steering
service
local demand
module(s)
Virtual
Organization(s)
WAN
Space
Local
network
operating
environment
Local
Network
Space
LambdaStation
Layer 4: TCP, UDP, advanced stacks
Layer 5:
Layer 5: Layer 5:
management
direct file file
transport (SRM)
access
Storage System(s)
Network
awareness
module
Network
awareness
module
Middleware &
Applications
Storage &
Application Space
 Path termination
 Traffic termination
10.07/2004
Demar, Petravick: Lambda Station
11
n
tio
f
La
ta
aS
d
mb
Authorization &
Authentication
Fermilab
• Initial AA capability:
 LambdaStation internal
 Site-controlled
 But some trust-level
between sites
• Longer term goal:
 Utilize VO AA
infrastructure
 Raises site-level control
issues
10.07/2004
Advanced
network
scheduling
&
provisioning
Local network
equipment
Resource
Predictive
Service
Peer
lambdastation
WAN
module
AA
module
local steering
service
local demand
module(s)
Virtual
Organization(s)
WAN
Space
Local
network
operating
environment
Local
Network
Space
LambdaStation
Layer 4: TCP, UDP, advanced stacks
Layer 5:
Layer 5: Layer 5:
management
direct file file
transport (SRM)
access
Storage System(s)
Demar, Petravick: Lambda Station
Network
awareness
module
Network
awareness
module
Middleware &
Applications
Storage &
Application Space
12
n
tio
Technical Approach
f
La
ta
aS
d
mb
Fermilab
• Flow identification:
 Source/destination IP and DSCP code point tag
 Source/destination IP & Source/destination port #
• Local path forwarding reconfigurations:
 Based on policy routing ACLs
• Advanced WAN interface model:
 Site egress/ingress router to alternate path(s)
 Layer 2 site-to-site connections (vlan) or MPLS paths
• Transparent failover back to production path:
 Bandwidth limits or QBSS on fall-back traffic
10.07/2004
Demar, Petravick: Lambda Station
13
1.
n
tio
Use Case
f
La
ta
aS
d
mb
Fermilab
FNAL SRM gets request to copy 3.5TB data set to CMS tier-2 site
- Transfer involves 3500 1GB files stored on 50 dCache systems
2.
3.
4.
SRM initiates data transfer optimized for production network path
SRM makes alternate path request to local LambdaStation
LambdaStation:
-
5.
SRM makes parallelism changes to re-optimize data transfer
•
6.
Checks authentication & authorization
Schedules & coordinates establishment WAN path
Coordinates with remote site LambdaStation for symmetrical path
Verifies alternate WAN path continuity
Reconfigures local LAN for selective flow forwarding
Notifies SRM that alternate path is established
SS Potentially shapes traffic to avoid congestion at pipe entrance.
Path reservation expires; LambdaStation gracefully tears path down
-
Remaining data movement traffic reverts to production network path
LambdaStation notifies SRM of alternate path teardown
7.
SRM re-optimizes data
transfer parallelism to complete the transfer
10.07/2004
Demar, Petravick: Lambda Station
14
n
tio
f
La
ta
aS
d
mb
Test Environment
Fermilab
• LambdaStation Project Sites:
 FNAL (US-CMS Tier-1 Center)
 Cal Tech (US-CMS Tier-2 Site)
 Working to include UCSD as
well (US-CMS Tier-2 Site)
 CERN CMS Edge Computing
also planned
Ultra
e t (2
nce Net
UltraScie er ESnet
o
SunnyvaMPLS path10 GvE)
(<
Sunnyvale
• LambdaStation Test WANs:
Scie
nce
N
N
ER )
C GE
To (10
le
x 10
GE)
FNA t Starlight
FNAL
LANL
Cal Tech
UCSD
 US DOE UltraScience Net
 Two 10 GE (OC192) lambda’s
 MPLS tunnels over DOE’s
ESnet
10.07/2004
 Sub-10GE chunk out of ESnet
production network
Demar, Petravick: Lambda Station
15
Year 1
n
tio
Project Outline
f
La
ta
aS
d
mb
Fermilab
LAN integration:
Dynamic LAN reconfiguration
WAN integration:
Interface to DOE UltraScience Net;
Establish alternate path to Cal Tech
Application integration:
SRM allocated bandwidth awareness
----------------------
----------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------
Year 2
LAN integration:
Automated LAN path setup/teardown,
& scheduling; Peer LambdaStation
coordination
WAN integration:
Automated UltraScience Net WAN path
setup/teardown & scheduling
Application integration:
VO sensitivity; Adapt storage systems
to schedule transfers to path avail.
----------------------
----------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------
Year 3
LAN integration:
Harden to production use quality
WAN integration:
Full integration w/ UltraScience Net;
10.07/2004
Demar, Petravick:
Lambda Station
Application integration:
Harden
to production use quality
16
n
tio
Current Status
f
La
ta
aS
d
mb
Fermilab
• A “Pre-LambdaStation” prototype exists:
 Dynamically reconfigures LAN devices to forward select
traffic over alternate paths
 End system-applied DSCP tags for special-treatment flows
• Prototype work-arounds:
 Manual input of reservation parameters
 Static alternate WAN path – no setup required
 Asymmetric – no reverse path setup
• Results:
 Alternate traffic path established & torn down cleanly
 Path reroute & failback unnoticed by application
10.07/2004
Demar, Petravick: Lambda Station
17
n
tio
Summary
f
La
ta
aS
d
mb
Fermilab
• LambdaStation’s role in data-intensive science:
 Dynamically connect production end-systems to
advanced high-performance wide-area networks
 Bring production-use systems to advanced networks
 Bring advanced networks to production-use applications
• LambdaStation’s status:
 Prototyping has demonstrated feasibility of dynamically
reconfiguring local network paths for specific data flows
 Much work remains to be done for complete proof-ofconcept
 It’s a research project…
10.07/2004
Demar, Petravick: Lambda Station
18