Transcript 20070717-bobyshev-demar-grigoriev

Lambda Station: On-demand flow based routing for data
intensive GRID applications over multitopology networks
Fermi National Accelerator Laboratory
Don Petravick (PI), Phil Demar, Matt Crawford,
Andrey Bobyshev, Maxim Grigoriev
California Institute of Technology.
Harvey Newman (co-PI)
ESCC/Internet2 Joint Techs Workshop
Fermilab, July 15-19, 2007
Outline of the talk
●
goals and major directions of the project
●
software architecture, API, middleware
●
results of using Lambda Station service
●
problems and challenges, plans
Basic terms
Lambda Station – a host with special software to control traffic path
across LAN and WAN on-demand of applications
●
●
PBR – policy based routing
PBR Client – a system or cluster and applications running on it
sourcing traffic flows that can be subject for policy based routing
●
Flow - a stream of packets with some attributes in common such as
endpoint IP addresses (or range of addresses), protocols, protocol's
ports if applicable and differentiated services code point (DSCP).
●
The goal of the project...
deals with the last-mile problem in local area networks
The main goal of Lambda Station project is to design, develop and deploy a
network path selection services to interface production storage and computing
facilities with advanced research networks.
–
selective forwarding on a per flow basis
–
alternate network paths for high impact data movement
–
access control in site edge routers for those selected flows
–
on-demand from applications (authentication & authorization)
–
current implementation based on policy-based routing & including the
support of DSCP marking
Flows and DSCP tagging
Any combination of flow's attributes can be used by Lambda Station software
to identify flows on per ticket's basis.
Typical steps of alternative path reservation:
generate request for service (application, application's proxy,
admin)
● LS negotiates service and parameters with remote site
● configure local and wide area network (not yet available)
● marking traffic (if specified).
●
Current LS software is capable to complete all these steps within 3 – 5 mins. That is
why it is desirable to know flow selection parameters before transferring is started:
● endpoint IP addresses
● DSCP
Lambda Station Building Blocks
Storage & application space
Management
RemoteLambdaStation
SOAP/JClarens
SOAP/JClarens
LSInterface
LS-Management &
Reporting Interface
LS request interface
LS-to-LS service
mySQL:requests,
Authorization
SOAP over HTTPS
Service-based Architecture:
LS Controller – manages LS persistence,
controls other services
●LS persistence is stores current state of the
system
●LS request Iinterface webservice for
placing all kinds of “ticket” related requests
●LS-to-LS – webservice for LS definitions
propagation and LS discovery
● NETWORK CONFIGURATOR –
dynamic reconfiguring of LAN and WAN
●
local
definitions
LS Controller
LS persistence
online
updates
NETWORK CONFIGURATOR
Vendor specific modules
Force10
CISCO
Data Exchange
Control & Management
Si
Si
WAN
LS-2-LS service
LS JAVA API Components
Client API
Tomcat + Axis container
JClarens
LS Service Request
Interface
LS PBR clients
Interface
LS/PBR Config Manager
LS Controller
LS NetConfig
Java implementation of LS Software
Service Oriented Architecture
● utilized JClarens and Axis framework as a web-services toolkit
● messages are defined and strongly validated by XML schema
●LS service is multi-threaded, one thread for LSController, one thread
for LS2LS service and threads pool for openSvcTicket requests
● LS2LS and client-2-LS authentication is based on gLite library and
supports standard Grid proxies and KCA-issued certificates
●Authorization is based on rules set
●General framework persistence is accomplished by MySQL DB
backend
● secure document/literal wrapped SOAP messages, Web Services
Interoperability Profile (WS-I Basic Profile Version 1.1)
●
Java implementation of LS Software
(continued)
Automated LS and PBR client configuration management
●Automated deployment ( one can install on any Linux box)
●LS Controller, LS2LS , LS AAA, LS client interface are
ready for deployment. Supported java and perl clients.
●Some interest from ANL to support C client for Globus
toolkit
●Network Config calls implemented in interface and may
relay requests to perl service ( SOA at work )
●Currently deployed and work ( exchanging PBR and LS
configurations) at FNAL ( 2 stations) and CalTech ( 1 station)
●
DSCP Tagging
Complexity of using DSCP tagging:
●
preservation of DSCP is not guaranteed in WAN
DSCP tagging needs to be synchronize between sites for
dynamically configurable networks ( asymmetry is bad for highperformance transferring )
●
LS software does support two different modes of DSCP tagging :
●
fixed DSCP values to identify site's traffic.
●
DSCP value is assigned dynamically on per ticket base.
Netconfig Module
dynamically modifies the configurations of local network
devices.
● a vendor dependent components.
● Cisco routers with IOS version supporting sequencing
type of named ACLs.
●
Tasks to configure PBR in Cisco devices:
● interface on which PBR is applied needs to be configured
with “ip policy route-map” statement
● route map needs to be configured as ordered list of
match/action statements
● match criteria need to be associated with ACLs
openSvcTicket request
Accepts many input parameters, most of them could also be determined
automatically:
● Remote LambdaStation site's identifiers
● Source/Destination PBRClient identifier or/and list of IP netblocks
●IP protocols, protocol's ports or port range
● localPath, remotePath identifiers ( subject to available resources)
● Bandwidth out, Bandwidth in
● boardTime, startTime, endTime, travelTime
Returns ID of locally assigned ticket
Operational modes (subject to authentication, authorization):
new ticket - creating a new ticket
● join - provides ID and required parameters(DSCP) of already opened ticket
● extend ticket – allows to extend already created ticket.
●
LSiperf End-to-End Test
1. Data transfer started:
–
10GE host; 5 tcp streams
–
Network path is via ESnet
–
OC12 bottleneck…
–
Path MTU is 1500B
–
LambdaStation service ticket is
opened
4
2. LambdaStation changes
network path to USN
3. Host path MTUD check
detects a larger path MTU
2
4. LambdaStation service ticket
expires:
–
Network path changed back to
ESnet
1
3
Lambda Station TestBed
SRM/DCache 1.7 LS-awareness
Advanced Networks
CalTech
Wide Area Network
SRM
CalTech LambdaStation
StarLight
FNAL LambdaStation
CMS core router
Site Network
CMS SRM
USCMS Tier1
normal traffic flow
High Impact traffic
Production USCMS SRM server sends requests to
Lambda Station to stir a high-impact traffic into
Advanced Network infrastructure
How to demonstrate increasing transfer rate ?? In tests we
could generate a data stream and monitor its rates when
switching traffic between alternative paths. Production traffic
is not deterministic.
Project accomplishments
Software version 1.0 (a fully functional prototype supporting whole
cycle of LS functionality)
● positive results of testing between Fermilab and Caltech
●lsiperf, lsTraceroute – wrappers around well known applications to
add Lambda Station awareness ( based on prototype version 1.0)
● SRM/dCache integration – testing, added in production 1.7.0 release
● run LS-aware SRM/dCache on production cluster at Fermilab
●Interoperable Java implementation of the major components of
LambdaStation ( perl, java clients available)
●
Problems and challenges
Traffic Asymmetry is bad for high performance applications
● Network (Lambda Station) awareness is too complex
● Definition of PBR Clients is a complex issue, auto definition is
not yet available
●
Plans
●
release fully functional Java LS
SRM/dCache with production quality LS support
● add real-time monitoring of resources ( perfSONAR )
● add WAN control plane module
● integration with OSCARS ( unified Network Path Reservation Model ?)
●
References: http://www.lambdastation.org/
END
Miscellaneous Slides
Simulation of multiple Lss
● screen shoots of ticket's queue
● SC05 Demo
●
Netconfig Module
dynamically modifies the configurations of local network
devices.
● a vendor dependent components.
● Cisco routers with IOS version supporting sequencing
type of named ACLs.
●
Tasks to configure PBR in Cisco devices:
● interface on which PBR is applied needs to be configured
with “ip policy route-map” statement
● route map needs to be configured as ordered list of
match/action statements
● match criteria need to be associated with ACLs
openSvcTicket request
Accepts many input parameters, most of them could also be determined
automatically:
● Dst site's identifiers
● Src/Dst PBRClient identifier or/and list of Src/Dst IP in (CIDR)
● IP protocols, protocol's ports
● localPath, remotePath identifiers
● BWout,BWin
● boardTime, startTime, endTime
Returns ID of locally assigned ticket
Operational modes (subject authentication, authorization and quoting):
new ticket - creating a new ticket
● join - provides ID and required parameters(DSCP) of already opened ticket
● extend ticket – allows to extend already created ticket.
●
Directions of the project.
building a wide-are testbed infrastructure
● designing, developing Lambda Station software, Lambda
Station (network) aware applications
● researching effects of flow based switching on applications
●
Application's behavior in flow based
switching environment
Tuning of end systems for maximum rates is not a subject
of the Lambda Station project, however, we need to see an
increase of data movement performance when selectively
switch flows
DSCP tagging with IPtables
● switching between two paths with different MTUs
●
Effect of DSCP tagging with IPTables
Motivation of the project
unprecedented demands for data movement in physics experiments such
as CDF, D0, BarBar and coming LHC experiments
● massive, globally distributed datasets growing to the 100 petabytes by
2010
● collaborative data analysis by global communities of thousands of
scientists
● available data communication technology will not be able to satisfy
these demands simply by plain increasing bandwidth in LANs and WANs
due to technology limitations and high deployment and operation costs.
● Advance Research optical networks – greater capacity, no production
quality of service, are not universally available for all pairs of
communicating endpoints
●
LS multitopology network model
NG-ADM
Multiple Network Toplogies
Admission Group
of network devices
Blue
NG-B
Green
RT1
RT3
RT2
Red
RED-B-IN
NG-C
RT1
RT3
RT2
BLUE is Production
Path
ClientA rules for Red&
GREEN topologies:
GREEN-OUT
RT1
H1
H2
PBR-clients or regular clients
at the remote sites
RED-ClientA-IN
GREEN-ClientB-IN
RED-ClientB-IN
GREEN-ClientB-IN
Client A, RED & GREEN
rules for NGC
PBR-client
H3
PBR-client B
H1
RT2
H2
RED-OUT
RT3
H3
PBR-client A
NG-A
Cisco IOS, dynamic configuring
of PBR, extended sequencing
ACLs + access policy ACLs
LambdaStation SC05 Demo
Fermilab
SC05/Seattle
Commodity
Internet/SCinet
lambdastation@FNAL
default path
LS-2-LS
lambdastation@SC05
protocol
netconfig
alternative path for
specific flows only
SC05/HighSpeed
Links
reply
ls-request
NAA-2-NAA
protocol
nws-lab.fnal.gov
A122.302.sc05.org
lsiperf
charley.fnal.gov
srmcp
A126.302.sc05.org
PMTU
D
Note A
Note A: We believe it is a HW/ASIC problem with SNMP monitoring, a time to time SNMP -get returns the same counters as in previous cycle.