Transcript A Distributed Resource Management Architecture that Supports

Globus Advance
Reservation and Network Environment
Testbed
GARNET
Volker Sander, Alain Roy
[email protected]
Argonne National Laboratory
10/07/1999
Outline

The Basic concept of GARA

GARNET and Experiences

How to join the Effort

Demonstration
2
Goals

Provide end-to-end Quality of Service to
applications. This requires:
– Discovery and selection of resources
– Allocation of resources
– Advance reservation of resources
Supercomputer
Workstation
Workstation
3
Difficulties/Solutions

Lack of support for advanced reservations
– NGI Bandwidth Broker effort is based on
immediate reservations

Heterogeneous resources
– More than just Networking QoS
– We provide uniform interfaces

Need to work with complex sets of resources
– Multi-domain, Multi-Resource-type
– We use co-reservation agents
4
Solution: GARA
Globus Architecture for Reservation and
Allocation

Three important contributions:
– Support for Advance Reservations (AG)
– Uniform treatment of underlying resources
> Network, disk, CPU etc…
> Currently supported:
Differentiated Services, DPSS, and DSRT
– Layered architecture enables generic coreservation agents
> User is able to select API best suitable for his need
5
GARA Basics—Reservations

There is a generic “reservation”, which has
several properties:
– Start Time (“now” or future) and Duration
– Resource type/Underlying resource identifier
– Resource-specific (bandwidth, % CPU…)

All reservations are treated uniformly:
– Create/Modify (Given properties)
=> Returns Reservation Handle
– Destroy
– Monitor (Callbacks or Polling)
6
A picture of what happens
(creating a reservation)
User Program
or Agent
Create
Reservation Handle
Gatekeeper
(Authenticate and
Authorize)
Gatekeeper Service
Local Resource
Manager API (LRAM)
Resource Manager
Resource (Router...)
7
A picture of what happens
(claiming a reservation)
User Program
bind(handle)
Resource Manager
Setup reservation
LRAM API
Resource
Enforce Reservation
Note: GARA supports third party binding
8
Example: Network reservation

In advance, make a reservation:
 RSL:
& (reservation-type=network)
(start-time=now) (duration=3600)
(endpoint-a=128.135.11.4) (endpoint-b=128.135.11.1)
(bandwidth=1000)
 Handle: Opaque string H1

At run-time:
– Bind(H1)
9
The Resource Manager



To track advanced reservations, we use a simple
“slot table”
We assume exclusive access to resource through the
resource manager
We aren’t tied to this particular resource manager
100%
% of
available
resource
allocated
0%
Time
10
Co-Reservation Agents

When multiple resources are needed, an
agent:
– Discovers applicable resources via the MDS
– Reserves resources
– Informs user when resources are ready.
– Meta-RM support for integrating other RMs

The uniform interface enables these agents to
be created easily.
– Given this ease, it is convenient to express
different strategies such as:
> Depth-first vs. best-first
> Coping with failures
11
The Big Picture
MDS
Co-Reservation Agent
Gatekeeper
Gatekeeper
GRIO LRAM
Scheduler LRAM
Gatekeeper
Gatekeeper
Diffserv LRAM
DSRT LRAM
Workstation
Supercomputer
Router
Router
Workstation
12
End-to-End Network Reservations

Algorithm for a single network reservation
– Discover which gatekeepers/resource
managers need to be contacted (MDS)
– Make reservation at each one.
End-to-End Co-Reservation Agent
Resource Manager
User process
MDS
Resource Manager
Workstation
13
The GARA API

globus_gara_reservation_create()
 Gatekeeper contact
 RSL reservation specification
 Reservation Handle or Error

globus_gara_reservation_modify()
 Old Reservation Handle
 RSL reservation specification
 New Reservation Handle or Error
14
The GARA API (continued)

globus_gara_reservation_cancel()
 Reservation Handle
 Error

globus_gara_reservation_status()
 Reservation Handle
 Status/Error

globus_gara_reservation_callback_register()
 Reservation Handle
 Callback Function/User Parameter
 Error
15
The End-to-End Agent API

For making network reservations
– Deals with co-reservation along path.

Very similar to GARA API:
– Ex: globus_gara_end2end_create()



Handles MDS lookups.
Handles making multiple reservations, but
provides user with single reservation
handle.
Probably the API you want to use.
16
GARNET goals

Provide dedicated testbed for evaluating current
and future mechanisms used for implementing
GARA
– QoS-Type Network: similar to IETF’s diffserv model
– QoS-Type CPU: DSRT-Scheduler
– QoS-Type Disk: Distributed Parallel Storage System (DPSS)

Demonstrate validity of concept
– End-to-End Reservation
– Application level QoS
17
Quality of Service for IP Traffic


The ability to define or predict the performance
of traffic on a network
Needed in case of congestion
– Leave the current best-effort model
– End-User is able to specify his requirements
Guaranteed bandwidth major concept

Building blocks
– Link Layer QoS (ATM QoS)
Create VC with specific parameters (ABR, CBR, UBR, …)
– IP-QoS
Flow Based QoS: Integrated Services (RSVP)
Aggregate Based QoS: Differentiated Services (Premium service)
18
The Basic Concept
Aggregate based scheduling
Flow based
Set IP precedence for
Set IP precedence for
non exceeding flows
non exceeding flows
with bound
with bound
GARA Reservation
GARA Reservation
19
The Extension
Server
Client
GARA API
Diffserv Resource Manager
20
The Multidomain Extension
Server
Client
GARA API
Diffserv
Diffserv
Resource
Resource
Manager
Manager
21
Building Blocks for Diffserv

Traffic marking based on a specific policy

Admission control

Congestion handling
– Congestion avoidance
– Congestion management

Traffic shaping
Service Level Agreement (SLA)
22
Actual Mechanisms (Cisco)

Committed Access Rate (CAR)
– Per flow based at End-Systems
> Mark packets if they are not exceeding a rate limit
> Drop packets beyond the limit or transmit them via best effort
– Per Precedence between domains
> Corresponds to a static SLS
> RM will configure the limits on demand in later releases
– Allows short term bursts

Weighted Random Early Detection (WRED)

Weighted Fair Queuing

Generic Traffic Shaping or PVC attributes
23
GARnet- Globus Advance
Reservation and Network Env Testbed
Linux
Linux
SUN Ultra 60
SUN Ultra 60
WFQ
CAR
WFQ
Cisco
7507
SUN Ultra2
WFQ
Cisco
7507
CAR
WFQ
Cisco
7507
SUN Ultra2
FastEthernet link
OC-3 link
24
25
Evaluation Tools

Several Modified versions of ttcp (premium)
– GARA-enabled (wait for reservation)
– Support for a desired application rate
> long term control
> short term control
– Consecutive bandwidth reporting
– Bulk transfer ttcp in work

UDP traffic generator (competetive traffic)
– Andy Adamson, University of Michigan
– Added support for automated testing

Modified version of DPSS (premium)
– GARA enabled
– Nice example on how to use GARA
26
UDP Rate comparison for different CAR buffer sizes
(Desired rate: >40Mbps; Average rate limit 40 Mbps)
Packet size 530 Bytes
40500
39500
39000
Buffer size
100KB
1MB
2MB
Overhead for packet
38500
headers (40 Bytes)
38000
37500
37000
29
27
25
23
21
19
17
15
13
11
9
7
5
3
36500
1
Actual Rate (Kbps)
40000
Time (s)
27
Receiving TCP rate with different CAR buffer size
6000
5500
5000
BUFFER
4000
3500
2MB
1MB
500KB
200KB
Limit
3000
2500
2000
1500
1000
500
0
30
00
32
00
34
00
36
00
38
00
40
00
42
00
44
00
46
00
48
00
50
00
52
00
54
00
56
00
58
00
60
00
Actual rate (KB/s)
4500
Desired rate (KB/s)
28
The TCP “Problem”
Receiver
Transmitter
TCP’s sliding window
introduces burstiness of Traffic
Socket rcv buffer
Socket send buffer
MSS-sized (media specific)
TCP segments
40 Byte Header overhead
Ack
Ack
Ack
Maximum outstanding of 1 actual window
LastByteSent - LastByteAcked <= RcvWindow.
29
The TCP “Problem:
45
40
Reno-TCP
Slow Start
Exponential Growth
35
30
Average
25
window
20
size
15
10
5
Congestion Avoidance Phase
Linear Growth
Half Window Size in case of duplicate Acks
0
time
30
The TCP “Problem”

TCP’s flow control
– Traffic becomes bursty if the actual window size
is large
– Bandwidth*Latency Product as ideal window size

TCP’s congestion control
– “Slow Start Phase”
> shrink Window to 1
> increase size with every Ack
– Congestion Avoidance
> Half Window size
> Increase size by one for every full window roundtrip
31
Example of Oversubscribing with TCP
(Attempted Rate: 6000KB/s, Buffer size: 2000000 bytes)
9000
8000
6000
Rate
Limit
5000
4000
3000
96
91
86
81
76
71
66
61
56
51
46
41
36
31
26
21
16
11
6
2000
1
Actual Rate (KB/s)
7000
Time
32
Oversubscribing TCP Rate due to Header Overhead
(Attempted Rate: 6000KB/s, Buffer size: 200000 bytes)
9000
7000
6000
Rate
Limit
5000
4000
3000
96
91
86
81
76
71
66
61
56
51
46
41
36
31
26
21
16
11
6
2000
1
Actual Rate (KBytes/s)
8000
Time
33
Stable Long Term TCP Traffic
(Attempted Rate: 6000KB/s, Buffer size: 2000000 bytes)
9000
8000
Actual Rate (Kbs)
7000
6000
Rate
Limit
5000
4000
3000
2000
1
7
13
19
25
31
37
43
49 55
Time
61
67
73
79
85
91
97
34
WRED impacts on RTT
35
WRED impacts on the Window size
36
… the Throughput
37
… some updates
38
… increasing tcp_cwnd_max
39
… new throughput
40
41
… new throughput
42
Using WFQ
43
Using WFQ
44
Using WFQ
45
Current Status

An early working prototype (alpha!) of
GARA exists
– Differentiated Services-like QoS
– Real-Time CPU Scheduling (DSRT)
– DPSS Disk Access



An early prototype of the end-to-end
reservation agent exists.
Successful demonstration on GARNET and
over ESnet’s NGI testbed
Work with bleeding-edge adopters has
already started.
46
The GARA Source Tree

Subdirectories represent different layers
– end2end
– gara
– lram
– resource_manager
– slot_manager
– (dsrt)


We are providing examples for every layer
docs directory contains lots of useful
information
47
Doing Tests

Select your favorite application

Determine port numbers




edit (set endpoint systems) and run script:
gara/gara/utilities/demo_make_reservation
Create congestion
Try demo_modify_reservation for getting
different premium bandwidth
Try demo_cancel_reservation
48
Install GARA

Get GARA (currently through E-Mail from sander/roy
@mcs.anl.gov)

Get SSLeay 0.9.0 and install it

Get Globus 1.1 configure and compile it
deploy is not necessary for doing tests
Run grid-cert-request to apply for a Grid X509 Certificate

Read gara/docs/*

Edit gara/build

Run gara/build

Customize GARA
– gara/resource_manager/programs/diffserv_manager.conf
– gara/resource_manager/programs/setup_flow.cfg
– (Samples are in the documentation directory)

Experiment!
49
Router Requirements

Cisco 75xx VIP2-50

Get new IOS XE beta release
Experimental Version 12.0(19990728:032821) [rolsen-conn_4xe_fix 243]
Available at ftp://ftpeng.cisco.com/rolsen/rsp-pv-mz.091099





ip cef distributed
Define an empty policy-map set-precedence on the edge
router
Define a consecutive set of access-lists to be used by GARA
(100-199)
Define class-map for precedence 3 or 5 on every router
Define a policy-map containing the class assigned with the
supported premium BW

Assign this as service policy on each output interface

Define CAR per precedence on the ingress site
50
Future Work

Add new features
– Authorization
– Dynamic SLS
– Enhance CLI
– Increase support for adaptive applications

Deployment with Globus
– Packaging
– Easy install

Supporting more resource types
– Job Schedulers
– Alternate Network QoS (RSVP)
– Disk (GRIO)
– Others (Memory, etc…)
51
Conclusions

GARNET is ready ...

GARA could prove its capabilities
– Try it…
– Once installed, it is easy to use!
– Suggestions are welcome!!!!

Take a look at http://www.mcs.anl.gov/qos
52