Transcript STARTAP_2001Mtg_Leigh.

Application-Level Network
Performance / Measurement Tools
Jason Leigh, Oliver Yu,
Alan Verlo, Linda Winkler
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
Tele-Immersion is the synthesis of Virtual Reality, video
conferencing, and advanced computation
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
Example of a Data Intensive Tele-Immersion
Application
• TIDE: Tele-Immersive Data
Explorer
• Collaborative Large Scale
Data Visualization
• In collaboration with National
Center for Data Mining
• General framework for
collaborative visualization of
massive data-sets
• Current data-set is ozone data
from NOAA
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
Example of a Data Intensive Tele-Immersion
Application
• CIBRView: Collaborative Image Based
Rendering Viewer
• Cosmology Hydrodynamic code by Julian
Borrill, LBNL/NERSC shows theoretical
condensation of diffuse matter into stringlike formations during early stages of
universe evolution
• Accesses volume data:
512x256x256x 256 frames ~ 40Gig datasets
• Generates image slices that are
distributed to collaborating clients
• Sent about 500, 1M slices/files
from Chicago to Japan using parallel
TCP.
• It was also the application over which
we tested DiffServ.
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
Characteristics of Tele-Immersion Flows
BW
Latency
Sensitive
Jitter
Sensitive
Reliability
Reqrmnt
Burstiness
Packet
Size
Protocol
Avatar (virtual
representation)
Low
Y
Y
Low
Constant
Small
RTP / UDP
Audio
conference
Med
Y
Y
Low
Constant
Med
RTP / UDP
Video
conference
High
Y
Y
Low
Constant
Large
RTP / UDP
Realtime state
updates
Low
Y
Y
Med
Short
bursts
Small
FEC
Non-realtime
state updates
Low
N
N
High
App
dependent
Small
TCP
Small bulk data
(image files)
Med
N
N
High
Medium
bursts
Large
TCP/CPTCP
Large bulk data
(raw data sets)
High
N
N
High
Very long
bursts
Large
PTCP /
RBUDP
Streaming bulk
data
(high quality
audio/video/bit
map/polygons)
High
Y
N
High
Long
bursts
Large
SRBUDP /
AFEC
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
Motivation for Application-level
networking tools
• Bandwidth is becoming increasingly available.
• Networking QoS is still under research and difficult to
deploy and use. It is not as easy as “flipping a switch.”
• Network QoS is not only about bandwidth, it’s about latency
and jitter.
• Applications today are not ready to use the extra bandwidth
even if available.
• Application developers have to be increasingly network
savvy in order to be able to convert application
requirements to networking services.
• Need to make advanced networking easier for the average
application developer.
• Need to provide a higher level application framework to
keep pace with network advances.
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
Advanced Data Transport Techniques for
Tele-Immersion
• Maybe we can compensate for latency:
– Reliable Low-Latency Data Transfer for TeleImmersion
• Even if you had QoS could you really take
advantage of it?
– High Throughput Techniques for Tele-Immersion
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
Reliable Low-Latency Data Transfer for
Tele-Immersion
• In Tele-Immersion it is desirable to be able to
transmit state information with minimum
latency and jitter while preserving reliability
• Rather than use TCP which uses
acknowledgments to obtain reliability, try
UDP augmented with error correction codes:
Forward Error Correction
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
`
1-way latency in ms
Latency of transmitting 100 packets under
UDP, TCP, FEC/UDP with 3:1 redundancy from EVL (Chicago) to
SARA (Amsterdam)
UDP
400
TCP
350
FEC over UDP
300
250
FEC greatest benefit
is in small packets.
goal
200
Larger packets
impose greater
overhead.
150
100
As redundancy
decreases FEC
approaches UDP.
50
0
0
500
1000
1500
2000
2500
Packet size in bytes
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
Jitter for UDP, TCP and FEC over UDP
Moving average (over 20 successive data points)
of deviations of Short Term Latency (also over 20
successive data points)
UDP
TCP
14
FEC/UDP
12
Jitter
10
8
6
Goal
4
2
Electronic Visualization Laboratory (EVL)
77
73
69
65
61
57
53
49
45
41
37
33
29
25
21
17
13
9
5
1
0
University of Illinois at Chicago
Packet Loss over UDP vs FEC/UDP
between Chicago & Amsterdam
50Mbps UDP or FEC
+50Mbps UDP congestion
Packet Loss
UDP
1.90%
FEC
0.05%
UDP with congestion 17.40%
FEC with congestion 4.15%
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
High Throughput Techniques for TeleImmersion
• In Tele-Immersion it is desirable to share data files
as rapidly as possible
• Even if you had QoS, you couldn’t take advantage
of it
• Long Fat Network problem: an ftp session will max
out at 3.5Mbps over a 100Mbps link between
Chicago and Amsterdam (and Switzerland)
• 2 Techniques:
– Parallel TCP Socket Striping
– Reliable Blast UDP
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
Parallel Sockets : 64K Window Size
Amsterdam (SARA) to Chicago (EVL)
Plot of Average Achievable Bandwidth vs # of Paralle TCP Sockets Used to Deliver a 50M
File from Amsterdam to Chicago over 100Mbps Link
Bandwidth (Mbps)
60
50
40
30
20
10
0
Found it difficult to achieve more than
50Mbps on a 100Mbps link.
Have been able to achieve 80Mbps
on rare occasions.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
# of Sockets
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
Reliable Blast UDP (RBUDP)
• RBUDP - An old idea that may be useful now
that networking bandwidth is increasing
• Use UDP for bulk data transmission rather
than TCP
• If bandwidth can be guaranteed by QoS –
reliability will be high- chances of errors will
be few
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
RBUDP Performance
(EVL to SARA)
Average Thoughtput
(Mbps)
Average throughput vs transmission unit size
(size before an ACK is required) over 100Mbps link between EVL and SARA.
90
80
70
60
50
40
30
20
10
0
Realtime Throughput
1
2
3
4
6
8
Maximum UDP throughput
10 15 20 22 23 25 30 40 60 80 100 150 200 400
Unit Size (MBytes)
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
RBUDP Performance
(EVL to CERN)
Streaming RBUDP from CERN to EVL over
45Mbps link
Throughput (Mbps)
35
30
25
20
15
RUDP sending at 33Mbps
10
Maximum throughput achieved by UDP
(33Mbps)
5
0
15
0
10
75
50
40
30
20
15
10
8
6
4
3
2
1
0
Unit Size (Mbytes)
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
DiffServ results over a testbed between EVL and Argonne
National Lab
15
Bandwidth recovery good
10
5
163
154
145
136
127
118
109
91
82
73
64
55
46
37
Cisco7507
100
x
28
0
19
100Mbps
+ DiffServ
20
10
100Mbps
+ background
25
1
EVL
Bandw idth (Mbps)
DiffServ Bandwidth
Tim e (s)
163
154
145
136
127
118
109
100
91
82
73
64
55
46
37
28
19
0
42Mbps
Tim e (s)
back
DiffServ Packet Loss
161
153
145
137
129
121
113
105
97
89
81
73
65
57
Packet loss double
49
Electronic Visualization Laboratory (EVL)
1600
1400
1200
1000
800
600
400
200
0
41
100Mbps
33
x
Latency recovery not good
50
25
100Mbps
100
17
x
150
9
ANL
150ms 1-way latency
200
1
42Mbps
Packet Loss (packets/s)
fore
250
1
x
300
10
25Mbps
80Mbps
1 w ay Latency (m s)
DiffServ Latency
University of Illinois at Chicago
Tim e (s)
Collaborative Coordination Experiments between
Chicago and Singapore
•
•
•
•
Tightly coordinated collaborative interaction task between 2 remote
users
200ms RTT is the threshold where performance begins to suffer
200ms RTT with 0 jitter is same as 10ms RTT with 7ms jitter
So DiffServ is not suited for Tele-Immersion
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
• G2 is C++ toolkit for building Tele-Immersive applications with special
emphasis on networking
• G2 is the Grid’s main tele-immersion library
• Networking:
– UDP, TCP, Multicast, HTTP.
– UDP reflector and multicast bridge.
– TCP reflector.
– Remote procedure calls.
– 32 and 64bit Remote file I/O.
– Parallel 32 & 64 bit TCP socket striping for high throughput data
delivery.
– FEC.
– Client/Server distributed shared memory persistent database.
– Threading, Mutual Exclusion.
– Built-in Instrumentation of networking services.
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
• Tools for higher level application development:
–
–
–
–
–
–
–
Audio streaming
Articulated Avatars
VR navigation
VR menus
Speech recognition with IBM ViaVoice
Collaborative application shell to jumpstart development
Network visualization tools
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
QoS Internet Monitoring Tool
QoSIMoto
•
•
•
Electronic Visualization Laboratory (EVL)
Provides real time viewing of
CAVERNsoft data streams
Visualizes bandwidth,
latency, jitter of multiple
network flows
Accepts Netlogger
compatible format
University of Illinois at Chicago
These techniques are needed more than
ever in STAR LIGHT
• Being able to dedicate lambdas will ease our latency and
jitter problems but we still can’t beat the speed of light.
• Lambda switching requires more intelligence at the edges
to perform traffic shaping, bandwidth management, error
correction.
• Need to provide high level API for applications to select
Lambdas and define which application flow will go over the
lambda.
• Long Fat Network problems do not go away with GMPLS.
• Need to provide a higher level application programmer’s
framework to keep pace with network advances. We need
more than just sockets( ) API.
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
What new things can we do with
STAR LIGHT?
• Streaming uncompressed high resolution stereoscopic 3D
movies (1024x768x24bitsx30fps) ~ 1.05 Gbps
• Imagine magnifying this to multi-tiled displays for ultra high
resolution displays such as CAVEs and Active Murals
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
What new things can we do with
STAR LIGHT?
Distributed Tera-snap:
• Perform a tera-mining correlation between
distributed databases and generate a visual
overview
• 1 PC can absorb ~500Mbps => min 4.6
hours to perform a tera-snap
• 20 PCs can absorb data and produce image
composites in a min of 13 mins using
10Gbps
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
What new things can we do with
STAR LIGHT?
Digital Continuums: Distributed collaboratories with linked
tiled, tele-immersive and mobile displays enhanced with
cluster computing and data mining services
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago
Thanks
• Thanks SARA (Amsterdam), CERN
(Switzerland), IHPC (Singapore) for
graciously participating in these network
experiments
• For more info:
– www.evl.uic.edu/cavern
– [email protected]
Electronic Visualization Laboratory (EVL)
University of Illinois at Chicago