Transcript Network speed tests - Indico

Network speed tests
CERN 14-dec-2010
Hardware used
✤
2 identical servers:
✤
Dell PowerEdge T610 with 2x Intel xeon X5650, 24GB ram
✤
2x PCI-Ex x8, 2x PCI-Ex x4
✤
2x10Gb on copper cards, Broadcom chip (NetXtreme II
57710), PCI-Ex x8
✤
1 switch Dell PowerConnect 6224:
✤
24 Gb ports plus 2x10Gbase-T ports module
IPerf test
✤
tcp benchmark, bi-directional
✤
do not really expect to use tcp, but still a good hw test
✤
very similar results using one-way or bi-directional transfers
✤
identical results with or without passing through the switch
IPerf results
[5] 9.0-10.0 sec 1.09 GB 9.37 Gbits/sec [5] 0.0-10.0 sec 10.8 GB 9.31 Gbits/sec (one direction)
[4] 9.0-10.0 sec 1.09 GB 9.39 Gbits/sec [4] 0.0-10.0 sec 10.7 GB 9.21 Gbits/sec (other direction)
~9.2 - 9.3 Gb/s
with switch
~9.3 - 9.4 Gb/s
without
UDP results
HPCBENCH tool
# UDP communication test -- Mon Dec 13 23:14:31 2010
# Fixed packet size unidirectional stream test
# Hosts: gtktest1 (client) <--> 192.168.0.10 (server)
# Client UDP socket buffer size (Bytes) -- SNDBUF: 65508 RCVBUF: 65508
# Server UDP socket buffer size (Bytes) -- SNDBUF: 65508 RCVBUF: 65508
# Client IP TOS type: Default
# Server IP TOS type: Default
# UDP datagram (packet) size (Bytes) -- Client: 1460 Server: 1460
# Data size of each read/write (Bytes) -- Client: 1460 Server: 1460
# Message size (Bytes): 16384
# Test time (Second): 10.000000
# Test repeat: 1
# Network(Mbps) Local(Mbps) SentPkg(C) SentByte(C) RecvPkg(S) RecvByte(S) LostPkg LossRate
1
5744.7525 5880.3193 5034526 7350406532 4918451 7180937032 116075 0.023
- Essentially same (disappointing) performance with
- HPCBENCH tool
- custom (very basic) client-server python application
UDP vs. TCP on PCI-Ex 4x slot
Mbit/s
- Due to wrong mounting (by Dell…), tests using the 10 GB cards in the
PCI-Ex 4x slots (in place of 8x)
- Degraded performance due to reduced PCI-Ex bandwidth
TCP
UDP
Bandwidth (average/min./max.) vs. Message size
Bandwidth vs. Message size
7000
7000
6000
6000
5000
5000
4000
4000
3000
3000
2000
2000
1000
1000
0
0
Bytes
UDP strange results
UDP performance is the same, 5.3 - 5.7 Gbits/sec (slightly depending on
packet size), in both configurations (“wrong”: cards in the PCI-Ex 4x slots,
and “correct”: cards in the PCI-Ex 8x slots)
We’re hitting some kind of wall:
-Collisions?
-Lost packets?
-Optimization of frame-buffer?
Lost packets per mille
40
35
30
25
20
15
10
5
0
More investigations needed!
UDT (UDX)
✤
Just started investigating !!
✤
UDP based application level data transfer protocol
✤
adds own reliability control and congestion control
✤
highly configurable, still looking at the possibilities
➡not
claiming we will use this protocol, but if we want reliability
control udp is a candidate
Still to do
✤
Lots of work; just to mention:
✤
understand if we need reliability control
✤
test behaviour with multiple 1Gb senders/one 10Gb receiver
✤
avoid memory-to-memory copy
✤
etc etc