Transcript SC06_sdp_update

OpenFabrics
Developers Summit SC06
SDP Update
Michael Tsirkin, Eitan Zahavi,
Mellanox Technologies
Nov 2006
OFED 1.1 SDP Perf’ Status
• ~10us latency
• ~960MBs/DDR 64k Msg
– BW CPU bound and goes down linearly with Msg size
• Nagle in OFED does not help as ACKs arrive too fast
– Nagle = When to send = LEN>MSS or all packets were ACKed.
– TCP Cork: is a socket option that cancels the second criteria such that
only len>MSS counts
2/7/06
Mellanox Technologies – Developers Summit @ SC06
Page 2
OFED 1.1 SDP Perf’ Status
• Improve small Msg BW by new module parameter send_poll=0
– Provides 300MB/sec for ½KB Msg
– Delaying the send completion by waiting for interrupts
(stops polling on sender – thus send completion late)
2/7/06
Mellanox Technologies – Developers Summit @ SC06
Page 3
SDP Future Work
• Memory Scalability
– Define an adaptive algorithm
• When to tighten memory allocation per connection
– New posted buffers reclaim protocol
• Allow one side to reclaim posted buffers
• Possibly require other side to NOP ACK
2/7/06
Mellanox Technologies – Developers Summit @ SC06
Page 4
SDP Future Work (Cont)
• Small Message BW
– Merge messages waiting on send queue
• Send-page
– Used by send-file API
– Maps to Send-Side-Zero-Copy
• Netstat
–
–
–
–
2/7/06
Provide info on active connections
Track active connections, user, src/dst IP and ports in /proc
Require migrating AF_INET_SDP to PROTO_SDP
Provide as netstat patch
Mellanox Technologies – Developers Summit @ SC06
Page 5
Other topics
•
ZCopy
– Mellanox published Gen1 based ZCopy implementation
– SDP protocol limits BW improvements to >4K messages
– Development delayed as low ROI (complex implementation)
• AIO
– Seems like Linux implementation is under retrofit
– See mail threads: “kevent: network AIO, socket notifications”
– Debate and proposal for new API on OLS 2006
• CMA work
– Ongoing: APM
– Support IPv6
2/7/06
Mellanox Technologies – Developers Summit @ SC06
Page 6