Transcript IP over InfiniBand: Connected Mode

IP over InfiniBand: Connected Mode
Vivek Kashyap
[email protected]
© 2002 IBM Corporation
IETF-55: IPoIB Connected Mode
IP over InfiniBand Modes
 Unreliable Datagram
AD review comments
IBTA 1.1 updates
 Reliable Connected
 Unreliable Connected
 Reliable Datagram
2
Vivek Kashyap, IBM
IETF-55: IPoIB Connected Mode
IPoIB non-UD options
 RC
 RD
Large MTU Chart Title 2^31 bytes
 2-4K links
 2^31 bytes
 2-4K links
 2^31 bytes
 2-4K links
Automatic Path
Migration
 Yes
 Yes
 Yes
Reliability
 Yes
 No
 Message lost
 Yes
RDMA
 Yes
 RDMA Write
 Yes
No. of
messages
outstanding
3
 UC
 One
Vivek Kashyap, IBM
IPoIB over Connected Mode
Reliable Connected
Unreliable Connected
© 2002 IBM Corporation
IETF-55: IPoIB Connected Mode
IPoIB Subnet
 IPoIB-Connected subnet is formed over a Non-Broadcast Multiple
Access network
IPoIB-UD is over an abstract broadcast link
Node A A
IPa
IPp
IPs
Node D
IPb
IPd
IPq
Node B
IPr IPc
Node C
IB switched network
IPoIB UD broadcast domain
IPoIB Connected network
5
Vivek Kashyap, IBM
IETF-55: IPoIB Connected Mode
Connection Setup: Address Resolution
 The Connection can be setup only when the peer’s GID is known
– Manually configured (or Permanent Virtual Circuit)
• Inverse ARP
– Dynamically resolved
• ARP Server as in Classical IP over ATM (RFC2225)
or
•
IPoIB address resolution
– Use a UD QP
• may reuse IPoIB-UD QP
– Define ARP server link address same as IPoIB-UD Broadcast GID
• FF12:<401B or 601B>:<P_Key>::< 255.255.255.255 or 1>
6
Vivek Kashyap, IBM
IETF-55: IPoIB Connected Mode
Connection Setup: Option 1
 SendFrom IPb to IPa
IPb creates ARP/ND packet with local hardware address of :
[RC|UC]:FFFFFF:GIDb
A bit indicates RC or UC to let the peer know of the transmitter’s view of
the subnet.
IPa receives the packet
Determine PathRecord to GIDb
Send reply over UD-QP with GIDa
IPb now creates the IB connection to IPa
Service ID of:
1:Protocol or FF:0:0:0:0:0:0
– Protocol indicates IPv4 or IPv6
– FF indicates both IPv4 and IPv6 packets will use this connection
– Private Data includes the IP address that was resolved
7
Vivek Kashyap, IBM
IETF-55: IPoIB Connected Mode
Address Resolution: Option 2
 SendFrom IPb to IPa
IPb creates ARP/ND packet with local hardware address of :
[RC|UC]:FFFFFF:GIDb
A bit indicates RC or UC to let the peer know of the transmitter’s view of
the subnet.
IPa receives the packet
Determine PathRecord to GIDb (same as for IPoIB-UD)
Creates QPc of type RC/UC as indicated
– may reuse existing QP
Send reply with QPc and GIDa
IPb now creates the IB connection to the peer
Service ID of:
1:QPc:0:0:0:0:0:0
IPa updates ARP/ND entry with QP value received in IB connection
8
Vivek Kashyap, IBM
IETF-55: IPoIB Connected Mode
Encapsulation
 Connection can carry IPv4 or IPv6 or both packets. Therefore
– Type of packet is known
– Type of packet is distinguishable from the IP header
 Two options:
– IP packets directly encapsulated in the IB connection mode payload
– IP packets encapsulated with the ‘Ethertype’ indication as in IPoIB-UD
 Address resolution over UD QP needs to follow IPoIB-UD
encapsulation
9
Vivek Kashyap, IBM
IETF-55: IPoIB Connected Mode
MTU
 MTU Negotiated per IB connection
Value requested sent in private data of CM REQ message to peer B
If not acceptable counter value in CM REP message to peer A
If not acceptable REJ with desired value in private data to peer B
10
Vivek Kashyap, IBM
IPoIB Connected Mode
Vivek Kashyap
[email protected]
© 2002 IBM Corporation