wdiff rfc7609.original rfc7609.txt

TCPM working group
Independent Submission M. Fox
Internet Draft
Request for Comments: 7609 C. Kassimis
Intended Status:
Category: Informational J. Stevens
Expires: 11/7/2015
ISSN: 2070-1721 IBM
May 7,
August 2015

IBM's Shared Memory Communications over RDMA
draft-fox-tcpm-shared-memory-rdma-07.txt

Status of this Memo (SMC-R) Protocol

Abstract

This Internet-Draft is submitted document describes IBM's Shared Memory Communications over RDMA
(SMC-R) protocol. This protocol provides Remote Direct Memory Access
(RDMA) communications to TCP endpoints in full conformance with the
provisions a manner that is
transparent to socket applications. It further provides for dynamic
discovery of BCP 78 partner RDMA capabilities and BCP 79.

Internet-Drafts dynamic setup of RDMA
connections, as well as transparent high availability and load
balancing when redundant RDMA network paths are working documents available. It
maintains many of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note traditional TCP/IP qualities of service such as
filtering that
other groups may also distribute working documents enterprise users demand, as Internet-
Drafts.

Internet-Drafts are draft documents valid well as TCP socket
semantics such as urgent data.

Status of This Memo

This document is not an Internet Standards Track specification; it is
published for informational purposes.

This is a maximum contribution to the RFC Series, independently of six months any other
RFC stream. The RFC Editor has chosen to publish this document at
its discretion and may be updated, replaced, makes no statement about its value for
implementation or obsoleted deployment. Documents approved for publication by other documents at
the RFC Editor are not a candidate for any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."

The list level of Internet
Standard; see Section 2 of RFC 5741.

Information about the current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt

The list status of Internet-Draft Shadow Directories can this document, any errata,
and how to provide feedback on it may be accessed obtained at
http://www.ietf.org/shadow.html

This Internet-Draft will expire on November 7, 2015.
http://www.rfc-editor.org/info/rfc7609.

This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.

Abstract

This document describes the IBM's Shared Memory Communications over
RDMA (SMC-R) protocol. This protocol provides RDMA communications to
TCP endpoints in a manner that is transparent to socket applications.
It further provides for dynamic discovery of partner RDMA
capabilities and dynamic setup of RDMA connections, transparent high
availability and load balancing when redundant RDMA network paths are
available, and it maintains many of the traditional TCP/IP qualities
of service such as filtering that enterprise users demand, as well as
TCP socket semantics such as urgent data.

Table of Contents

1. Introduction...................................................5 Introduction ....................................................5
1.1. Summary of changes in this draft..........................6
1.2. Protocol overview.........................................6
1.2.1. Overview ..........................................6
1.1.1. Hardware requirements................................8
1.3. Requirements ...............................8
1.2. Definition of common terms................................8 Common Terms .................................8
1.3. Conventions Used in This Document .........................11
2. Link Architecture.............................................10 Architecture ..............................................11
2.1. Remote Memory Buffers (RMBs).............................12 (RMBs) ..............................12
2.2. SMC-R Link groups........................................16 Groups .........................................18
2.2.1. Link group types....................................17 Group Types ...................................18
2.2.2. Maximum number Number of links Links in link group...............20 Link Group ..............21
2.2.3. Forming and managing link groups....................21 Managing Link Groups ...................23
2.2.4. SMC-R link identifiers..............................22 Link Identifiers .............................24
2.3. SMC-R resilience Resilience and load balancing......................23 Load Balancing .......................24
3. SMC-R Rendezvous architecture.................................25 Architecture ..................................26
3.1. TCP options..............................................25 Options ...............................................26
3.2. Connection Layer Control (CLC) messages..................26 Messages ...................27
3.3. LLC messages.............................................26 Messages ..............................................27
3.4. CDC Messages.............................................28 Messages ..............................................29
3.5. Rendezvous flows.........................................28 Flows ..........................................29
3.5.1. First contact.......................................28 Contact ......................................29
3.5.1.1. Pre-negotiation of TCP Options pre-negotiation....................28 ............29
3.5.1.2. Client Proposal................................29 Proposal ...........................30
3.5.1.3. Server acceptance..............................30 Acceptance .........................32
3.5.1.4. Client confirmation............................31 Confirmation .......................32
3.5.1.5. Link (QP) confirmation.........................31 Confirmation ....................32
3.5.1.6. Second SMC-R link setup........................34 Link Setup ...................35
3.5.1.6.1. Client processing Processing of "Add Link" ADD LINK
LLC message Message from server..........................................34 Server ........35
3.5.1.6.2. Server processing Processing of "Add Link" reply ADD LINK
Reply LLC
message Message from the client..............................35 Client ..36
3.5.1.6.3. Exchange of Rkeys RKeys on second
Second SMC-R link....37 Link ..............38
3.5.1.6.4. Aborting SMC-R and falling back
Falling Back to IP.....37 IP .............38
3.5.2. Subsequent contact..................................37 Contact .................................38
3.5.2.1. SMC-R proposal.................................38 Proposal ............................39
3.5.2.2. SMC-R acceptance...............................39 Acceptance ..........................40
3.5.2.3. SMC-R confirmation.............................40 Confirmation ........................41
3.5.2.4. TCP data flow race Data Flow Race with SMC
Confirm CLC message40 Message .......................41
3.5.3. First contact variation: creating Contact Variation: Creating a parallel link group
...........................................................41
Parallel Link Group ................................42
3.5.4. Normal SMC-R link termination.......................42 Link Termination ......................43
3.5.5. Link group management flows.........................43 Group Management Flows ........................44
3.5.5.1. Adding and deleting links Deleting Links in an
SMC-R link group43
3.5.5.1.1. Server initiated Add Link processing......43 Group ..........................44
3.5.5.1.1. Server-Initiated ADD
LINK Processing ................45
3.5.5.1.2. Client initiated Add Link processing......44 Client-Initiated ADD
LINK Processing ................45
3.5.5.1.3. Server initiated Delete Link Processing...44 Server-Initiated DELETE
LINK Processing ................46
3.5.5.1.4. Client initiated Delete Link request......46 Client-Initiated DELETE
LINK Request ...................48
3.5.5.2. Managing multiple Rkeys Multiple RKeys over multiple
Multiple SMC-R links Links in a link group.........................................48 Link Group ......49
3.5.5.2.1. Adding a new New RMB to an
SMC-R link group...49 Link Group ...............50
3.5.5.2.2. Deleting an RMB from an
SMC-R link group..52 Link Group ...............53
3.5.5.2.3. Adding a new New SMC-R link Link to a link group
Link Group with
multiple RMBs........................................53 Multiple RMBs ..54
3.5.5.3. Serialization of LLC exchanges, Exchanges,
and collisions.54 Collisions ............................56
3.5.5.3.1. Collisions with ADD
LINK / CONFIRM LINK
exchange.............................................56 Exchange ...57
3.5.5.3.2. Collisions during
DELETE LINK exchange....57 Exchange ...........58
3.5.5.3.3. Collisions during CONFIRM_RKEY exchange...57
CONFIRM RKEY Exchange ..........59
4. SMC-R memory sharing architecture.............................59 Memory-Sharing Architecture ..............................60
4.1. RMB element allocation considerations....................59 Element Allocation Considerations .....................60
4.2. RMB and RMBE format......................................59 Format .......................................60
4.3. RMBE control information.................................59 Control Information ..................................60
4.4. Use of RMBEs.............................................60 RMBEs ..............................................61
4.4.1. Initializing and accessing RMBEs....................60 Accessing RMBEs ...................61
4.4.2. RMB element reuse Element Reuse and conflict resolution...........61 Conflict Resolution ..........62
4.5. SMC-R protocol considerations............................62 Protocol Considerations .............................63
4.5.1. SMC-R protocol optimized window size updates........62 Protocol Optimized Window Size Updates .......63
4.5.2. Small data sends....................................63 Data Sends ...................................64
4.5.3. TCP Keepalive processing............................63 Processing ...........................65
4.6. TCP connection failover Connection Failover between SMC-R links..............66 Links ...............67
4.6.1. Validating data integrity...........................66 Data Integrity ..........................67
4.6.2. Resuming the TCP connection Connection on a new SMCR link......67 New SMC-R Link ....68
4.7. RMB data flows...........................................67 Data Flows ............................................69
4.7.1. Scenario 1: Send flow, window size unconstrained....68 Flow, Window Size Unconstrained ...69
4.7.2. Scenario 2: Send/Receive flow, window unconstrained.70 Flow, Window Size
Unconstrained ......................................71
4.7.3. Scenario 3: Send Flow, window constrained...........71 Window Size Constrained .....72
4.7.4. Scenario 4: Large send, flow control, full window size
writes.....................................................73 Send, Flow Control, Full
Window Size Writes .................................74
4.7.5. Scenario 5: Send flow, urgent data, window size
unconstrained..............................................76 Flow, Urgent Data, Window
Size Unconstrained .................................77
4.7.6. Scenario 6: Send flow, urgent data, window size closed78 Flow, Urgent Data, Window
Size Closed ........................................79
4.8. Connection termination...................................80 Termination ....................................81
4.8.1. Normal SMC-R connection termination flows...........80
4.8.1.1. Connection Termination Flows ..........81
4.8.2. Abnormal SMC-R connection termination flows....85
4.8.1.2. Connection Termination Flows ........86
4.8.3. Other SMC-R connection termination conditions..87 Connection Termination Conditions ......88
5. Security considerations.......................................88 Considerations ........................................89
5.1. VLAN considerations......................................88 Considerations .......................................89
5.2. Firewall considerations..................................88 Considerations ...................................89
5.3. Host-based Host-Based IP Filters....................................89 Filters .....................................89
5.4. Intrusion Detection Services.............................89 Services ..............................90
5.5. IP Security (IPSec)......................................89 (IPsec) .......................................90
5.6. TLS/SSL..................................................89 TLS/SSL ...................................................90
6. IANA considerations...........................................89 Considerations ............................................90
7. References....................................................90
7.1. Normative References.....................................90
7.2. Informative References...................................90
8. Acknowledgments...............................................90
9. Conventions used in this document.............................90 References ...........................................91
Appendix A. Formats..............................................91 Formats ...............................................92
A.1. TCP option...............................................91 Option .................................................92
A.2. CLC messages.............................................91 Messages ...............................................92
A.2.1. Peer ID format......................................91 Format ......................................93
A.2.2. SMC Proposal CLC message format.....................93 Message Format .....................94
A.2.3. SMC Accept CLC message format.......................96 Message Format .......................98
A.2.4. SMC Confirm CLC message format......................99 Message Format .....................102
A.2.5. SMC Decline CLC message format.....................102 Message Format .....................105
A.3. LLC messages............................................103 Messages ..............................................106
A.3.1. CONFIRM LINK LLC message format....................104 Message Format ....................107
A.3.2. ADD LINK LLC message format........................106 Message Format ........................109
A.3.3. ADD LINK CONTINUATION LLC message format...........108 Message Format ...........112
A.3.4. DELETE LINK LLC message format.....................111 Message Format .....................115
A.3.5. CONFIRM RKEY LLC message format....................113 Message Format ....................117
A.3.6. CONFIRM RKEY CONTINUATION LLC message format.......116 Message Format .......120
A.3.7. DELETE RKEY LLC message format.....................118 Message Format .....................122
A.3.8. TEST LINK LLC message format.......................120 Message Format .......................124
A.4. Connection Data Control (CDC) Message Format ..............125
Appendix B. Socket API considerations...........................126 Considerations ............................129
B.1. setsockopt() / getsockopt() Considerations ................130
Appendix C. Rendezvous Error scenarios..........................128 Scenarios ...........................131
C.1. SMC Decline during CLC negotiation......................128 Negotiation ........................131
C.2. SMC Decline during LLC negotiation......................128 Negotiation ........................131
C.3. The SMC Decline window..................................130 Window ....................................133
C.4. Out of synch conditions Out-of-Sync Conditions during SMC-R negotiation........130 Negotiation ...........133
C.5. Timeouts during CLC negotiation.........................131 Negotiation ...........................134
C.6. Protocol errors Errors during CLC negotiation..................131 Negotiation ....................134
C.7. Timeouts during LLC negotiation.........................132 Negotiation ...........................135
C.7.1. Recovery actions Actions for LLC timeouts Timeouts and failures.....133 Failures .....136
C.8. Failure to add second Add Second SMC-R link Link to a link group........140 Link Group ..........142
Authors' Addresses ...............................................143

1. Introduction

This document specificies specifies IBM's Shared Memory Communications over RDMA
(SMC-R) protocol. SMC-R is a protocol for Remote Direct Memory
Access (RDMA) communication between TCP socket endpoints. SMC-R runs
over networks that support RDMA over Converged Ethernet (ROCE). (RoCE). It
is designed to permit existing TCP applications to benefit from RDMA
without requiring modifications to the applications or predefinition
of RDMA partners.

SMC-R provides dynamic discovery of the RDMA capabilities of TCP
peers and automatic setup of RDMA connections that those peers can
use. SMC-R also provides transparent high availability and load
balancing
load-balancing capabilities that are demanded by enterprise
installations but are missing from current RDMA protocols. If
redundant RoCE
capable RoCE-capable hardware such as RDMA NICs (RNICs)and RoCE capable RDMA-capable Network
Interface Cards (RNICs) and RoCE-capable switches is present, SMC-R
can load balance load-balance over that redundant hardware and can also
non-disruptively move TCP traffic from failed paths to surviving
paths, all seamlessly to the application and the sockets layer.
Because SMC-R preserves socket semantics and the TCP three-way
handshake, many TCP qualities of service such as filtering, load
balancing, and SSL Secure Socket Layer (SSL) encryption are preserved, as
are TCP features such as urgent data.

Because of the dynamic discovery and setup of SMC-R connectivity
between peers, no RDMA connection manager (RDMA-CM) is required.
This also means that support for UD queue pairs Unreliable Datagram (UD) Queue Pairs
(QPs) is also not required.

It is recommended that the SMC-R services be implemented in kernel
space, which enables optimizations such as resource sharing resource-sharing between
connections across multiple processes and also permits applications
using SMC-R to spawn multiple processes (e.g. (e.g., fork) without losing
SMC-R functionality. A user space user-space implementation is compatible with
this architecture, but it may not support spawned processes (i.e.
fork) (e.g.,
fork), which limits sharing and resource optimization to TCP
connections that originate from the same process. This might be an
appropriate design choice if the use case is a system that hosts a
large single process application that creates many TCP connections to
a peer host, or in implementations where a kernel space kernel-space
implementation is not possible or introduces excessive overhead for
kernel
"kernel space to user space space" context switches.

1.1. Summary of changes in this draft

Changed the title to add "IBM's".

1.2. Protocol overview Overview

SMC-R defines the concept of the SMC-R Link, link, which is a logical
point-to-point link using reliably connected queue pairs between
TCP/IP stack peers over a RoCE fabric. An SMC-R link is bound to a
specific hardware path, meaning a specific RNIC on each peer. SMC-R
links are created and maintained by an SMC-R layer, which may reside
in kernel space or user space space, depending upon operating system and
implementation requirements. The SMC-R layer resides below the
sockets layer and directs data traffic for TCP connections between
connected peers over the RoCE fabric using RDMA rather than over a
TCP connection. The TCP/IP stack stack, with its requirements for
fragmentation, packetization, etc. requirements etc., is bypassed bypassed, and the application
data is moved between peers using RDMA.

Multiple SMC-R links between the same two TCP/IP stack peers are also
supported. A set of SMC-R links called a link group can be logically
bonded together to provide redundant connectivity. If there is
redundant hardware, hardware -- for example example, two RNICs on each peer, peer -- separate SMC-
R
SMC-R links are created between the peers to exploit that redundant
hardware. The link group architecture with redundant links provide provides
load balancing, balancing and increased bandwidth bandwidth, as well as seamless failover.

Each SMC-R link group is associated with an area of memory called
Remote Memory Buffers (RMBs), which are areas of memory that are
available for SMC-R peers to write into using RDMA writes. Multiple
TCP connections between peers may be multiplexed over a single SMC-R
link, in which case the SMC-R layer manages the partitioning of the
RMBs between the TCP connections. This multiplexing reduces the RDMA
resources
resources, such as queue pairs QPs and RMBs RMBs, that are required to support
multiple connections between peers, and it also reduces the
processing and delays related to setting up queue pairs, QPs, pinning memory, and
other RDMA setup tasks when new TCP connections are created. In a
kernel space
kernel-space SMC-R implementation in which the RMBs reside in kernel
storage, this sharing and optimization works across multiple
processes executing on the same host. In a user space user-space SMC-R
implementation in which the RMBs reside in user space, this sharing
and optimization is limited to multiple TCP connections created by a
single process, as separate RMBs and QPs will be required for each
process.

SMC-R also introduces a rendezvous protocol that is used to
dynamically discover the RDMA capabilities of TCP connection partners
and exchange credentials necessary to exploit that capability if
present. TCP connections are set up using the normal TCP 3-way
handshake, three-way
handshake [RFC793], with the addition of a new TCP option that
indicates SMC-R capability. If both partners indicate SMC-R capability
capability, then at the completion of the 3-way three-way TCP handshake the
SMC-R layers in each peer take control of the TCP connection and use
it to exchange additional
connection level control Connection Layer Control (CLC) messages to
negotiate SMC-R credentials such as queue pair (QP) information, QP information; addressability
over the RoCE fabric, fabric; RMB buffer sizes, sizes; and keys and addresses for
accessing RMBs over RDMA, etc. RDMA. If at any time during this negotiation a
failure or decline occurs, the TCP connection falls back to using the
IP fabric.

If the SMC-R negotiation succeeds and either a new SMC-R link is set
up or an existing SMC-R link is chosen for the TCP connection, then
the SMC-R layers open the sockets to the applications and the
applications use the sockets as normal. The SMC-R layer intercepts
the socket reads and writes and moves the TCP connection data over
the SMC-R link, "out of band" to the TCP connection connection, which remains
open and idle over the IP fabric, except for termination flows and
possible keepalive flows. Regular TCP sequence numbering methods are
used for the TCP flows that do occur; data flowing over RDMA does not
use or affect TCP sequence numbers.

This architecture does not support fallback of active SMC-R
connections to IP. Once connection data has completed the switch to
RDMA, a TCP connection cannot be switched back to IP and will reset
if RDMA becomes unusable.

The SMC-R protocol defines the format of the Remote Memory Buffers RMBs that are used to
receive TCP connection data written over RDMA, as well as the
semantics for managing and writing to these buffers using Connection
Data Control (CDC) messages.

Finally, SMC-R defines link level control Link Layer Control (LLC) messages that are
exchanged over the RoCE fabric between peer SMC-R layers to manage
the SMC-R links and link groups. These include messages to test and
confirm connectivity over an SMC-R link, add and delete SMC-R links
to or from the link group, and exchange RMB addressability
information.

1.2.1.

1.1.1. Hardware requirements Requirements

SMC-R does not require full Converged Enhanced Ethernet switch
functionality. SMC-R functions over standard Ethernet fabrics fabrics,
provided that endpoint RNICs are provided and IEEE 802.3x Global
Pause Frame is supported and enabled in the switch fabric.

While SMC-R as specified in this document is designed to operate over
RoCE fabrics, adjustments to the rendezvous methods could enable it
to run over other RDMA fabrics fabrics, such as Infiniband InfiniBand [RoCE] and iWARP.

1.3.

1.2. Definition of common terms Common Terms

This section provides definitions of terms that have a specific
meaning to the SMC-R protocol and are used throughout this document.

SMC-R link Link

An SMC-R Link link is a logical point to point point-to-point connection over the RoCE
fabric via specific physical adapters (MAC/GID). (Media Access Control /
Global Identifier (MAC/GID)). The Link link is formed during the first contact
"first contact" sequence of the TCP/IP 3 way three-way handshake
sequence that occurs over the IP fabric. During this
handshake handshake,
an RDMA RC-QP reliably connected queue pair (RC-QP) connection is formed
between the two peer SMC hosts and is defined as the SMC Link. SMC-R link.
The SMC Link SMC-R link can then support multiple TCP connections between
the two peers. An SMC SMC-R link is associated with a single LAN (or
VLAN) segment and is not routable.

SMC-R link group

An SMC-R Link Group

An SMC-R link group is a group of SMC-R Links typically each over
unique RoCE adapters links between the same two
SMC-R peers. peers, typically with each link over unique RoCE adapters.
Each link in the link group has equal characteristics characteristics, such as the
same VLAN ID (if VLANs are in use), access to the same RMB(s) RMB(s), and
access to the same TCP server / client server/client.

SMC-R peer Peer

The SMC-R Peer peer is the peer software stack within the peer
Operating System
operating system with respect to the Shared Memory Communications
(messaging) protocol.

SMC-R Rendezvous

The

SMC-R Rendezvous is the SMC-R peer discovery and handshake
sequence that occurs transparently over the IP (Ethernet) fabric
during and immediately after the TCP connection 3 way three-way
handshake by exchanging the SMC SMC-R capabilities and credentials
using experimental TCP option and CLC messages.

RoCE SendMsg

RoCE SendMsg is a send operation posted to a reliably connected
queue pair with inline data, for the purpose of transferring
control information between peers.

TCP Client

The TCP client is the TCP socket-based peer that initiates a TCP connection
connection.

TCP Server

The TCP server is the TCP socket-based peer that accepts a TCP connection
connection.

CLC messages Messages

The SMC-R protocol defines a set of Connection Layer Control
Messages
messages that flow over the TCP connection that are used to manage SMC
SMC-R link rendezvous at TCP connection setup time. This
mechanism is analogous to SSL setup messages messages.

LLC Commands

The SMC-R protocol defines a set of RoCE Link Layer Control
Commands
commands that flow over the RoCE fabric using RDMA sendmsg, RoCE SendMsg, that
are used to manage SMC Links, SMC Link Groups SMC-R links, SMC-R link groups, and SMC Link Group SMC-R
link group RMB expansion and contraction.

CDC message Message

The SMC-R protocol defines a Connection Data Control message that
flows over the RoCE fabric using RDMA sendmsg RoCE SendMsg that is used to
manage the SMC-R connection data. This message provides
information about data being transferred over the out of band out-of-band RDMA
connection, such as data cursors, sequence numbers, and data flags
(for example example, urgent data). The receipt of this message also
provides an interrupt to inform the receiver that it has received
RDMA data.

RMB

A Remote (RDMA) Memory Buffer is a fixed or pinned buffer
allocated in each of the peer hosts for a TCP (via SMC-R)
connection. The RMB is registered to the RNIC and allows remote
access by the remote peer using RDMA semantics. Each host is
passed the peer's RMB specific RMB-specific access information (RKey (RMB Key (RKey)
and RMB
Element element offset) during the SMC-R rendezvous Rendezvous process. The
host stores socket application user data directly into the peer's
RMB using RDMA over RoCE.

Rtoken

RToken

The RToken is the combination of an RMB's Rkey RKey and RDMA virtual addressing, an
Rtoken
address. An RToken provides addressability to an RMB addressability information to an
RDMA peer peer.

RMBE

The Remote Memory Buffer Element (RMBE) is an area of an RMB that
is allocated to a specific TCP connection. The RMBE contains data
for the TCP connection. The RMBE represents the TCP receive
buffer
buffer, whereby the remote peer writes into the RMBE and the local
peer reads from the local RMBE. The alert token resolves to a
specific RMBE.

Alert Token

The SMC-R alert token is a four byte 4-byte value that uniquely identifies
the TCP connection over an SMC-R connection. The alert token
allows the SMC peer to quickly identify the target TCP connection
that now has new work. The format of the token is defined by the
owning SMC-R end point endpoint and is considered opaque to the remote peer. However
However, the token should not simply be an index to an RMBE element; RMBE; it
should reference a TCP connection and be able to be validated to
avoid reading data from stale connections.

RNIC

The RDMA capable RDMA-capable Network Interface Card (RNIC) is an Ethernet NIC
that supports RDMA semantics and verbs using RoCE.

First Contact

Describes

"First contact" describes an SMC-R negotiation to set up the first
link in a link
group group.

Subsequent Contact

Describes

"Subsequent contact" describes an SMC-R negotiation between peers
who are using an
already existing already-existing SMC-R link group group.

1.3. Conventions Used in This Document

In the rendezvous flow diagrams, dashed lines (----) are used to
indicate flows over the TCP/IP fabric and dotted lines (....) are
used to indicate flows over the RoCE fabric.

2. Link Architecture

An SMC-R link is based on reliably connected queue pairs (QPs) that
form a "logical point to point point-to-point link" between the two SMC-R peers over
a RoCE fabric. An SMC-R link extends from SMC-R peer to SMC-R peer,
where typically each peer would be a TCP/IP stack and would reside on
separate hosts.

,,.--..,_
+----+ _-`` `-, +-----+
|QP 8| - RoCE ', |QP 64|
| | / VLAN M . | |
+----+--------+/ \+-------+-----+
| RNIC 1 | SMC-R Link | RNIC 2 |
| |<--------------------->| |
+------------+ , /+------------+
MAC A (GID A) MAC B (GID B)
. .`
`', ,-`
``''--''``

Figure 1 1: SMC-R Link Overview
Figure 1 illustrates an overview of the basic concepts of SMC-R peer
to peer connectivity which peer-
to-peer connectivity; this is called the SMC-R Link. link. The SMC-R Link link
forms a logical point to point point-to-point connection between two SMC-R peers via
RoCE. The SMC Link SMC-R link is defined and identified by the following
attributes:

SMC-R Link link = RC QPs
(source VMAC GID QP + target VMAC GID QP + VLAN ID)

The SMC-R Link link can optionally be associated with a VLAN ID. If VLANs
are in use for the associated IP (LAN) connection connection, then the VLAN
attribute is carried over on the SMC-R link. When VLANs are in use use,
each SMC-R link group is associated with a single and specific VLAN.
The RoCE fabric is the same physical Ethernet LAN used for standard
TCP/IP over Ethernet
TCP/IP-over-Ethernet communications, with switches as described in
1.2.1.
Section 1.1.1.

An SMC-R Link link is designed to support multiple TCP connections between
the same two peers. An SMC Link SMC-R link is intended to be long lived lived,
while the underlying TCP connections can dynamically come and go.
The associated RMBs can also be dynamically added and removed from
the link as needed. The first TCP connection between the peers
establishes the SMC-R link. Subsequent TCP connections then use the
previously established link. When the last TCP connection terminates
terminates, the link can then be terminated, typically after an implementation
defined
implementation-defined idle time-out timeout period has elapsed. The TCP
server is responsible for initiating and terminating the SMC Link. SMC-R link.

2.1. Remote Memory Buffers (RMBs)

Figure 2 shows the hosts -- Hosts X and Y -- and their associated
RMBs within each host. With the SMC-R link link, and the associated RMB keys (Rkeys)and RKeys
and RDMA virtual addresses addresses, each SMC-R enabled SMC-R-enabled TCP/IP stack can
remotely access its peer's RMBs using RDMA. The RKeys and virtual
addresses are exchanged during the rendezvous processing when the
link is established. The combination of the Rkey RKey and the virtual
address is the Rtoken. RToken. Note that the SMC-R Link link ends at the QP
providing access to the RMB (via the Link link + RToken).

Host X Host Y
+-------------------+ ,.--.,_ +-------------------+
| | .'` '. | |
| Protection | ,' `, | Protection |
| Domain X | / \ | Domain Y |
| +------+ / \ +------+ |
| QP 8 |RNIC 1| | SMC-R Link | |RNIC 2| QP 64 |
| | | |<-------------------->| | | |
| | | || || | | |
| | +------+| VLAN A |+------+ | |
| | || || | |
| | | | RoCE | | | |
| |RTokenX) |RToken X | \ / |RToken (Y)| Y | |
| | | \ / | | |
| V | `. ,' | V |
| +--------+ | '._ ,' | +--------+ |
| | | | `''-'`` | | | |
| | RMB | | | | RMB | |
| | | | | | | |
| +--------+ | | +--------+ |
+-------------------+ +-------------------+

Figure 2 SMC link 2: SMC-R Link and RMBs

An SMC-R link can support multiple RMBs which that are independently
managed by each peer. The number of and the size of RMBs are managed by
the peers based on host the host's unique memory management requirements;
however
however, the maximum number of RMBs that can be associated to a link
group on one peer is 255. The QP has a single protection domain, but
each RMB has a unique RToken. All RTokens must be exchanged with the
peer.

Each peer manages the RMBs in its local memory for its remote SMC-R
peer by sharing access to the RMBs via Rtokens RTokens with its peers. The
remote peer writes into the RMBs via RDMA RDMA, and the local peer (RMB
owner) then reads from the RMBs.

When two peers decide to use SMC-R for a given TCP connection, they
each allocate a local RMB Element element for the TCP connection and
communicate the location of this local RMB Element element during rendezvous
processing. To that end, RMB elements are created in pairs, with one
RMB element allocated locally on each peer of the SMC-R link.

--- +-----------+----------------+ +------------+---------------+
/\ |Eyecatcher |Eye Catcher | |
| +-----------+ +------------+ |
| | |
RMB Element 1 | |
| | Receive Buffer |
| | |
| | |
\/ | |
--- +-----------+----------------+ +------------+---------------+
/\ |Eyecatcher |Eye Catcher | |
| +-----------+ +------------+ |
| | |
RMB Element 2 | |
| | Receive Buffer |
| | |
| | |
\/ | |
--- +----------------------------+
| . |
| . |
| . |
| . |
| (up to 255 elements) |
+----------------------------+

Figure 3 3: RMB Format

Figure 3 illustrates the basic format of an RMB. The RMB is a
virtual memory buffer whose backing real memory is pinned, which can
support up to 255 TCP connections to exactly one remote SMC-R peer.
Each RMB is therefore associated with the SMC-R links within a link
group for the two peers and a specific RoCE Protection Domain. Other
than the 2 two peers identified by the SMC-R link link, no other SMC-R peers
can have RDMA access to an RMB; this requires a unique Protection
Domain for every SMC-R Link. link. This is critical to ensure integrity of
SMC-R communications.

RMBs are subdivided into multiple elements for efficiency, with each
RMBE element
RMB Element (RMBE) is associated with a single TCP connection.
Therefore
Therefore, multiple TCP connections across an SMC SMC-R link group can
share the same memory for RDMA purposes, reducing the overhead of
having to register additional memory with the RNIC for every new TCP
connection. The number of elements in an RMB and the size of each RMB
Element is
RMBE are entirely governed by the owning peer peer, subject to the SMC-R
architecture rules, rules; however, all RMB elements within a given RMB must
be the same size. Each peer can decide the level of resource sharing resource-sharing
that is desirable across TCP connections based on local constraints constraints,
such as available system memory, etc. memory. An RMB Element element is identified to the
remote SMC-R peer via an RMB Element Token Token, which consists of the
following:

o RMB RToken: The combination of the Rkey RKey and virtual address
provided by the RNIC that identifies the start of the RMB for RDMA
operations.

o RMB Index: Identifies the RMB element index in the RMB. Used to
locate a specific RMB element within an RMB. Valid value range is
1-255.

o RMB element length: Element Length: The length of the RMB element's eyecatcher eye catcher
plus the length of the receive buffer. This length is equal for
all RMB elements in a given RMB. This length can be variable
across different RMBs.

Multiple RMBs can be associated to an SMC-R link group group, and each peer
in an SMC-R link group manages allocation of its RMBs. RMB
allocation can be asymmetric. For example, server Server X can allocate 2 two
RMBs to an SMC-R link group while server Server Y allocates 5. five. This
provides maximum implementation flexibility to allow hosts to
optimize RMB management for their own local requirements. The
maximum number of RMBs that can be allocated on one peer to a link
group is 255. If more RMBs are required, the peer may fall back to
IP for subsequent connections or, if the peer is the server, create a
parallel link group.

One use case for multiple RMBs is multiple receive buffer sizes.
Since every element in an RMB must be the same size, multiple RMBs
with different element sizes can be allocated if varying receive
buffer sizes are required.

Also

Also, since the maximum number of TCP connections whose receive
buffers can be allocated to an RMB is 255, multiple RMBs may be
required to provide capacity for large numbers of TCP connections
between two peers.

Separately from the RMB, the TCP/IP stack that owns each RMB
maintains control data for each RMB element within its local control
structures. The control data contains flags for maintaining the
state of the TCP data (for example, urgent data indicator) and and, most
importantly, the following two cursors cursors, which are illustrated below
in Figure 4:

o The peer producer cursor: This is a wrapping offset into the
RMB element's receive buffer that points to the next byte of data
to be written by the remote peer. This cursor is provided by the
remote peer in a Connection Data Control (CDC message), (CDC) message, which is
sent using RDMA sendmsg RoCE SendMsg processing, and tells the local peer how
far it can consume data in the RMBE buffer.

o The peer consumer cursor: This is a wrapping offset into the
remote peer's RMB element's receive buffer that points to the next
byte of data to be consumed by the remote peer in its own RMBE.
The local peer cannot write into the remote peer's RMBE beyond
this point without causing data loss. This cursor is also
provided by the peer using a Connection Data Control message.

Each TCP connection peer maintains its cursors for a TCP connection's
RMBE in its local control structures. In other words, the peer who
writes into a remote peer's RMBE provides its producer cursor to the
peer whose RMBE it has written into. The peer who reads from its
RMBE provides its consumer cursor to the writing peer. In this
manner
manner, the reads and writes between peers are kept coordinated.

For example, referring to Figure 4, peer Peer B writes the hashed data
into the receive buffer of peer Peer A's RMBE. After that write
completes, peer Peer B uses a CDC message to update its producer cursor to
peer
Peer A, to indicate to peer Peer A how much data is available for peer Peer A
to consume. The CDC message that peer Peer B sends to peer Peer A wakes up
peer
Peer A and notifies it that there is data to be consumed.

Similarly, when peer Peer A consumes data written by peer Peer B, it uses a CDC
message to update its consumer cursor to peer Peer B to let peer Peer B know
how much data it has consumed, so peer Peer B knows how much space is
available for further writes. If peer Peer B were to write enough data to
peer
Peer A that it would wrap the RMBE receive buffer and exceed the
consumer cursor, data loss would result.

Note that this is a simplistic description of the control flows flows, and
they are optimized to minimize the number of CDC messages required,
as described in 4.7. RMB data flows. Section 4.7 ("RMB Data Flows").

Peer A's RMBE Control Info Peer B's RMBE Control Info
+--------------------------+ +--------------------------+
| | | |
/----Peer producer cursor | +-----+-Peer consumer cursor |
/| | | | |
| +--------------------------+ | +--------------------------+
| Peer A's RMBE |
| +--------------------------+ |
| | +------------------+
| | | |
| | \/ |
| | +------------|
| |-------------+/////////// |
| |//RMA |//RDMA data written by /// | ///|
| |/// peer Peer B that is ////// |
| |/available to be consumed/|
| |///////////////////////// |
| |///////// +---------------|
| |----------+/\ |
| | | |
\| | |
\ / |
|\---------/ |
| |
| |

Figure 4 4: RMBE cursors Cursors

Additional flags and indicators are communicated between peers. In
all cases, these flags and indicators are updated by the peer using
CDC messages with the control information contained in inline data. messages, which are sent using RoCE SendMsg. More details on
these additional flags and indicators are described in . 4.3. RMBE control information. Section 4.3
("RMBE Control Information").

2.2. SMC-R Link groups Groups

SMC-R links are logically grouped together to form an SMC-R Link
Group. link
group. The purpose of the Link Group link group is for supporting multiple
links between the same two peers to provide for:

o Resilience: Provides transparent and dynamic switching of the link
used by existing TCP connections during link failures, typically
hardware related. TCP traffic using the failing link can be
switched to an active link within the link group group, thereby avoiding
disruptions to application workloads.

o Link utilization: Provides an active/active link usage model
allowing TCP traffic to be balanced across the links, which
increases bandwidth and also avoids hardware imbalances and
bottlenecks. Note that both adapter and switch utilization can
become potential resource constraint issues issues.

SMC-R Link Group link group support is required. Resilience is not optional.
However, the user can elect to provision a single RNIC (on one or
both hosts).

Multiple links that are formed between the same two peers fall into
two distinct categories:

1. Equal Links: Links providing equal access to the same RMB(s) at
both endpoints endpoints, whereby all TCP connections associated with the
links must have the same VLAN ID and have the same TCP server and
TCP client roles or relationship.

2. Unequal Links: Links providing access to unique, unrelated and
isolated RMB(s) (i.e. (i.e., for unique VLANs or unique and isolated
application workloads, etc.) or have having unique TCP server or client
roles.

Links that are logically grouped together forming an SMC Link Group SMC-R link group
must be equal links.

2.2.1. Link group types Group Types

Equal links within a link group also have another "Link Group Type"
attribute based on the link's associated underlying physical path.
The following SMC-R link types are defined:

1. Single Link: link: the only active link within a link group

2. Parallel Link: link: not allowed - SMC Links -- SMC-R links having the same physical
RNIC at both hosts
3. Asymmetric Link: link: links that have unique RNIC adapters at one host
but share a single adapter at the peer host

4. Symmetric Link: link: links that have unique RNIC adapters at both hosts

These link group types are further explained in the following figures
and descriptions.

Figure 2 above shows the single link single-link case. The single link
illustrated in Figure 2 also establishes the SMC-R Link Group. link group. Link
groups are supposed to have multiple links, but when only one RNIC is
available at both hosts then only a single link can be created. This
is expected to be a transient case.

Figure 5 shows the symmetric link symmetric-link case. Both hosts have unique and
redundant RNIC adapters. This configuration meets the objectives for
providing full RoCE redundancy required to provide the level of
resilience required for high availability for SMC-R. While this
configuration is not required, it is a strongly recommended "best
practice" for the exploitation of SMC-R. Single and asymmetric links
must be supported but are intended to provide for short term short-term
transient conditions, conditions -- for example example, during a temporary outage or
recycle of a an RNIC.

Host X Host Y
+-------------------+ +-------------------+
| | | |
| Protection | | Protection |
| Domain X | | Domain Y |
| +------+ +------+ |
| QP 8 |RNIC 1| SMC-R Link 1 |RNIC 2| QP 64 |
|RToken X| | |<-------------------->| | | |
| | | | | | |RToken Y|
| \/ +------+ +------+ \/ |
|+--------+ | | +--------+ |
|| | | | | | |
|| RMB | | | | RMB | |
|| | | | | | |
|+--------+ | | +--------+ |
| /\ +------+ +------+ /\ |
|RToken Z| | | SMC-R Link 2 | | |RToken W|
| | |RNIC 3|<-------------------->|RNIC 4| | |
| QP 9 | | | | QP 65 |
| +------+ +------+ |
+-------------------+ +-------------------+

Figure 5 5: Symmetric SMC-R links Links
Host X Host Y
+-------------------+ +-------------------+
| | | |
| Protection | | Protection |
| Domain X | | Domain Y |
| +------+ +------+ |
| QP 8 |RNIC 1| SMC-R Link 1 |RNIC 2| QP 64 |
|RToken X| | |<-------------------->| | | |
| | | | .->| | |RToken Y|
| \/ +------+ .` +------+ \/ |
|+--------+ | .` | +--------+ |
|| | | .` | | | |
|| RMB | | .` | | RMB | |
|| | | .`SMC-R | | | |
|+--------+ | .` Link 2 | +--------+ |
| /\ +------+ .` +------+ |
|Rtoken
|RToken Z| | | .` | |down or |
| | |RNIC 3|<-` |RNIC 4|unavailable |
| QP 9 | | | | |
| +------+ +------+ |
+-------------------+ +-------------------+

Figure 6 6: Asymmetric SMC-R links Links

In the example provided by Figure 6, host Host X has two RNICs but Host Y
only has one RNIC. RNIC because RNIC 4 is not available. This
configuration allows for the creation of an asymmetric link. While
an asymmetric link will provide some resilience (i.e. (for example, when
RNIC 1 fails) fails), ideally each host should provide two redundant RNICs.
This should be a transient case, and when RNIC 4 becomes available,
this configuration must transition to a
symmetric link symmetric-link configuration.
This transition is accomplished by first creating the new symmetric link,
link and then deleting the asymmetric link with reason code
"Asymmetric link no longer needed" specified in the DELETE LINK LLC
message.

Host X Host Y
+-------------------+ +-------------------+
| | | |
| Protection | | Protection |
| Domain X | | Domain Y |
| +------+ SMC-R link Link 1 +------+ |
| QP 8 |RNIC 1|<-------------------->|RNIC 2| QP 64 |
|RToken X| | | | | | |
| | | |<-------------------->| | |Rtoken |RToken Y|
| \/ +------+ SMC-R link Link 2 +------+ \/ |
|+--------+ QP 9 | | QP 65 +--------+ |
|| | | | | | | | |
|| RMB |<-- + | | +---->| RMB | |
|| | | | | | |
|+--------+ | | +--------+ |
| +------+ +------+ |
| down or| | | |down or |
| unavailale|RNIC unavailable|RNIC 3| |RNIC 4|unavailable |
| | | | | |
| +------+ +------+ |
+-------------------+ +-------------------+

Figure 7 7: SMC-R parallel links (not supported) Parallel Links (Not Supported)

Figure 7 shows parallel links, which are two links in the link group
that use the same hardware. This configuration is not permitted.
Because SMC-R multiplexes multiple TCP connections over an SMC-R link
and both links are using the exact same hardware, there is no
additional redundancy or capacity benefit obtained from this
configuration. However In addition to providing no real benefit, this
configuration does add adds the unnecessary overhead of additional queue
pairs, generation of additional Rkeys, RKeys, etc.

2.2.2. Maximum number Number of links Links in link group Link Group

The SMC-R protocol defines a maximum of 8 eight symmetric SMC-R links
within a single SMC-R link group. This allows for support for up to
8
eight unique physical paths between peer hosts. However, in terms of
meeting the basic requirements for redundancy redundancy, support for at least 2
two symmetric links must be implemented. Supporting greater more than 2 two
links also simplifies implementation for practical matters relating
to dynamically adding and removing links, links -- for example example, starting a
third SMC-R link prior to taking down one of the two existing links.
Recall that all links within a link group must have equal access to
all associated RMBs.

The SMC-R protocol allows an implementation to implement assign an
implementation specific
implementation-specific and appropriate value for maximum symmetric
links. The implementation value must not exceed the architecture
limit of 8 and 8; also, the implementation value must not be lower than 2, because the
SMC-R protocol requires redundancy. This does not mean that two
RNICs are physically required to enable SMC-R connectivity, but at
least two RNICs for redundancy are strongly recommended.

The SMC-R peers exchange their implementation maximum link values
during the link group establishment using the defined maximum link
value in the CONFIRM LINK LLC command. Once the initial exchange
completes
completes, the value is set for the life of the link group. The
maximum link value can be provided by both the server and client.
The server must supply a value, whereas the client maximum link value
is optional. When the client does not supply a value, it indicates
that the client accepts the server supplied server-supplied maximum value. If the
client provides a value value, it can not cannot exceed the server server-supplied maximum
value. If the client passes a lower value then value, this lower value then
becomes the final negotiated maximum number of symmetric links for
this link group. Again, the minimum value is 2.

During run time time, the client must never request that the server add a
symmetric link to a link group that would exceed the negotiated
maximum link value. Likewise Likewise, the server must never attempt to add a
symmetric link to a link group that would exceed the negotiated
maximum value.

In terms of counting the number of active link count links within a link group,
the initial link (or the only / last) only/last) link is always counted as 1. Then
Then, as additional links are added added, they are either symmetric or
asymmetric links.

With regards to enforcing the maximum link rules, asymmetric links
are an exception having a unique set of rules:

o Asymmetric links are always limited to one asymmetric link allowed
per link group group.

o Asymmetric links must not be counted in the maximum symmetric link symmetric-link
count calculation. When tracking the current count or enforcing
the negotiated maximum number of links, an asymmetric link is not
to be counted counted.

2.2.3. Forming and managing link groups Managing Link Groups

SMC-R link groups are self-defining. The first SMC-R link in a link
group is created using TCP option flows on the TCP three-way
handshake followed by CLC message flows over the TCP connection.
Subsequent SMC-R links in the link group are created by sending LLC
messages over an SMC-R link that already exists in the link group.
Once an SMC-R link group is created, no additional SMC-R links in
that group are created using TCP and CLC negotiation. Because
subsequent SMC-R links are created exclusively by sending LLC
messages over an existing SMC-R link in a link group, the membership
of SMC-R links to in a link group is self-defining.

This architecture does not define a specific identifier for an SMC-R
link group. This identification may be useful for network management
and may be assigned in a platform specific platform-specific manner, or in an extension
to this architecture.

In each SMC-R link group, one peer is the server for all TCP
connections and the other peer is the client. If there are
additional TCP connections between the peers that use SMC-R and have
the client and server roles reversed, another SMC-R link group is set
up between them with the opposite client-server relationship.

This is required because there are specific responsibilities divided
between the client and server in the management of an SMC-R link
group.

In this architecture, the decision of whether or not to use an existing
SMC-R link group or create a new SMC-R link group for a TCP
connection is made exclusively by the server.

Management of the links in an SMC-R link group is also a server
responsibility. The server is responsible for adding and deleting
links in a link group. The client may request that the server take
certain actions actions, but the final responsibility is the server's.

2.2.4. SMC-R link identifiers Link Identifiers

This architecture defines multiple identifiers to identify SMC-R
links and peers.

o Link number: This is a one-byte 1-byte value that identifies an SMC-R link
within a link group. Both the server and the client use this
number to distinguish an SMC-R link from other links within the
same link group. It is only unique within a link group. In order
to prevent timing windows that may occur when a server creates a
new link while the client is still cleaning up a previously
existing link, link numbers cannot be reused until the entire link
numbering space has been exhausted.

o Link User user ID: This is an architecturally opaque four byte 4-byte value that
a peer uses to uniquely define an SMC-R link within its own space.
This means that a link user ID is unique within one peer only.
Each peer defines its own link user ID for a link. The peers
exchange this information once during link setup setup, and it is never
used architecturally again. The purpose of this identifier is for
network management, display, and debugging purposes. debugging. For
example example, an
operator on a client could provide the operator on the server with
the server's link user ID if he requires the server's operator to
check on the operation of a link that the client is having trouble
with.

o Peer ID: The SMC-R peer ID uniquely identifies a specific instance
of a specific TCP/IP stack. It is required because in clustered
and load balancing load-balancing environments, an IP address does not uniquely
identify a TCP/IP stack. An RNIC's MAC/GID also doesn't uniquely
or reliably identify a TCP/IP stack stack, because RNICs can go up and
down and even be redeployed to other TCP/IP stacks in a multiple
partitioned
multiple-partitioned or virtualized environment. The peer ID is
not only unique per TCP/IP stack but is also unique per instance
of a TCP/IP stack, meaning that if a TCP/IP stack is restarted,
its peer ID changes.

2.3. SMC-R resilience Resilience and load balancing Load Balancing

The SMC-R multi-link multilink architecture provides resilience for network high
availability via failover capability to an alternate RoCE adapter.

The SMC-R multilink architecture does not define primary, secondary secondary,
or alternate roles to the links. Instead Instead, there are multiple active
links representing multiple redundant RoCE paths over the same LAN.

Assignment of TCP connections to links is unidirectional and
asymmetric. This means that the client and server may each choose a
separate link for their RDMA writes associated with a specific TCP
connection.

If a hardware failure occurs or a QP failure associated with an
individual link, link occurs, then the TCP connections that were associated
with the failing link are dynamically and transparently switched to
use another available link. The server or the client can detect a
failure and
failure, immediately move their TCP connections connections, and then notify
their peer via the DELETE LINK LLC command. While the client can
notify the server of an apparent link failure with the DELETE LINK
LLC command, the server performs the actual link deletion.

The movement of TCP connections to another link can be accomplished
with minimal coordination between the peers. The TCP connection
movement is also transparent to to, and non disruptive to non-disruptive to, the TCP
socket application workloads for most failure scenarios. After a
failure, the surviving links and all associated hardware must handle
the link group's workload.

As each SMC-R peer begins to move active TCP connections to another
link
link, all current RDMA write operations must be allowed to complete.
Then the
The moving peer then sends a signal to verify receipt of the last
successful write by its peer. If this verification fails, the TCP
connection must be reset. Once this verification is complete, all
writes that failed may then be retried, in order, over the new link.
Any data writes or CDC messages for which the sender did not receive
write completion must be replayed before any subsequent data or CDC
write operations are sent. LLC messages are not retried over the new
link
link, because they are dependent on a known link configuration, which
has just changed because of the failure. The initiator of an LLC
message exchange that fails will be responsible for retrying once the
link group configuration stabilizes.

When a new link becomes available and is re-added to the link group
then group,
each peer is free to rebalance its current TCP connections as needed
or only assign new TCP connections to the newly added link. Both the
server and client are free to manage TCP connections across the link
group as needed. TCP connection movement does not have to be
stimulated by a link failure.

The SMC-R architecture also defines orderly vs. versus disorderly
failover. The type of failover is communicated in the LLC Delete Link
DELETE LINK command and is simply a means to indicate that the link
has terminated (disorderly) or link termination is imminent
(orderly). The orderly link deletion could be initiated via operator
command or programmatically to bring down an idle link. For example example,
an operator command could initiate orderly
shut down shutdown of an adapter for
service. Implementation of the two types is based on implementation
requirements and is beyond the scope of the SMC-R architecture.

3. SMC-R Rendezvous architecture

Rendezvous Architecture

"Rendezvous" is the process that SMC-R capable SMC-R-capable peers use to
dynamically discover each others' capabilities, negotiate SMC-R
connections, set up SMC-R links and link groups, and manage those
link groups. A key aspect of SMC-R rendezvous Rendezvous is that it occurs
dynamically and automatically, without requiring SMC SMC-R link
configuration to be defined by an administrator.

SMC-R Rendezvous starts with the TCP/IP three-way handshake handshake, during
which connection peers use TCP options to announce their SMC-R
capabilities. If both endpoints are SMC-R capable, then Connection
Layer Control (CLC) messages are exchanged between the peers' SMC-R
layers over the newly established TCP connection to negotiate SMC-R
credentials. The CLC message mechanism is analogous to the messages
exchanged by SSL for its handshake processing.

If a new SMC-R link is being set up, Link Layer Control (LLC)
messages are used to confirm RDMA connectivity. LLC messages are
also used by the SMC-R layers at each peer to manage the links and
link groups.

Once an SMC-R link is set up or agreed to by the peers, the TCP
sockets are passed to the peer applications applications, which use them as
normal. The SMC-R layer, which resides under the sockets layer,
transmits the socket data between peers over RDMA using the SMC-R
protocol, bypassing the TCP/IP stack.

3.1. TCP options Options

During the TCP/IP three-way handshake, the client and server indicate
their support for SMC-R by including experimental TCP option 254 on
the three-way handshake flows, in accordance with RFC 6994 "Shared [RFC6994] ("Shared
Use of Experimental TCP Options". Options"). The ExID Experiment Identifier (ExID)
value used is the string
'SMCR' "SMCR" in EBCDIC (IBM-1047) encoding
(0xE2D4C3D9). This ExID has been registered in the TCP ExIDs "TCP Experimental
Option Experiment Identifiers (TCP ExIDs)" registry maintained
by IANA.

After completion of the 3-way three-way TCP handshake handshake, each peer queries
its peer's options. If both peers set the TCP option on the
three-way handshake, inline SMC-R negotiation occurs using CLC
messages. If neither peer peer, or only one peer set peer, sets the TCP option,
SMC-R cannot be used for the TCP connection, and the TCP connection
completes the setup using the IP fabric.

3.2. Connection Layer Control (CLC) messages Messages

CLC messages are sent as data payload over the IP network using the
TCP connection between SMC-R layers at the peers. They are analogous
to the messages used to exchange parameters for SSL.

Use

The use of CLC messages is detailed in the following sections. The
following list provides a summary of the defined CLC messages and
their purposes:

o SMC PROPOSAL: Proposal: Sent from the client to propose that this TCP
connection is eligible to be moved to SMC-R. The client
identifies itself and its subnet to the server and passes the
SMC-R elements for a suggested RoCE path via the MAC and GID.

o SMC ACCEPT: Accept: Sent from the server to accept the client's TCP
connection SMC proposal. Proposal. The server responds to the client's
proposal by identifying itself to the client and passing the
elements of a RoCE path that the client can use to to perform RDMA
writes to the server. This consists of such SMC-R ink link elements such
as RoCE MAC, GID, and RMB information etc. information.

o SMC CONFIRM: Confirm: Sent from the client to confirm the server's
acceptance of the SMC connection. The client responds to the
server's acceptance by passing the elements of a RoCE path that
the server can use to to perform RDMA writes to the client. This
consists of such SMC-R ink link elements such as RoCE MAC, GID, and RMB information etc.
information.

o SMC DECLINE: Decline: Sent from either the server or the client to reject
the SMC connection, indicating the reason the peer must decline
the SMC proposal Proposal and allowing the TCP connection to revert back to
IP connectivity.

3.3. LLC messages Messages

Link Layer Control (LLC) messages are sent between peer SMC-R layers
over an SMC-R link to manage the link or the link group. LLC
messages are sent using RoCE sendmsg with inline data SendMsg and are 44 bytes long. The 44 bytes
44-byte size is based on what can fit into a RoCE Work Queue Element
(WQE) without requiring the posting of receive buffers.

LLC messages generally follow a request-reply semantic. Each message
has a request flavor and a reply flavor, and each request must be
confirmed with a reply, except where otherwise noted. Use The use of LLC
messages is detailed in the following sections. The following list
provides a summary of the defined LLC messages and their purposes:

o ADD LINK: Add Used to add a new link to a link group. Sent from the
server to the client to initiate addition of a new link to the
link group, or from the client to the server to request that the
server initiate addition of a new link.

o ADD LINK CONTINUATION: This is a A continuation of ADD link LINK that allows the
ADD link LINK to span multiple commands, because all of the link
information cannot be contained in a single ADD LINK message message.

o CONFIRM LINK: Used to confirm that RoCE connectivity over a newly
created SMC-R link is working correctly. Initiated by the server,
and both server.
Both this message and its reply must flow over the SMC-R link
being confirmed.

o DELETE LINK: When initiated by the server, deletes a specific link
from the link group or deletes the entire link group. When
initiated by the client, requests that the server delete a
specific link or the entire link group.

o CONFIRM RKEY: Informs the peer on the SMC-R link of the addition
of an RMB to the link group.

o CONFIRM RKEY CONTINUATION: This is a A continuation of CONFIRM RKEY that
allows the ADD link CONFIRM RKEY to span multiple commands, in the event
that all of the information cannot be contained in a single
CONFIRM RKEY message.

o DELETE RKEY: Informs the peer on the SMC-R link of the deletion of
one or more RMBs from the link group group.

o TEST LINK: Verifies that an already-active SMC-R link is active
and healthy healthy.

o Optional LLC message: Any LLC message in which the two high order high-order
bits of the opcode are b'10' is an b'10'. This optional message and must be
silently discarded by a receiving peer that does not support the
opcode. No such messages are defined in this version of the
architecture, however
architecture; however, the concept is defined to allow for
toleration of possible advanced, optional functions.

CONFIRM LINK and TEST LINK are sensitive to which link they flow on
and must flow on the link being confirmed or tested. The other flows
may flow over any active link in the link group. When there are
multiple links in a link group, a response to an LLC message must
flow over the same link that the original message flowed over, with
the following exceptions:

o ADD LINK request from a server in response to an ADD LINK from a
client
client.

o DELETE LINK request from a server in response to a DELETE LINK
from a client client.

3.4. CDC Messages

Connection Data Control (CDC) messages are sent over the RoCE fabric
between peers using RoCE sendmsg with inline data, SendMsg and are 44 bytes
long which long. The 44-byte
size is based on the size that can fit into a RoCE Work Queue
Element (WQE) WQE without
requiring the posting of receive buffers. CDC messages are used to
describe the socket application data passed via RDMA write
operations, and as well as TCP connection state information information, including
producer cursors and consumer cursors, RMBE state information, and
failover data validation.

3.5. Rendezvous flows Flows

Rendezvous information for SMC-R is be exchanged as TCP options on the
TCP 3-way three-way handshake flows to indicate capability, followed by in-
line
inline TCP negotiation messages to actually do the SMC-R setup.
Formats of all rendezvous options and messages discussed in this
section are detailed in Appendix A.

3.5.1. First contact Contact

First contact between RoCE peers occurs when a new SMC-R link group
is being set up. This could be because no SMC-R links already exist
between the peers, or the server decides to create a new SMC-R link
group in parallel with an existing one.

3.5.1.1. Pre-negotiation of TCP Options pre-negotiation

The client and server indicate their SMC-R capability to each other
using TCP option 254 on the TCP 3-way three-way handshake flows.

A client who wishes to do SMC-R will include TCP option 254 using an
ExID equal to the EBCDIC (codepage IBM-1047) encoding of "SMCR" on
its SYN flow.

A server that supports SMC-R will include TCP option 254 with the
ExID value of EBCDIC "SMCR" on its SYN-ACK flow. Because the server
is listening for connections and does not know where client
connections will come from, the server implementation may choose to
unconditionally include this TCP option if it supports SMC-R. This
may be required for server implementations where extensions to the
TCP stack are not practical. For server implementations which that can add
code to examine and react to packets during the three-way handshake,
the server should only include the SMC-R TCP option on the SYN-ACK if
the client included it on its SYN packet.

A client who supports SMC-R and meets the three conditions outlined
above may optionally include the TCP option for SMC-R on its ACK
flow, regardless of whether or not the server included it on its SYN-
ACK
SYN-ACK flow. Some TCP/IP stacks may have to include it if the SMC-R
layer cannot modify the options on the socket until the 3-way three-way
handshake completes. Proprietary servers should not include this
option on the ACK flow, since including it on the SYN flow was
sufficient to indicate the client's capabilities.

Once the initial three-way TCP handshake is completed, each peer
examines the socket options. SMC-R implementations may do this by
examining what was actually provided on the SYN and SYN-ACK packets
or by performing a getsockopt() operation to determine the options
set
sent by the peer. If neither peer, or only one peer, specified the
TCP option for SMC-R, then SMC-R cannot be used on this connection
and it proceeds using normal IP flows and processing.

If both peers specified the TCP option for SMC-R, then the TCP
connection is not started yet and the peers proceed to SMC-R
negotiation using inline data flows. The socket is not yet turned
over to the applications; instead instead, the respective SMC layers exchange
CLC messages over the newly formed TCP connection.

3.5.1.2. Client Proposal

If SMC-R is supported by both peers, the client sends an SMC Proposal
CLC message to the server. On It is not immediately apparent on this
flow from client to server it is
not immediately apparent if whether this is a new or existing SMC-R link
link, because in clustered environments a single IP address may
represent multiple hosts. This type of cluster virtual IP address
can be owned by a network based network-based or host based layer host-based Layer 4 load balancer
that distributes incoming TCP connections across a cluster of
servers/hosts. Other For purposes of high availability, other clustered
environments may also support the movement of a virtual IP address
dynamically from one host in the cluster to another for high availability purposes. another. In summary, the
client can not pre-determine cannot predetermine that a connection is targeting the same
host simply by simply matching the destination IP address for outgoing TCP
connections. Therefore Therefore, it cannot pre-determine predetermine the SMC-R link that
will be used for a new TCP connection. This information will be
dynamically learned learned, and the appropriate actions will be taken as the
SMC-R negotiation handshake unfolds.

In the SMC-R proposal message, the initiator (client) proposes the
use of SMC-R by including its peer ID and GID ID, GID, and MAC addresses, as
well as the IP subnet number of the outgoing interface (if IPv4) or
the IP prefix list for the network that over which the proposal is sent over
(if IPv6). At this point in the flow, the client makes no local
commitments of resources for SMC-R.

When the server receives the SMC Proposal CLC message, it uses the
peer ID provided by the client client, plus subnet or prefix information
provided by the client, to determine if it already has a usable SMC-R
link with this SMC-R peer. If there is are one or more existing SMC-R
links with this SMC-R peer, the server then decides which SMC SMC-R link
it will use for this TCP connection. See subsequent sections Sections 3.5.2 and 3.5.3
for the cases of reusing an existing SMC-R link or creating a
parallel SMC SMC-R link group between SMC-R peers.

If this is a first contact between SMC-R peers peers, the server must
validate that it is on the same LAN as the client before continuing.
For IPv4, the server does this by verifying that it has an interface
with an IP subnet number that matches the subnet number set sent by the
client on in the SMC Proposal. For IPv6 IPv6, it does this by verifying that
it is directly attached to at least one IP prefix that was listed by
the client in its SMC Proposal message.

If the server agrees to use SMC-R, the server begins the setup of a
new SMC-R link by allocating local QP and RMB resources (setting its
QP state to INIT) and providing its full SMC-R information in an SMC
Accept CLC message to the client over the TCP connection, along with
a flag set indicating that this is a first contact flow. While the
SMC Accept message could flow over any IP route back to the client
depending upon Layer 3 IP routing, the SMC-R credentials provided
must be for the common subnet or prefix between the server and
client, as determined above. If the server cannot or does not want
to do SMC-R with the
client client, it sends an SMC Decline CLC message to
the client client, and the connection data may begin flowing using normal
TCP/IP flows.

3.5.1.3. Server acceptance Acceptance

When the client receives the SMC Accept from the server, it uses
determines whether this is a new or existing SMC-R link, using the
combination of the following: the first contact flag, its GID/MAC MAC/GID and
the GID/MAC MAC/GID returned by the server plus server, the LAN that VLAN over which the
connection is setting up
over up, and the QP number provided by the server to determine if this is
a new or existing SMC-R link. server.

If it is an existing SMC-R link, link and the client agrees to use that
link for the TCP connection, see 3.5.2. Subsequent contact Section 3.5.2 ("Subsequent Contact")
below. If it is a new SMC-R link between peers that already have an SMC
SMC-R link, then the server is starting a new SMC SMC-R link group.

Assuming that either (1) this is either a first contact between peers or
(2) the server is starting a new SMC SMC-R link group, the client now
allocates local QP and RMB resources for the SMC-R link (setting the
QP state to RTR or
"ready (ready to receive"), receive)), associates them with the server
QP as learned on from the SMC Accept CLC message, and sends an SMC
Confirm CLC message to the server over the TCP connection with its
SMC-R link information included. The client also starts a timer to
wait for the server to confirm the reliable reliably connected QP queue pair, as
described below.

3.5.1.4. Client confirmation Confirmation

Upon receipt of the client's SMC Confirm CLC message, the server
associates its QP for this SMC-R link with the client's QP as learned
on
from the SMC Confirm CLC message and sets its QP state to RTS (ready
to send). Now the The client and the server now have reliable reliably connected QPs.
queue pairs.

3.5.1.5. Link (QP) confirmation Confirmation

Since setting up the SMC-R link and its QPs did not require any
network flows on the RoCE fabric, the client and server must now
confirm connectivity over the RoCE fabric. To accomplish this, the
server will send a "Confirm Link" CONFIRM LINK Link Layer Control (LLC) message to
the client over the newly created SMC-R link, using the RoCE fabric.
The "Confirm Link" CONFIRM LINK LLC message will provide the server's MAC, GID, and
QP information for the connection, allow each partner to communicate
the maximum number of links it can tolerate in this link group (the
"link limit"), and will additionally provide two link IDs:

o a one-byte 1-byte server-assigned Link link number that is used by both peers to
identify the link within the link group and is only unique within
a link group.

o a four byte 4-byte link user id. ID. This opaque value is assigned by the
server for the server's local use and is provided to the client
for management purposes, purposes -- for example example, to use in network
management displays and products.

When the server sends this message, it will set a timer for receiving
confirmation from the client.

When the client receives the server's confirmation "Confirm Link" in the form of a
CONFIRM LINK LLC
message message, it will cancel the confirmation timer it
set when it sent the SMC Confirm message. It The client will also
advance its QP state to RTS and respond over the RoCE fabric with a "Confirm Link"
CONFIRM LINK response LLC
message, providing message that (1) provides its MAC, GID,
QP number, and link limit, confirming (2) confirms the one byte 1-byte link number sent
by the server, and providing (3) provides its own
four byte 4-byte link user id ID to the
server.

Host X -- Server Host Y -- Client
+-------------------+ +-------------------+
| PeerID Peer ID = PS1 | | PeerID Peer ID = PC1 |
| +------+ +------+ |
| QP 8 |RNIC 1| |RNIC 2| QP 64 |
|RToken X| |MAC MA| |MAC MB| | |
| | |GID GA| |GID GB| |Rtoken |RToken Y|
| \/ +------+ (Subnet S1) +------+ \/ |
|+--------+ | | +--------+ |
|| RMB | | | | RMB | |
|+--------+ | | +--------+ |
| +------+ +------+ |
| |RNIC 3| |RNIC 4| |
| |MAC MC| |MAC MD| |
| |GID GC| |GID GD| |
| +------+ +------+ |
+-------------------+ +-------------------+

SYN TCP options(254,"SMCR")
<---------------------------------------------------------

SYN-ACK TCP options(254, "SMCR") options(254,"SMCR")
--------------------------------------------------------->

ACK [TCP options(254, "SMCR")] options(254,"SMCR")]
<--------------------------------------------------------

SMC Proposal(PC1,MB,GB,S1)
<--------------------------------------------------------

SMC Accept(PS1,first contact,MA,GA,MTU,QP8,RToken=X,RMB elem ndx) index)
--------------------------------------------------------->

SMC Confirm(PC1,MB,GB,MTU,QP64,RToken=Y, RMB Confirm(PC1,MB,GB,MTU,QP64,RToken=Y,RMB element index)
<--------------------------------------------------------

Confirm Link (MA,GA,QP8,

CONFIRM LINK(MA,GA,QP8, link lim, server's server link userid, user ID, linknum)
.........................................................>

Confirm Link Rsp(MB,GB,QP64,

CONFIRM LINK rsp(MB,GB,QP64, link lim, client link userid, user ID, linknum)
<........................................................

Legend:
------------ TCP/IP and CLC flows
............ RoCE (LLC) flows
Square brackets ("[ ]") indicate optional information

Figure 8 8: First contact rendezvous flows Contact Rendezvous Flows
Technically, the data for the TCP connection could now flow over the
RoCE path. However However, if this is a first contact, there is no
alternate for this recently established RoCE path. Since in the
current architecture there is no failover from RoCE to IP once
connection data starts flowing, this means that a failure of this
path would disrupt the TCP connection, meaning that the level of
redundancy and failover is less than that provided by IP. If the
network has alternate RoCE paths available, they would not be usable
at this
point, which is an unacceptable condition point. This situation would be unacceptable.

3.5.1.6. Second SMC-R link setup Link Setup

Because of the unacceptable situation described above, TCP data will
not be allowed to flow on the newly established SMC-R link until a
second path has been set up, or at least attempted.

If the server has a second RNIC available on the same LAN, it
attempts to set up the second SMC-R link over that second RNIC. If
it only has one RNIC available on the LAN, it will attempt to set up
the second SMC-R link over that one RNIC. In the latter case, the
server is attempting to set up an asymmetric link, in case the client
does have a second RNIC on the LAN.

In either case case, the server allocates a new QP over the RNIC it is
attempting to use for the second link, link and assigns a link number to
the new link and link; the server also creates an RToken for the RMB over this
second QP (note that this means that the first and second QP each has its
have their own RToken to represent the same RMB). The server
provides this information, as well as the MAC and GID of the RNIC
over which it is attempting to set up the second link over link, in an "Add Link" ADD LINK
LLC message which that it sends to the client over the SMC-R link that is
already set up.

3.5.1.6.1. Client processing Processing of "Add Link" ADD LINK LLC message Message from server Server

When the client receives the server's "Add Link" ADD LINK LLC message, it
examines the GID and MAC provided by the server to determine if whether
the server is attempting to use the same server-side RNIC as the
existing SMC-R link, link or a different one.

If the server is attempting to use the same server-side RNIC as the
existing SMC-R link, then the client verifies that it has a second
RNIC on the same LAN. If it does not, the client rejects the "Add
Link"
ADD LINK request from the server, because the resulting link would be
a parallel link link, which is not supported within a link group. If the
client does have a second RNIC on the same LAN, it accepts the
request
request, and an asymmetric link will be set up.

If the server is using a different server-side RNIC from the existing
SMC-R link link, then the client will accept the request and a second
SMC-R link will be set up in this SMC-R link group. If the client
has a second RNIC on the same LAN, that second RNIC will be used for
the second SMC-R link, creating symmetric links. If the client does
not have a second RNIC on the same LAN, it will use the same RNIC as
was used for the initial SMC-R link, resulting in the setup of an
asymmetric link in the SMC-R link group.

In either case, when the client accepts the server's "Add Link" ADD LINK
request, it allocates a new QP on the chosen RNIC and creates an Rkey RKey
over that new QP for the client-side RMB for the SMC SMC-R link group,
then sends an "Add Link" ADD LINK reply LLC message to the server providing that
information as well as echoing the Link link number that was set sent by the
server.

If the client rejects the server's "Add Link" ADD LINK request, it sends an
"Add Link" ADD
LINK reply LLC message to the server with the reason code for the
rejection.

3.5.1.6.2. Server processing Processing of "Add Link" reply ADD LINK Reply LLC message Message from the
client Client

If the client sends a negative response to the server or no reply is
received, the server frees the RoCE resources it had allocated for
the new link. Having a single link in an SMC-R link group is
undesirable and the
undesirable. The server's recovery is detailed in C.8. Failure Appendix C.8
("Failure to
add second Add Second SMC-R link Link to a link group. Link Group").

If the client sends a positive reply to the server with
MAC/GID/QP/Rkey
MAC/GID/QP/RKey information, the server associates its QP for the new
SMC-R link to the QP that the client provided. Now Now, the new SMC-R
link is in the same situation that the first was in after the client
sent its ACK packet - -- there is a reliable reliably connected QP queue pair over
the new RoCE path, but there have been no RoCE flows to confirm that
it's actually usable. So So, at this point point, the client and server will
exchange "Confirm Link" CONFIRM LINK LLC messages just like they did on the first
SMC-R link.

If either peer receives a failure during this second "Confirm Link" CONFIRM LINK LLC
exchange (either an immediate failure -- which implies that the
message did not reach the partner, partner -- or a timeout), it sends a "Delete
Link" DELETE
LINK LLC message to the partner over the first (and now only) link in
the link group which group. This DELETE LINK LLC message must be acknowledged
before data can flow on the single link in the link group.

Host X -- Server Host Y -- Client
+-------------------+ +-------------------+
| PeerID Peer ID = PS1 | | PeerID Peer ID = PC1 |
| +------+ +------+ |
| QP 8 |RNIC 1| SMC-R Link 1 |RNIC 2| QP 64 |
|RToken X| |MAC MA| |MAC MA|<-------------------->|MAC MB| | |
| | |GID GA| |GID GB| |RToken Y|
| \/ +------+ +------+ \/ |
|+--------+ | | +--------+ |
|| | | | | | |
|| RMB | | | | RMB | |
|| | | | | | |
|+--------+ | | +--------+ |
| /\ +------+ +------+ /\ |
| | |RNIC 3| SMC-R Link 2 |RNIC 4| | |
|RToken Z| |MAC MC| |MAC MC|<-------------------->|MAC MD| |RToken W| W |
| QP 9 |GID GC| (being added) |GID GD| QP 65 |
| +------+ +------+ |
+-------------------+ +-------------------+

First SMC-R link setup as shown in Figure 8
<-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.->

ADD LINK request(QP9,MC,GC, link request (QP9,MC,GC, link number=2) number = 2)
............................................>

ADD LINK response(QP65,MD,GD, link response (QP65,MD,GD, link number=2) number = 2)
<............................................

ADD link continuation request (RToken=Z) LINK CONTINUATION request(RToken=Z)
............................................>

ADD link continuation LINK CONTINUATION response(RToken=W)
<............................................

Confirm Link(MC,GC,QP9,link number=2,

CONFIRM LINK(MC,GC,QP9, link number = 2, link userid) user ID)
.............................................>

Confirm Link response(MD,GD,QP65,link number=2,

CONFIRM LINK response(MD,GD,QP65, link number = 2, link userid) user ID)
<.............................................

Legend:
------------ TCP/IP and CLC flows
............ RoCE (LLC) flows

Figure 9 9: First contact, second link setup Contact, Second Link Setup

3.5.1.6.3. Exchange of Rkeys RKeys on second Second SMC-R link Link

Note that in the scenario described here, here -- first contact, contact -- there is
only one RMB Rkey RKey to exchange on the second SMC-R link link, and it is
exchanged in the Add Link Continuation ADD LINK CONTINUATION request and reply. In
scenarios other than first contact, contact -- for example, adding a new SMC-R
link to a longstanding link group with multiple RMBs, RMBs -- additional
flows will be required to exchange additional RMB Rkeys. RKeys. See
3.5.5.2.3. Adding
Section 3.5.5.2.3 ("Adding a new New SMC-R link Link to a link group Link Group with multiple RMBs
Multiple RMBs") for more details on these flows flows.

3.5.1.6.4. Aborting SMC-R and falling back Falling Back to IP

If both partners don't provide the SMC-R TCP option during the 3 way
three-way TCP handshake, the connection falls back to normal TCP/IP.
During the SMC-R negotiation that occurs after the 3 way three-way TCP
handshake, either partner may break off SMC-R by sending an SMC
Decline CLC message. The SMC Decline CLC message may be sent in
place of any expected message, message and may also be sent during the Confirm Link CONFIRM
LINK LLC exchange if there is a failure before any application data
has flowed over the RoCE fabric. For more detail details on exactly when an
SMC Decline can flow during link group setup, see C.1. SMC Appendices C.1
("SMC Decline during CLC
negotiation Negotiation") and C.2. SMC C.2 ("SMC Decline during
LLC negotiation Negotiation").

If this fallback to IP happens while setting up a new SMC-R link
group, the RoCE resources allocated for this SMC-R link group
relationship are torn down down, and it will be retried as a new SMC-R
link group next time a connection starts between these peers with
SMC-R proposed. Note that if this happens because one side doesn't
support SMC-R, there will be very little to tear down down, as the TCP
option will have failed to flow either on either the initial SYN or the SYN-ACK,
SYN-ACK before either side had reserved any local RoCE resources.

3.5.2. Subsequent contact Contact

"Subsequent contact" means setting up a new TCP connection between
two peers that already have an SMC-R link group between them, them and
reusing the existing SMC-R link group. In this case case, it is not
necessary to allocate new QPs. However However, it is possible that a new
RMB has been allocated for this TCP connection, if the previous TCP
connection used the last element available in the previously used
RMB, or for any other implementation-dependent reason. For this
reason, and for convenience and error checking, the same TCP
option
254 254, followed by the inline negotiation method described for
initial
contact contact, will be used for subsequent contact, but the
processing differs in some ways. That processing is described below.

3.5.2.1. SMC-R proposal Proposal

When the client begins the inline negotiation with the server, it
does not know if this is a first contact or a subsequent contact.
The client cannot know this information until it sees the server's
peer ID ID, to determine whether or not it already has an SMC-R link
with this peer that it can use. There are several reasons why it is
not sufficient to use the partner IP address, subnet, VLAN VLAN, or other
IP information to make this determination. The most obvious reason
is distributed systems: if the server IP address is actually a
virtual IP address representing a distributed cluster, the actual
host serving this TCP connection may not be the same as the host that
served the last TCP connection to this same IP address.

After the TCP three way three-way handshake, assuming that both partners
indicate SMC-R capability, the client builds and sends the
SMC Proposal CLC message to the server in exactly the same manner as
it does in the
first contact "first contact" case, and in fact at this point
doesn't know if it's a first contact or a subsequent contact. As in
the first contact "first contact" case, the client sends its Peer peer ID value,
suggested RNIC GID/MAC, MAC/GID, and IP subnet or prefix information.

Upon receiving the client's proposal, the server looks up the
provided peer ID
provided to determine if it already has a usable SMC-R
link group with this peer. If it does already have a usable SMC-R
link group, the server then needs to decide if whether it will use the
existing SMC-R link group, group or create a new link group. For the case
of the new link group
case, group, see 3.5.3. First contact variation: creating Section 3.5.3 ("First Contact Variation:
Creating a parallel link
group, Parallel Link Group") below.

For this discussion discussion, assume that the server decides to use the
existing SMC-R link group for the TCP connection, which is expected
to be the most common case. The server is responsible for making
this decision.
Then the The server then needs to communicate that information
to the client, but it is not necessary to allocate, associate, and
confirm QPs for the chosen SMC-R link. All that remains to be done
is to set up RMB space for this TCP connection.

If one of the RMBs already in use for this SMC-R link group has an
available element that uses the appropriate buffer size, the server
merely chooses one for this TCP connection and then sends an SMC
Accept CLC message, message providing the full RoCE information for the chosen
SMC-R link to the client, using the same format as the SMC Accept CLC
message described in the initial contact section Section 3.5.1 ("First Contact") above.

The server may choose to use the SMC-R link that matches the
suggested MAC/GID provided by the client on in the SMC Proposal for its
RDMA writes but is not obligated to. to do so. The final decision on
which specific SMC-R link to assign a TCP connection to is an
independent server and client decision.

It may be necessary for the server to allocate a new RMB for this
connection. The reasons for this are implementation dependent and
could include: include the following:

o no available space in existing RMB or RMBs, or

o desire to allocate a new RMB that uses a different buffer size
from the ones already created, or

o any other implementation dependent reason. implementation-dependent reason

In this case case, the server will allocate the new RMB and then perform
the flows described in 3.5.5.2.1. Adding Section 3.5.5.2.1 ("Adding a new New RMB to an
SMC-R link
group. Link Group"). Once that processing is complete, the server
then provides the full RoCE information, including the new Rkey, RKey, for
this connection
on in an SMC Confirm CLC message to the client.

3.5.2.2. SMC-R acceptance Acceptance

Upon receiving the SMC Accept CLC message from the server, the client
examines the RoCE information provided by the server to determine if
whether this is a first contact for a new SMC SMC-R link group, group or a
subsequent contact for an existing SMC-R link group. It is a
subsequent contact if the server side server-side peer ID, GID, MAC MAC, and QP
number provided on in the packet match a known SMC-R link, and the "first contact" first
contact flag is not set. If this is not the case, case -- for example example, the
GID and MAC match but the QP is new, new -- then the server is creating a
new, parallel SMC-R link
group group, and this is treated as a first
contact.

A different RMB RToken does not indicate a first contact contact, as the
server may have allocated a new RMB, RMB or may be using several RMBs for
this SMC-R link. The client needs the server's RMB information only
for its RDMA writes to the server, and since there is no requirement
for symmetric RMBs, this information is simply control information
for the RDMA writes on this SMC-R link.

The client must validate that the RMB element being provided by the
server is not in use by another TCP connection on this SMC-R link
group. This validation must validate the new <rtoken, index> across
all known <rtoken, index> on this link group. See 4.4.2. RMB element
reuse Section 4.4.2
("RMB Element Reuse and conflict resolution Conflict Resolution") for the case in which
the server tries to use an RMB element that is already in use on this
link group.

Once the client has determined that this TCP connection is a
subsequent contact over an existing SMC SMC-R link, it performs a similar an RMB
allocation process as similar to what the server did: it either
(1) allocates an element from an RMB already associated with this
SMC-R link, link or it (2) allocates a new RMB and RMB, associates it with this SMC-R link
link, and then chooses an element out of it.

If the client allocates a new RMB for this TCP connection, it
performs the processing described in 3.5.5.2.1. Adding Section 3.5.5.2.1 ("Adding a new New
RMB to an SMC-R link group. Link Group"). Once that processing is complete, the
client provides its full RoCE information for this TCP connection on in
an SMC Confirm CLC message.

Because an SMC-R link with a verified connected QP already exists and
is being reused, there is no need for verification or alternate QP
selection flows or timers.

3.5.2.3. SMC-R confirmation Confirmation

When the server receives the client's SMC Confirm CLC message on a
subsequent contact, it verifies the following:

o the The RMB element provided by the client is not already in use by
another TCP connection on this SMC-R link group (see section
4.4.2. RMB element reuse Section 4.4.2
("RMB Element Reuse and conflict resolution Conflict Resolution") for the case in
which it is).

o The MAC/GID/QP info information provided by the client matches an
active link within the link group. The client is free to select
any valid /
active valid/active link. The client is not required to select the
same link as the server.

If this validation passes, the server stores the client's RMB
information for this connection connection, and the RoCE setup of the TCP
connection is complete.

3.5.2.4. TCP data flow race Data Flow Race with SMC Confirm CLC message Message

On a subsequent contact TCP/IP connection, a peer may send data as
soon as it has received the peer RMB information for the connection.
There are no additional RoCE confirmation flows, since the QPs on the
SMC
SMC-R link are already reliably connected and verified.

In the majority of cases cases, the first data will flow from the client to
the server. The client must send the SMC Confirm CLC message before
sending any connection data over the chosen SMC-R link, however link; however, the
client need not wait for confirmation of this message, and in fact
there will be no such confirmation. Since the server is required to
have the RMB fully set up and ready to receive data from the client
before sending an SMC Accept CLC message, the client can begin
sending data over the SMC-R link immediately upon completing the send
of the SMC Confirm CLC message.

It is possible that data from the client will arrive into at the server
side
server-side RMB before the SMC Confirm CLC message from the client
has been processed. In this case case, the server must handle this race condition,
condition and not provide the arrived TCP data to the socket
application until the SMC Confirm CLC message has been received and
fully processed, opening the socket.

If the server has initial data to send to the client which that is not a
response to the client (this case should be rare), it can send the
data immediately upon receiving and processing the SMC Confirm CLC
message from the client. The client must have opened the TCP socket
to the client application upon sending of the SMC Confirm CLC message so
the client will be ready to process data from the server.

3.5.3. First contact variation: creating Contact Variation: Creating a parallel link group Parallel Link Group

Recall that parallel SMC-R links within an SMC-R link group are not
supported. These are multiple SMC-R links within a link group that
use the same network path. However, multiple SMC-R link groups
between the same peers are supported. This means that if multiple
SMC-R links over the same RoCE path are desired, it is necessary to
use multiple SMC-R link groups. While not a recommended practice,
this could be done for platform specific platform-specific reasons, like QP separation
of different workloads. Only the server can drive the creation of
multiple SMC-R link groups between peers.

At a high level, when the server decides to create an additional SMC-
R
SMC-R link group with a client with which it already has an SMC-R
link group with, group, the flows are basically the same as the normal
"first contact" case described above. The following text provides
more detail and clarification of processing in this case.

When the server receives the SMC Proposal CLC message from the client
and
and, using the GID/MAC info MAC/GID information, determines that it already has an
SMC-R link group with this client, the server can either reuse the
existing SMC-R link group (detailed in 3.5.2. Subsequent contact Section 3.5.2 ("Subsequent
Contact") above) or it
can create a new SMC-R link group in addition to the
existing one.

If the server decides to create a new SMC-R link group, it does the
same processing it would have done for first contact: allocate QP and
RMB resources as well as alternate QP resources, and communicate the
QP and RMB information to the client on in the SMC Accept CLC message
with the "first contact" first contact flag set.

When the client receives the server's SMC Accept CLC message with the
new QP information and the "first contact" flag, first contact flag set, it knows that the
server is creating a new SMC-R link group even though it already has
an SMC-
R SMC-R link group with the server. In this case case, the client will
also allocate a new QP for this new SMC link and SMC-R link, allocate an RMB for this
link
it, and generate an Rkey RKey for it.

Note that multiple SMC-R link groups between the same peers must
access different RMB resources, so new RMBs will be required. Using
the same RMBs that are in use in another SMC-R link group is not
permitted.

The client then associates its new QP with the server's new QP and
sends its SMC Confirm CLC message back to the server providing the
new QP/RMB information information, and then sets its confirmation timer for the
new SMC-R link.

When the server receives the client's SMC Confirm CLC message message, it
associates its QP with the client's QP as learned on from the SMC
Confirm CLC message and sends a confirmation LLC message. The rest
of the flow, with the confirmation QP and setup of additional SMC-R
links, unfolds just like the first contact "first contact" case.

3.5.4. Normal SMC-R link termination SMC-R Link Termination

The normal sockets socket API trigger points are used by the SMC-R layer to
initiate SMC-R connection termination flows. The main design point
for SMC-R normal connection flows is to use the SMC-R protocol to
first shutdown shut down the SMC-R connection and free up any SMC-R RDMA
resources
resources, and then allow the normal TCP connection termination
protocol (i.e. (i.e., FIN processing) to drive cleanup of the TCP
connection that exists on the IP fabric. This design point is very
important in ensuring that RDMA resources such as the RMBEs are only
freed and reused when both SMC-R end points endpoints are completely done with
their RDMA
Write write operations to the partner's RMBE.

When the last TCP connection over an SMC-R link group terminates, the
link group can be terminated. Similar to creation of SMC-R links and
link groups, the primary responsibility for determining that normal
termination is needed and initiating it lies with the server.

Implementations may opt to set timers to keep SMC-R link groups up
for a specified time after the last TCP connection ends, to avoid
churn in cases when where TCP connections come and go regularly.

The link or link group may also be terminated as a result of an
operator a
command initiated command. by the operator. This command can be entered at
either the client or the server. If entered at the client, the
client requests that the server perform link or link group
termination, and the responsibility for doing so ultimately lies with
the server.

When the server determines that the SMC-R link group is to be
terminated, it sends a DELETE LINK LLC message to the client, with a
flag set indicating that all links in the link group are to be
terminated. After receiving confirmation from the adapter that the
DELETE LINK LLC message has been sent, the server can clean up its
end of the link group (QPs, RMBs, etc). etc.). Upon receipt of the DELETE
LINK message from the server, the client must immediately comply and
clean up its end of the link group. Any TCP connections that the
client believes to be active on the link group must be immediately
terminated.

The client can request that the server delete the link group as well.
The client does this by sending a DELETE LINK message to the server server,
indicating that cleanup of all links is requested. The server must
comply by sending a DELETE LINK to the client and processing as
described above. in the previous paragraph. If there are TCP connections
active on the link group when the server receives this request, they
are immediately terminated by sending a RST flow over the IP fabric.

3.5.5. Link group management flows Group Management Flows

3.5.5.1. Adding and deleting links Deleting Links in an SMC-R link group Link Group

The server has the lead role in managing the composition of the link
group. Links are added to the link group by the server. The client
may notify the server of new conditions that may result in the server
adding a new link, but the server is ultimately responsible. In
general
general, links are deleted from the link group by the server, however server;
however, in certain error cases the client may inform the server that
a link must be deleted and treat it as deleted without waiting for
action from the server. These flows are detailed in the following sections
that follow.

3.5.5.1.1. Server initiated Add Link processing Server-Initiated ADD LINK Processing

As described in previous sections, the server initiates an Add Link ADD LINK
exchange to create redundancy in a newly created link group. Once a
link group is established established, the server may also initiate Add Link ADD LINK for
other reasons, including:

o Availability of additional resources on the server host to support
an additional SMC-R link. This may include the provisioning of an
additional RNIC, more storage becoming available to support
additional QP resources, operator command, or any other
implementation dependent
implementation-dependent reason. Note that, that in order to be
available for an existing link group, group a new RNIC must be attached
to the same RoCE LAN that the link group is using.

o Receipt of notification from the client that additional resources
on the client are available to support an additional SMC-R link.
See 3.5.5.1.2. Client initiated Add Link processing.

Server initiated Add Link Section 3.5.5.1.2 ("Client-Initiated ADD LINK Processing").

Server-initiated ADD LINK processing in an established SMC-R link
group is the same as the Add Link ADD LINK processing described in 3.5.1.6.
Second
Section 3.5.1.6 ("Second SMC-R link setup Link Setup"), with the following
changes:

o If an asymmetric SMC-R link already exists in the link group group, a
second asymmetric link will not be created. Only one asymmetric
link is permitted in a link group.

o TCP data flow on already existing already-existing link(s) in the link group is not
halted or otherwise affected during the process of setting up the
additional link.

In no case will the

The server will not initiate Add Link ADD LINK processing if the link group
already has the maximum number of links negotiated by the partners.

3.5.5.1.2. Client initiated Add Link processing Client-Initiated ADD LINK Processing

If an additional RNIC becomes available for an existing SMC-R link
group on the client's side, the client notifies the server by sending
an Add Link ADD LINK request LLC message to the server. Unlike an Add Link ADD LINK
request sent by the server to the client, this Add Link ADD LINK request
merely informs the server that the client has a new RNIC. If the
link group lacks redundancy, redundancy or has redundancy only on an asymmetric
link with a single RNIC on the client side, the server must initiate
an Add Link ADD LINK exchange in response to this message, to create or
improve the link group's redundancy.

If the link group already has symmetric link symmetric-link redundancy but has fewer
than the negotiated maximum number of links, the server may respond
by initiating an Add Link ADD LINK exchange to create a new link using the
client's new resource but is not required to. to do so.

If the link group already has the negotiated maximum number of links,
the server must ignore the client's Add Link ADD LINK request LLC message.

Because the server is not required to respond to the client's Add
Link
ADD LINK LLC message in all cases, the client must not wait for a
response or throw an error if one does not come.

3.5.5.1.3. Server initiated Delete Link Server-Initiated DELETE LINK Processing

Reasons that a server may delete a link include: include the following:

o The link has not been used for TCP connections for an
implementation defined
implementation-defined time interval, and deleting the link will
not cause the link group to lack redundancy redundancy.

o An error Errors in resources supporting the link. link occur. These errors may include
include, but are not limited to: to, RNIC errors, QP errors, and
software errors errors.

o The RNIC supporting this SMC-R link is being taken down, either
because of an error case or because of an operator or software
command.

If a link being deleted is supporting TCP connections, connections and there are
one or more surviving links in the link group, the TCP connections
are moved to the surviving links. For more information on this
processing
processing, see 2.3. SMC-R resilience Section 2.3 ("SMC-R Resilience and load balancing. Load Balancing").

The server deletes a link from the link group by sending a Delete
Link
DELETE LINK request LLC message to the client over any of the usable
links in the link group. Because the Delete Link DELETE LINK LLC message
specifies which link is to be deleted, it may flow over any link in
the link group. The server must not clean up its RoCE resources for
the link until the client responds.

The client responds to the server's Delete Link DELETE LINK request LLC message
by sending the server a Delete Link DELETE LINK response LLC message. The client
must respond positively; it cannot decline to delete the link. Once
the server has received the client's Delete Link DELETE LINK response, both sides
may clean up their resources for the link.

Positive

Either a positive write completion or some other indication from the
RNIC on the client's side is sufficient to indicate to the client
that the server has received the Delete Link DELETE LINK response.

Host X Host Y
+-------------------+ +-------------------+
| +------+ +------+ |
| QP 8 |RNIC 1| SMC-R Link 1 |RNIC 2| QP 9 |
|RToken X| |Failed|<--X----X----X----X-->| | |
| | | | | | |
| \/ +------+ +------+ |
|+--------+ | | |
|| deleted| Deleted| | | |
|| RMB | | | |
|| | | | |
|+--------+ | | |
| /\ +------+ +------+ |
|RToken Z| | | SMC-R Link 2 | | |
| | |RNIC 3|<-------------------->|RNIC 4| |
| QP 64| | | | QP 65 |
| +------+ +------+ |
+-------------------+ +-------------------+

DELETE LINK(Request, LINK(request, link number = 1,
................................................>
reason code = RNIC failure)

DELETE LINK(Response, LINK(response, link number = 1)
<................................................

(note, architecturally

(Note: Architecturally, this exchange can flow over either
SMC-R link but most likely flows over link 2 Link 2, since
the RNIC for link Link 1 has failed) failed.)

Figure 10 Server initiated Delete Link flow 10: Server-Initiated DELETE LINK Flow

3.5.5.1.4. Client initiated Delete Link request Client-Initiated DELETE LINK Request

The client may request that the server delete a link for the same
reasons that the server may delete a link, except for inactivity
timeout.

Because the client depends on the server to delete links, there are
two types of delete requests from client to server:

o Orderly: the The client is requesting that the server delete the link
when able. This would result from an operator command to bring
down the RNIC or some other nonfatal reason. In this case case, the
server is required to delete the link, link but may not do it right
away.

o Disorderly: the The server must delete the link right away, because
the client has experienced a fatal error with the link.

In either case case, the server responds by initiating a Delete Link DELETE LINK
exchange with the client client, as described in the previous section. The
difference between the two is whether the server must do so
immediately or can delay for an opportunity to gracefully delete the
link.

Host X Host Y
+-------------------+ +-------------------+
| +------+ +------+ |
| QP 8 |RNIC 1| SMC-R Link 1 |RNIC 2| QP 9 |
|RToken X| | |<---X--X--X--X--X--X->|Failed| |
| | | | | | |
| \/ +------+ +------+ |
|+--------+ | | |
|| deleted| Deleted| | | |
|| RMB | | | |
|| | | | |
|+--------+ | | |
| /\ +------+ +------+ |
|RToken Z| | | SMC-R Link 2 | | |
| | |RNIC 3|<-------------------->|RNIC 4| |
| QP 64| | | | QP 65 |
| +------+ +------+ |
+-------------------+ +-------------------+

DELETE LINK(Request, LINK(request, link number = 1, disorderly,
<...............................................
reason code = RNIC failure)

DELETE LINK(Request, LINK(request, link number = 1,
................................................>
reason code = RNIC failure)

DELETE LINK(Response, LINK(response, link number = 1)
<................................................

(note, architecturally

(Note: Architecturally, this exchange can flow over either
SMC-R link but most likely flows over link 2 Link 2, since
the RNIC for link Link 1 has failed) failed.)

Figure 11 Client-initiated Delete Link 11: Client-Initiated DELETE LINK Flow

3.5.5.2. Managing multiple Rkeys Multiple RKeys over multiple Multiple SMC-R links Links in a link
group
Link Group

After the initial contact sequence completes and the number of TCP
connections increases increases, it is possible that the SMC peers could add
additional
more RMBs to the Link Group. link group. Recall that each peer independently
manages its RMBs. Also recall that an RMB's RToken is specific to a
QP, which means that when there are multiple SMC-R links in a link
group, each RMB accessed with the link group requires a separate
RToken for each SMC-R link in the group.

Each RMB that is added to a link must be added to all links within
the Link Group. link group. The set of RMBs created for the Link link is called the
"RToken Set". set". The RTokens must be exchanged with the peer. As RMBs
are added and deleted, the RToken Set set must remain in sync.

3.5.5.2.1. Adding a new New RMB to an SMC-R link group Link Group

A new RMB can be added to an SMC-R link group on either the client
side or the server side. When an additional RMB is added to an
existing SMC-
R SMC-R link group, that RMB must be associated with the QPs
for each link in the link group. Therefore Therefore, when an RMB is added to
an SMC-R link group, its RMB RToken for each SMC-R link's QP must be
communicated to the peer.

The tokens for a new RMB added to an existing SMC-R link group are
communicated using "Confirm Rkey" CONFIRM RKEY LLC messages, as shown in Figure 12.
The RToken set is specified as pairs: an SMC SMC-R link number, paired
with the new RMB's RToken over that SMC Link. SMC-R link. To preserve failover
capability, any TCP connection that uses a newly added RMB cannot go
active until all RTokens for the RMB have been communicated for all
of the links in the link group.

Host X Host Y
+-------------------+ +-------------------+
| +------+ +------+ |
| QP 8 |RNIC 1| SMC-R Link 1 |RNIC 2| QP 9 |
|RToken X| | |<-------------------->| | |
| | | | | | |
| \/ +------+ +------+ |
|+--------+ | | |
|| new New | | | |
|| RMB | | | |
|| | | | |
|+--------+ | | |
| /\ +------+ +------+ |
|RToken Z| | | SMC-R Link 2 | | |
| | |RNIC 3|<-------------------->|RNIC 4| |
| QP 64| | | | QP 65 |
| +------+ +------+ |
+-------------------+ +-------------------+

CONFIRM RKEY(Request, RKEY(request, Add,
................................................>
RToken set((Link 1,RToken X),(Link2,RToken X),(Link 2,RToken Z)))

CONFIRM RKEY(Response, RKEY(response, Add,
<................................................
RToken set((Link 1,RToken X),(Link2,RToken X),(Link 2,RToken Z)))

(note, this

(Note: This exchange can flow over either SMC-R link) link.)

Figure 12 12: Add RMB to existing link group Existing Link Group

Implementations may choose to proactively add RMBs to link groups in
anticipation of need. For example, an implementation may add a new
RMB when a certain usage threshold (e.g., percentage used) for all of
its existing RMBs are over a certain threshold
percentage used. has been exceeded.

A new RMB may also be added to an existing link group on an as needed
basis. For as-needed
basis -- for example, when a new TCP connection is added to the link
group but there are no available RMB elements. In this case case, the CLC
exchange is paused while the peer that requires the new RMB adds it.
An example of this is illustrated in figure Figure 13.

Host X -- Server Host Y -- Client
+-------------------+ +-------------------+ +--------------------+
| PeerID Peer ID = PS1 | | PeerID Peer ID = PC1 |
| +------+ +------+ |
| QP 8 |RNIC 1| SMC-R link Link 1 |RNIC 2| QP 64 |
|RToken X| |MAC MA|<-------------------->|MAC MB| | |
| | |GID GA| |GID GB| |RTokenY2| |RToken Y2|
| \/ +------+ +------+ \/ |
|+--------+ | | +--------+ |
|| | | SUBNET Subnet S1 | | New | |
|| RMB | | | | RMB | |
|+--------+ | | +--------+ |
| /\ +------+ +------+ /\ |
| | |RNIC 3| SMC-R link Link 2 |RNIC 4| |RTokenW2| |RToken W2|
| | |MAC MC|<-------------------->|MAC MD| | |
| QP 9 |GID GC| |GID GD| QP65 QP 65 |
| +------+ +------+ |
+-------------------+ +-------------------+ +--------------------+

SYN / SYN-ACK / ACT ACK TCP 3-way three-way handshake with TCP option
<--------------------------------------------------------->

SMC Proposal(PC1,MB,GB,S1)
<--------------------------------------------------------

SMC Accept(PS1,not 1st contact,MA,GA,QP8,RToken=X,RMB elem index)
--------------------------------------------------------->

Confirm Rkey(Request,

CONFIRM RKEY(request, Add,
<........................................................
RToken set((Link1, RToken Y2),{Link2, RToken set((Link 1,RToken Y2),(Link 2,RToken W2)))

Confirm Rkey(Response,

CONFIRM RKEY(response, Add,
........................................................>
RToken set((Link1, RToken Y2),{Link2, RToken set((Link 1,RToken Y2),(Link 2,RToken W2)))

SMC Confirm(PC1,MB,GB,QP64,RToken=Y2, RMB element index)
<--------------------------------------------------------

Legend:
------------ TCP/IP and CLC flows
............ RoCE (LLC) flows

Figure 13 13: Client adds Adds RMB during TCP connection setup Connection Setup

3.5.5.2.2. Deleting an RMB from an SMC-R link group Link Group

Either peer can delete one or more of its RMBs as long as it is not
being used for any TCP connections. Ideally Ideally, an SMC-R peer would use
a timer to avoid freeing an RMB immediately after the last TCP
connection stops using it, to keep the RMB available for later TCP
connections and avoid thrashing with addition and deletion of RMBs.
Once an SMC-R peer decides to delete an RMB, it sends a DELETE RKEY
LLC message to its peer. It can then free the RMB once it receives
a response from the peer. Multiple RMBs can be deleted in a
DELETE RKEY exchange.

Note that in a DELETE RKEY message, it is not necessary to specify
the full RToken for a deleted RMB. The RMB's Rkey RKey over one link in
the link group is sufficient to specify which RMB is being deleted.

Host X Host Y
+-------------------+ +-------------------+
| +------+ +------+ |
| QP 8 |RNIC 1| SMC-R Link 1 |RNIC 2| QP 9 |
|RToken X| | |<-------------------->| | |
| | | | | | |
| \/ +------+ +------+ |
|+--------+ | | |
|| deleted| Deleted| | | |
|| RMB | | | |
|| | | | |
|+--------+ | | |
| /\ +------+ +------+ |
|RToken Z| | | SMC-R Link 2 | | |
| | |RNIC 3|<-------------------->|RNIC 4| |
| QP 9 | | | | |
| +------+ +------+ |
+-------------------+ +-------------------+

DELETE RKEY(Request, Rkey list(Rkey RKEY(request, RKey list(RKey X))
................................................>

DELETE RKEY(Response, Rkey list(Rkey RKEY(response, RKey list(RKey X))
<................................................

(note, this

(Note: This exchange can flow over either SMC-R link) link.)

Figure 14 14: Delete RMB from SMC-R link group Link Group

3.5.5.2.3. Adding a new New SMC-R link Link to a link group Link Group with multiple Multiple RMBs

When a new SMC-R link is added to an existing link group, there could
be multiple RMBs on each side already associated with the link group.
There could also be a different number of RMBs on one side as than on
the other, because each peer manages its RMBs independently. Each of
these RMBs will require a new RToken to be used on the new SMC-R
link, and then those new RTokens must then be communicated to the peer.
This requires two-way communication communication, as the server will have to
communicate its RTokens to the client and vice versa.

RTokens are communicated between peers in pairs. Each RToken pair
consists of:

o The RToken for the RMB, as is already known on an existing SMC-R
link in the link group group.

o The RToken for the same RMB, to be used on the new SMC-R link.

These pairs are required to ensure that each peer knows which RTokens
across QPs are equivalent.

The "Add Link" ADD LINK request and response LLC messages do not have room enough
space to contain any RToken pairs. "Add Link continuation" ADD LINK CONTINUATION LLC
messages are used to communicate these pairs, as shown in Figure 15.
The "Add
Link Continuation" ADD LINK CONTINUATION LLC messages are sent on the same SMC-R
link that the "Add Link" ADD LINK LLC messages were sent over, and in both the "Add
Link"
ADD LINK and the "Add Link Continuation" ADD LINK CONTINUATION LLC messages, messages the first RToken in
each RToken pair will be the RToken for the RMB as known on the SMC-R
link that over which the LLC message is being sent over. sent.

Host X -- Server Host Y -- Client
+-------------------+ +-------------------+
| PeerID Peer ID = PS1 | | PeerID Peer ID = PC1 |
| +------+ +------+ |
| QP 8 |RNIC 1| SMC-R Link 1 |RNIC 2| QP 64 |
|Rkey Set| |MAC MA|
|RKey set| |MAC MA|<-------------------->|MAC MB| |Rkey |RKey set|
|X,Y,Z | |GID GA| |GID GB| |Q,R,S,T |
| \/ +------+ +------+ \/ |
|+--------+ | | +--------+ |
|| 3 RMBs | | | | 4 RMBs | |
|+--------+ | | +--------+ |
| /\ +------+ +------+ /\ |
|Rkey
|RKey set| |RNIC 3| SMC-R Link 2 |RNIC 4| | Rkey RKey set|
|U,V,W | |MAC MC| |MAC MC|<-------------------->|MAC MD| | L,M,N,P |
| QP 9 |GID GC| (being added) |GID GD| QP 65 |
| +------+ +------+ |
+-------------------+ +-------------------+

ADD link LINK request (QP9,MC,GC, link number=2) number = 2)
............................................>

ADD link LINK response (QP65,MD,GD, link number=2) number = 2)
<............................................

ADD link continuation LINK CONTINUATION req(RToken Pairs=((X,U),(Y,V),(Z,W))) pairs=((X,U),(Y,V),(Z,W)))
............................................>

ADD link continuation LINK CONTINUATION rsp(RToken Pairs=((Q,L),(R,M),(S,N),(T,P))) pairs=((Q,L),(R,M),(S,N),(T,P)))
<.............................................

Confirm Link Req/Rsp

CONFIRM LINK req/rsp exchange on link Link 2
<.............................................>

Legend:
------------ TCP/IP and CLC flows
............ RoCE (LLC) flows

Figure 15 15: Exchanging Rkeys RKeys when a new link is added New Link Is Added to a link group Link Group

3.5.5.3. Serialization of LLC exchanges, Exchanges, and collisions Collisions

LLC flows can be divided into two main groups for serializaion serialization
considerations.

The first group is LLC messages that are independent and can flow at
any time. These are one-time, unsolicited messages that either do
not have a required response, response or that have a simple response that does not
interfere with the operations of another group of messages. These
messages are: are as follows:

o TEST LINK from either the client or the server: This message
requires a TEST LINK response to be returned, returned but does not affect
the configuration of the link group or the Rkeys. RKeys.

o ADD LINK from the client to the server: This message is provided
as an "FYI" to the server to let it know that the client has an
additional RNIC available. The server is not required to act upon
or respond to this message.

o DELETE_LINK DELETE LINK from the client to the server: This message informs
the server that either (1) the client has either experienced an error or
problem that requires a link or link group to be terminated, terminated or
that
(2) an operator has commanded that a link or link group be
terminated. The server does not respond directly to the message,
rather message;
rather, it initiates a DELETE LINK exchange as a result of
receiving it.

o DELETE LINK from the server to the client client, with the "delete entire
link group" flag set: This message informs the client that the
entire link group is being deleted.

The second group is LLC messages that are part of an exchange of LLC
messages that affects link group configuration that configuration; this exchange must
complete before another exchange of LLC messages that affects link
group configuration can be processed. When a peer knows that one of
these exchanges is in progress, it must not start another exchange.
These exchanges are: are as follows:

o ADD LINK / ADD LINK response / ADD LINK CONTINUATION / ADD LINK
CONTINUATION response / CONFIRM LINK / CONFIRM LINK RESPONSE: response: This
exchange, by adding a new link, changes the configuration of the
link group.

o DELETE LINK / DELETE LINK response initiated by the server: server,
without the "delete entire link group" flag set: This exchange, by
deleting a link, changes the configuration of the link group.

o CONFIRM RKEY / CONFIRM RKEY response or DELETE RKEY / DELETE RKEY
response: This exchange changes the RMB configuration of the link
group. RKeys can not cannot change while links are being added or deleted
(while an ADD LINK or DELETE LINK is in progress). However,
CONFIRM RKEY and DELETE RKEY are unique in that both the client
and server can independently manage (add or remove) their own
RMBs. This allows each peer to concurrently change their RKeys
and therefore concurrently send CONFIRM RKEY or DELETE RKEY
requests. The concurrent CONFIRM RKEY or DELETE RKEY requests can
be independently processed and do not represent a collision collision.

Because the server is in control of the configuration of the link
group, many timing windows and collisions are avoided avoided, but there are
still some that must be handled.

3.5.5.3.1. Collisions with ADD LINK / CONFIRM LINK exchange Exchange

Colliding LLC message: TEST LINK

Action to resolve: Send immediate TEST LINK reply reply.

Colliding LLC Message: message: ADD LINK from client to server

Action to resolve: Server ignores the ADD LINK message. When
client receives server's ADD LINK, client will consider that
message to be in response to its ADD LINK message and the flow
works. Since both client and server know not to start this
exchange if an ADD LINK operation is already underway, this can
only occur if the client sends this message before receiving the
server's ADD LINK and this message crosses with the server's ADD
LINK message, therefore message; therefore, the server's ADD LINK arrives at the
client immediately after the client sent this message.

Colliding LLC Message: message: DELETE LINK from client to server, specific
link specified

Action to resolve: Server queues the DELETE link LINK message and
processes it after the ADD LINK exchange completes. If it is an
orderly link termination, it can wait until after this exchange
continues. If it is disorderly and the link affected is the one
that the current exchange is using, the server will discover the
outage when a message in this exchange fails.

Colliding LLC Message: message: DELETE LINK from client to server, entire link
group to be deleted

Action to resolve: Immediately clean up the link group group.

Colliding LLC message: CONFIRM RKEY from the client

Action to resolve: Send a negative CONFIRM_RKEY CONFIRM RKEY response to the
client. Once the current exchange finishes, client will have to
recompute its Rkey RKey set to include the new link, link and then start a
new CONFIRM RKEY exchange.

3.5.5.3.2. Collisions during DELETE LINK exchange Exchange

Colliding LLC Message: message: TEST LINK from either peer

Action to resolve: Send immediate TEST LINK response response.

Colliding LLC message: ADD LNK LINK from client to server

Action to resolve: Server queues the ADD LINK and processes it
after the current exchange completes completes.

Colliding LLC message: DELETE LINK from client to server (specific
link)

Action to resolve: Server queues the DELETE link LINK message and
processes it after the current exchange completes. If it is an
orderly link termination, it can wait until after this exchange
continues. If it is disorderly and the link affected is the one
that the current exchange is using, the server will discover the
outage when a message in this exchange fails fails.

Colliding LLC message: DELETE LINK from either client or server,
deleting the entire link group

Action to resolve: immediately Immediately clean up the link group group.

Colliding LLC message: CONFIRM_RKEY CONFIRM RKEY from client to server

3.5.5.3.3. Collisions during CONFIRM_RKEY exchange CONFIRM RKEY Exchange

Colliding LLC Message: message: TEST LINK

Action to resolve: Send immediate TEST LINK reply reply.

Colliding LLC message: ADD LINK from client to server

Action to resolve: Queue the ADD LINK LINK, and process it after the
current exchange completes completes.

Colliding LLC message: ADD LINK from server to client (CONFIRM RKEY
exchange was initiated by the client client, and it crossed with the server
initiating an ADD LINK exchange)

Action to resolve: Process the ADD LINK. Client will receive a
negative CONFIRM RKEY from the server and will have to redo this
CONFIRM RKEY exchange after the ADD LINK exchange completes.

Colliding LLC message: DELETE LINK from client to server, specific
link to be deleted (CONFIRM RKEY exchange was initiated by the server
server, and it crossed with the client's DELETE LINK request request)

Action to resolve: Server queues the DELETE link LINK message and
processes it after the ADD LINK CONFIRM RKEY exchange completes. If it is
an orderly link termination, it can wait until after this exchange
continues. If it is disorderly and the link affected is the one
that the current exchange is using, the server will discover the
outage when a message in this exchange fails.

Colliding LLC message: DELETE LINK from server to client, specific
link deleted (CONFIRM RKEY exchange was initiated by the client client, and
it crossed with the server's DELETE LINK)

Action to resolve: Process the DELETE LINK. Client will receive a
negative CONFIRM RKEY from the server and will have to redo this
CONFIRM RKEY exchange after the ADD LINK exchange completes.

Colliding LLC message: DELETE LINK from either client or server,
entire link group deleted

Action to resolve: immediately Immediately clean up the link group group.

Colliding LLC message: CONFIRM LINK from the peer that did not start
the current CONFIRM LINK exchange

Action to resolve: Queue the request request, and process it after the
current exchange completes.

4. SMC-R memory sharing architecture Memory-Sharing Architecture

4.1. RMB element allocation considerations Element Allocation Considerations

Each TCP connection using SMC-R must be allocated a an RMBE by each SMC-
R
SMC-R peer. This allocation is performed by each end point endpoint
independently to allow each end point endpoint to select an RMBE that best
matches the characteristics on its TCP socket end point. endpoint. The RMBE
associated with a TCP socket endpoint must have a Receive receive buffer that
is at least as large as the TCP receive buffer size in effect for
that connection. The receive buffer size can be determined by what
is specified explicitly by the application using setsockopt() or
implicitly via the system configured system-configured default value. This will allow
sufficient data to be RDMA written RDMA-written by the SMC-R peer to fill an
entire receive buffer
size size's worth of data on a given data flow.
Given that each RMB must have fixed length RMBEs fixed-length RMBEs, this implies that
an SMC-R end point endpoint may need to maintain multiple RMBs of various sizes
for SMC-R connections on a given SMC SMC-R link and can then select an
RMBE that most closely fits a connection.

4.2. RMB and RMBE format Format

An RMB is a virtual memory buffer whose backing real memory is
pinned, which
pinned. The RMB is divided subdivided into a whole number of equal sized equal-sized RMB
Elements (RMBEs). Each RMBE begins with a four byte 4-byte eye catcher for
diagnostic and service purposes, followed by the receive data buffer.
The contents of this diagnostic eyecatcher eye catcher are implementation
dependent and should be used by the local SMC-R peer to check for
overlay errors by verifying an intact eyecatcher eye catcher with every RMBE
access.

The RMBE is a wrapping receive buffer for receiving RDMA writes from
the peer. Cursors, as described below, are exchanged between peers
to manage and track RDMA writes and local data reads from the RMBE
for a TCP connection.

4.3. RMBE control information Control Information

RMBE control information consists of consumer and cursors, producer
cursors, wrap counts, CDC message sequence numbers, control flags
such as urgent data and writer blocked "writer blocked" indicators, and TCP
connection information such as termination flags. This information
is exchanged between SMC-R peers using CDC messages, which are passed
using RDMA
message passing with inline data, with the control information
contained in the inline data. RoCE SendMsg. A TCP/IP stack implementing SMC-R must receive
and store this information in its internal data structures structures, as it is
used to manage the RMBE and its data buffer.

The format and contents of the CDC message is are described in detail in
4.3. RMBE control information.
Appendix A.4 ("Connection Data Control (CDC) Message Format"). The
following is a high level high-level description of what this control
information contains.

o Connection state flags such as sending done, connection closed,
failover data validation, and abnormal close close.

o A sequence number that is managed by the sender. This sequence
number starts at 1, is increased each send, and wraps to 0. This
sequence number tracks the CDC message sent and is not related to
the number of bytes sent. It is used for failover data
validation.

o Producer cursor: a wrapping offset into the receiver's RMBE data
area. Set by the peer that is writing into the RMBE, it points to
where the writing peer will write the next byte of data into an
RMBE. This cursor is accompanied by a wrap sequence number to
help the RMBE owner (the receiver) identify full window size
wrapping writes. Note that this cursor must account for (i.e.,
skip over) the RMBE eyecatcher eye catcher that is in the beginning of the
data area.

o Consumer cursor: a wrapping offset into the receiver's RMBE data
area. Set by the owner of the RMBE (the peer that is reading from
it), this cursor points to the offset of the next byte of data to
be consumed by the peer in its own RMBE. The sender cannot write
beyond this cursor into the receiver's RMBE without causing data
loss. Like the producer cursor, this is accompanied by a wrap
count to help the writer identify full window size wrapping reads.
Note that this cursor must account for (i.e., skip over) the RMBE
eyecatcher
eye catcher that is in the beginning of the data area.

o Data flags such as urgent data, writer blocked indicator, and
cursor update requests.

4.4. Use of RMBEs

4.4.1. Initializing and accessing Accessing RMBEs

The RMBE eyecatcher eye catcher is initialized by the RMB owner prior to
assigning it to a specific TCP connection and communicating its RMB
index to the SMC-R partner. After an RMBE index is communicated to
the SMC-R partner partner, the RMBE can only be referenced in "read only "read-only
mode" by the owner owner, and all updates to it are performed by the remote
SMC-R partner via RDMA write operations.

Initialization of an RMBE must include the following:

o Zeroing out the entire RMBE receive buffer, which helps minimize
data integrity issues (e.g. (e.g., data from a previous connection
somehow being presented to the current connection).

o Setting the beginning RMBE eye catcher. This eye catcher plays an
important role in helping detect accidental overlays of the RMBE.
The RMB owner must should always validate these eye catchers before
each new reference to the RMBE. If the eye catchers are found to
be
corrupted corrupted, the local host must reset the TCP connection
associated with this RMBE and log the appropriate diagnostic
information.

4.4.2. RMB element reuse Element Reuse and conflict resolution Conflict Resolution

RMB elements can be reused once their associated TCP and SMC-R
connections are terminated. Under normal and abnormal SMC-R
connection termination processing processing, both SMC-R peers must explicitly
acknowledge that they are done using an RMBE before that element can
be freed and reassigned to another SMC-R connection instance. For
more details on SMC-R connection termination termination, refer to section Section 4.8.

However, there are some error scenarios where this 2 way two-way explicit
acknowledgement
acknowledgment may not be completed. In these scenarios (mentioned
explicitly elsewhere in this document) scenarios, an RMBE
owner may chose choose to re-
assign reassign this RMBE to a new SMC-R connection
instance on this SMC SMC-R link group. When this occurs occurs, the partner
SMC-R peer must detect this condition during SMC-R rendezvous Rendezvous
processing when presented with an RMBE that it believes is already in
use for a different SMC-R connection. In this case, the SMC-R peer
must abort the existing SMC-R connection associated with this RMBE.
The abort processing
Resets resets the TCP connection (if it is still active)
active), but it must not attempt to perform any RDMA writes to this
RMBE and must also ignore any data sitting in the local RMBE
associated with the existing connection. It then proceeds to free up
the local RMBE and notify the local application that the connection
is being abnormally reset.

The remote SMC-R peer then proceeds to normal processing for this new
SMC-R connection.

4.5. SMC-R protocol considerations Protocol Considerations

The following sections describe considerations for the SMC-R protocol
as compared to the TCP protocol. TCP.

4.5.1. SMC-R protocol optimized window size updates Protocol Optimized Window Size Updates

An SMC-R receiver host sends its Consumer Cursor consumer cursor information to the
sender to convey the progress that the receiving application has made
in consuming the sent data. The difference between the writer's
Producer Cursor
producer cursor and the associated receiver's Consumer Cursor consumer cursor
indicates the window size available for the sender to write into.
This is somewhat similar to TCP window update processing and
therefore has some similar considerations, such as silly window
syndrome avoidance, whereby the TCP protocol has an optimization that minimizes
the overhead of very small, unproductive window size updates
associated with sub-optimal suboptimal socket applications consuming very small amount
amounts of data on every receive() invocation. For SMC-R, the
receiver only updates its Consumer Cursor consumer cursor via a unique CDC message
under the following conditions:

o The current window size (from a sender's perspective) is less than
half of the Receive Buffer space receive buffer space, and the Consumer Cursor consumer cursor update
will result in a minimum increase in the window size of 10% of the
Receive
receive buffer space. Some examples:

a. Receive Buffer buffer size: 64K, Current current window size (from a sender's
perspective): 50K. No need to update the Consumer
Cursor. consumer cursor.
Plenty of space is available for the sender.

b. Receive Buffer buffer size: 64K, Current current window size (from a sender's
perspective): 30K, Current current window size from a receiver's
perspective: 31K. No need to update the Consumer
Cursor; consumer cursor; even
though the sender's window size is < 1/2 of the 64K, the window
update would only increase that by 1K 1K, which is < 1/10th of the
64K buffer size.

c. Receive Buffer buffer size: 64K, Current current window size (from a sender's
perspective): 30K, Current current window size from a receiver's
perspective: 64K. The receiver updates the
Consumer Cursor consumer cursor
(sender's window size is < 1/2 of the 64K, 64K; the window update
would increase that by > 6.4K).

o The receiver must always include a Consumer Cursor consumer cursor update whenever
it sends a CDC message to the partner for another flow (i.e. (i.e., send
flow in the opposite direction). This allows the window size
update to be delivered with no additional overhead. This is
somewhat similar to TCP DelayAck processing and quite effective
for request/response data patterns.

o If a peer has set the B-bit in a CDC message message, then any consumption
of data by the receiver causes a CDC message to be sent sent, updating
the consumer cursor until that a CDC message with that bit cleared is
received from the peer.

o The optimized window size updates are overridden when the sender
sets the Consumer Cursor Update Requested flag in a CDC message to
the receiver. When this indicator is on on, the consumer must send a
Consumer Cursor
consumer cursor update immediately when data is consumed by the
local application or if the cursor has not been updated for a
while (i.e. (i.e., local copy of the consumer cursor does not match the
last consumer cursor value sent to the the partner). This allows the
sender to perform optional diagnostics for detecting a stalled
receiver application (data has been sent but not consumed). It is
recommended that the Consumer Cursor Update Requested flag only be
sent for diagnostic procedures procedures, as it may result in non-optimal
data path performance.

4.5.2. Small data sends Data Sends

The SMC-R protocol makes no special provisions for handling small
data segments sent across a stream socket. Data is always sent if
sufficient window space is available. There In contrast to the TCP Nagle
algorithm, there are no special provisions in SMC-R for coalescing
small data segments, similar to the TCP Nagle
algorithm. segments.

An implementation of SMC-R may can be configured to optimize its sending
processing by coalescing outbound data for a given SMC-R connection
so that it can reduce the number of RDMA write operations it performed
performs, in a similar fashion similar to Nagle's algorithm. However, any
such coalescing would require a timer on the sending host that would
ensure that data was eventually sent. And Also, the sending host would
have to opt out of this processing if Nagle's algorithm had been
disabled (programmatically or via system configuration).

4.5.3. TCP Keepalive processing Processing

TCP keepalive processing allows applications to direct the local
TCP/IP host to periodically "test" the viability of an idle TCP
connection. Since SMC-R connections have both a TCP representation along
with an SMC-R representation representation, there are unique keepalive processing
considerations:

o SMC-R layer SMC-R-layer keepalive processing: If keepalive is enabled for an
SMC-R connection connection, the local host maintains a keepalive timer that
reflects how long an SMC-R connection has been idle. The local
host also maintains a timestamp of last activity for each SMC SMC-R
link (for any SMC-R connection on that link). When it is
determined that an SMC-R connection has been idle longer than the
keepalive
interval interval, the host checks to see whether or not the
SMC-R link has been idle for a duration longer than the keepalive
timeout. If both conditions are met, the local host then performs
a Test Link TEST LINK LLC command to test the viability of the SMC SMC-R link
over the RoCE fabric (RC-QPs). If a Test Link TEST LINK LLC command
response is received within a reasonable amount of time time, then the
link is considered viable viable, and all connections using this link are
considered viable as well. If
however If, however, a response is not
received in a reasonable amount of time or there's a failure in
sending the Test Link TEST LINK LLC command command, then this is considered a
failure in the SMC link SMC-R link, and failover processing to an alternate SMC
SMC-R link must be triggered. If no alternate SMC SMC-R link exists
in the SMC SMC-R link group group, then all of the SMC-R connections on this
link are abnormally terminated by resetting the TCP connections
represented by these SMC-R connections. Given that multiple SMC-R
connections can share the same SMC SMC-R link, implementing an SMC link level SMC-R
link-level probe using the Test Link TEST LINK LLC command will help reduce
the amount of unproductive keepalive traffic for SMC-R
connections; as long as some SMC-R connections on a given SMC SMC-R
link are active (i.e. (i.e., have had I/O activity within the keepalive interval)
interval), then there is no need to perform additional link
viability testing.

o TCP layer keepalives TCP-layer keepalive processing: Traditional TCP "keepalive"
packets are not as relevant for SMC-R connections connections, given that the
TCP path is not used for these connections once the SMC-R
rendezvous
Rendezvous processing is completed. All SMC-R connections by
default have associated TCP connections that are idle. Are TCP
keepalive probes still needed for these connections? There are
two main scenarios to consider:

1. TCP keepalives that are used to determine whether or not the
peer TCP endpoint is still active. This is not needed for
SMC-R
connections connections, as the SMC-R level SMC-R-level keepalives mentioned
above will determine whether or not the remote endpoint
connections are still active.

2. TCP keepalives that are used to ensure that TCP connections
traversing an intermediate proxy maintain an active state. For
example, stateful firewalls typically maintain state
representing every valid TCP connection that traverses the
firewall. These types of firewalls are known to expire idle
connections by removing their state in the firewall to conserve
memory. TCP keepalives are often used in this scenario to
prevent firewalls from timing out otherwise idle connections.
When using SMC-R, both end points endpoints must reside in the same layer
Layer 2 network (i.e. (i.e., the same subnet). As a result,
firewalls can
not cannot be injected in the path between two SMC-R
endpoints. However, other intermediate proxies, such as TCP/IP layer
TCP/IP-layer load
balancers balancers, may be injected in the path of two
SMC-R endpoints. These types of load balancers also maintain
connection state so that they can forward TCP connection
traffic to the appropriate cluster end point. endpoint. When using SMC-R SMC-R,
these TCP connections will appear to be completely idle idle, making
them susceptible to potential timeouts at the LB load-balancing
proxy. As a result, for this scenario, TCP keepalives may
still be relevant.

The following are the TCP level TCP-level keepalive processing requirements for
SMC-R enabled
SMC-R-enabled hosts:

o SMC-R peers should allow TCP keepalives to flow on the TCP path of
SMC-R connections based on existing TCP keepalive configuration
and programming options. However, it is strongly recommended that
platforms provide the ability to specify very granular keepalive
timers (for example, single digit second single-digit-second timers) and should
consider providing a configuration option that limits the minimum
keepalive timer that will be used for TCP layer TCP-layer keepalives on
SMC-R connections. This is important to minimize the amount of
TCP keepalive packets transmitted in the network for SMC-R
connections.

o SMC-R peers must always respond to inbound TCP layer TCP-layer keepalives
(by sending ACKs for these packets) even if the connection is
using SMC-R. Typically, once a TCP connection has completed the
SMC-R rendezvous Rendezvous processing and is using SMC-R for data flows, no
new inbound TCP segments are expected on that TCP connection connection,
other than TCP termination segments (FIN, RST, etc). etc.). TCP
keepalives are the one exception that must be supported. And Also,
since TCP keepalive probes do not carry any application layer data application-layer
data, this has no adverse impact on the application's inbound data
stream.

4.6. TCP connection failover Connection Failover between SMC-R links Links

A peer may change which SMC-R link within a link group it sends its
writes over in the event of a link failure. Since each peer
independently chooses which link to send writes over for a specific
TCP connection, this process is done independently by each peer.

4.6.1. Validating data integrity Data Integrity

Even though RoCE is a reliable transport transport, there is a small subset of
failure modes that could cause unrecoverable loss of data. When an
RNIC acknowledges receipt of an RDMA write to its peer, that creates
a write completion event to the sending peer, which allows the sender
to release any buffers it is holding for that write. In normal
operation and in most failures, this operation is reliable.

However

However, there are failure modes possible in which a receiving RNIC
has acknowledged an RDMA write but then was not able to place the
received data into its host memory, memory -- for example example, a sudden,
disorderly failure of the interface between the RNIC and the host.
While rare, these types of events must be guarded against to ensure
data integrity. The process for switching SMC-R links during failover that
is
failover, as described in this section section, guards against this possibility,
possibility and is mandatory.

Each peer must track the current state of the CDC sequence numbers
for a TCP connection. The sender must keep track of SS, which is the sequence
number of the CDC message that described the last write acknowledged
by the peer RNIC. RNIC, or Sequence Sent (SS). In other words, SS
describes the last write that the sender believes its peer has
successfully received. The receiver must keep track of SR, the sequence
number of the CDC message that described the last write that it has
successfully received,
i.e., received (i.e., the data has been successfully placed
into an RMBE. RMBE), or Sequence Received (SR).

When an RNIC fails and the sender changes SMC-R links, the sender
must first send a CDC message with the 'F' flag F-bit (failover validation
indicator; see Appendix A.4) set over the new SMC-
R SMC-R link. This is
the failover data validation message. The sequence number in this
CDC message is equal to SS. The CDC message key, the length, and the
SMC-R alert token are the only other fields in this CDC message that
are significant. No reply is expected from this validation message,
and once the sender has sent it, the sender may resume sending on the
new SMC-R link as described in Section 4.6.2. below

Upon receipt of the failover validation message, the receiver must
verify that its SR value for the TCP connection is equal to or
greater than the sequence number in the failover validation message.
If so, no further action is required required, and the TCP connection resumes
on the new SMC-R link. If SR is less than the sequence number value
in the validation message, data has been lost lost, and the receiver must
immediately reset the TCP connection.

4.6.2. Resuming the TCP connection Connection on a new SMCR link New SMC-R Link

When a connection is moved to a new SMC-R link and the failover
validation message has been sent, the sender can immediately resume
normal transmission. In order to preserve the application message
stream
stream, the sender must replay any RDMA writes (and their associated
CDC messages) that were in progress or failed when the previous SMC-R
link failed, before sending new data on the new SMC-R link. The
sender has two options for accomplishing this:

o Preserve the sequence numbers "as is": Retry all failed and
pending operations as they were originally done, including
reposting all associated RDMA write operations and their
associated CDC messages without making any changes. Then resume
sending new data using new sequence numbers.

o Combine pending messages and possibly add new data: Combine failed
and pending messages into a single new write with a new sequence
number. This allows the sender to combine pending messages into
fewer operations. As a further optimization optimization, this write can also
include new data, as long as all failed and pending data is are also
included. If this approach is taken, the sequence number must be
increased beyond the last failed or pending sequence number.

4.7. RMB data flows Data Flows

The following sections describe the RDMA wire flows for the SMC-R
protocol after a TCP connection has switched into SMC-R mode (i.e. (i.e.,
SMC-R rendezvous Rendezvous processing is complete and a pair of RMB elements
has been assigned and communicated by the SMC-R peers). The ladder
diagrams below include the following:

o RMBE control information kept by each peer. Only a subset of the
information is depicted, specifically only the fields that reflect
the stream of data written by Host A and read by Host B.

o Time line 0-x that 0-x, which shows the wire flows in a time relative fashion time-relative
fashion.

o Note that RMBE control information is only shown in a time
interval if its value changed (otherwise (otherwise, assume that the value is
unchanged from the previously depicted value) value).

o The local copy of the producer cursors and consumer cursors that
is maintained by each host is not depicted in these figures. Note
that the cursor values in the diagram reflect the necessity of
skipping over the eyecatcher eye catcher in the RMBE data area. They start
and wrap at 4, not 0.

4.7.1. Scenario 1: Send flow, window size unconstrained Flow, Window Size Unconstrained

SMC Host A SMC HostB Host B
RMBE A Info RMBE B Info
(Consumer Cursors) (Producer Cursors)
Cursor Wrap Seq# Time Time Cursor Wrap Seq# Flags
4 0 0 0 4 0 0
0 0 1 ---------------> 1 0 0 0
RDMA-WR Data
(4:1003)
4 0 2 ...............> 2 1004 0 0
CDC Message

Figure 16 16: Scenario 1: Send flow, window size unconstrained Flow, Window Size Unconstrained

Scenario assumptions:

o Kernel implementation implementation.

o New SMC-R connection, connection; no data has been sent on the connection connection.

o Host A: Application issues send for 1,000 1000 bytes to Host B B.

o Host B: RMBE receive buffer size is 10,000, 10,000; application has issued
a recv for 10,000 bytes bytes.

Flow description:

1. Application The application issues a send() for 1,000 bytes, 1000 bytes; the SMC-R layer
copies data into a kernel send buffer. It then schedules an RDMA
write operation to move the data into the peer's RMBE receive
buffer, at relative position 4-1003 (to skip the four byte eyecatcher 4-byte
eye catcher in the RMBE data area). Note that no immediate data
or alert (i.e. (i.e., interrupt) is provided to host Host B for this RDMA
operation.

2. Host A sends a CDC message to update the Producer Cursor producer cursor to
byte 1004. This CDC message will deliver an interrupt to Host B.
At this point, the SMC-R layer can return control back to the
application. Host B, once notified of the completion of the
previous RDMA operation, locates the RMBE associated with the RMBE
alert token that was included in the message and proceeds to
perform normal receive side receive-side processing, waking up the suspended
application read thread, copying the data into the application's
receive buffer, etc. It will use the Producer
Cursor producer cursor as an
indicator of how much data is available to be delivered to the
local application. After this processing is complete, the SMC-R
layer will also update its local Consumer
Cursor consumer cursor to match the Producer Cursor (i.e.
producer cursor (i.e., indicating that all data has been
consumed). Note that a message to the peer updating the Consumer Cursor consumer
cursor is not needed at this time time, as the window size if is
unconstrained (> 1/2 of the receive buffer size). The window size
is calculated using by taking the difference between the Producer producer cursor
and the Consumer cursors consumer cursor in the RMBEs
(10,000-1,004=8,996). (10,000 - 1004 = 8996).

4.7.2. Scenario 2: Send/Receive flow, window unconstrained Flow, Window Size Unconstrained

0 0 3 <-------------- 3 1004 0 0
RDMA-WR Data
(4:503)
1004 0 4 <.............. 4 1004 0 0
CDC Message

Figure 17 17: Scenario 2: Send/Recv flow, window size unconstrained Send/Receive Flow, Window Size Unconstrained

Scenario assumptions:

o New SMC-R connection, connection; no data has been sent on the connection connection.

o Host A: Application issues send for 1,000 1000 bytes to Host B B.

o Host B: RMBE receive buffer size is 10,000, 10,000; application has
already issued a recv for 10,000 bytes. Once the receive is
completed, the application sends a 500 byte 500-byte response to Host A.

Flow description:

1. Application The application issues a send() for 1,000 bytes, 1000 bytes; the SMC-R layer
copies data into a kernel send buffer. It then schedules an RDMA
write operation to move the data into the peer's RMBE receive
buffer, at relative position 4-1003. Note that no immediate data
or alert (i.e. (i.e., interrupt) is provided to host Host B for this RDMA
operation.

3. Host B, once notified of the receipt of the previous CDC message,
locates the RMBE associated with the RMBE alert token and proceeds
to perform normal receive side receive-side processing, waking up the suspended
application read thread, copying the data into the application's
receive buffer, etc. After this processing is complete, the SMC-R
layer will also update its local Consumer
Cursor consumer cursor to match the Producer Cursor (i.e.
producer cursor (i.e., indicating that all data has been
consumed). Note that an update of the Consumer
Cursor consumer cursor to the peer
is not needed at this time time, as the window size is unconstrained
(> 1/2 of the receive buffer size). The application then performs
a send() for 500 bytes to Host A. The SMC-R layer will copy the
data into a kernel buffer and then schedule an RDMA Write write into the
partner's RMBE receive buffer. Note that this RDMA write
operation includes no immediate data or notification to Host A.

4. Host B sends a CDC message to update the partner's RMBE Control control
information with the latest Producer Cursor producer cursor (set to 503 and not
shown in the diagram above) and to also inform the peer that the
Consumer Cursor
consumer cursor value is now 1004. It also updates the local
Current Consumer Cursor
current consumer cursor and Last Sent Consumer Cursor the last sent consumer cursor to 1004.
This CDC message includes notification notification, since we are updating our Producer Cursor which
producer cursor; this requires attention by the peer host.

4.7.3. Scenario 3: Send Flow, window constrained Window Size Constrained

SMC Host A SMC HostB Host B
RMBE A Info RMBE B Info
(Consumer Cursors) (Producer Cursors)
Cursor Wrap Seq# Time Time Cursor Wrap Seq# Flags
4 0 0 0 4 0 0
4 0 1 ---------------> 1 4 0 0
RDMA-WR Data
(4:3003)
4 0 2 ...............> 2 3004 0 0
CDC Message
4 0 3 3 3004 0 0
4 0 4 ---------------> 4 3004 0 0
RDMA-WR Data
(3004:7003)
4 0 5 ................> 5 7004 0 0
CDC Message
7004 0 6 <................ 6 7004 0 0
CDC Message

Figure 18 18: Scenario 3: Send Flow, window size constrained Window Size Constrained
Scenario assumptions:

o New SMC-R connection, connection; no data has been sent on this connection connection.

o Host A: Application issues send for 3,000 3000 bytes to Host B and then
another send for 4,000 4000 bytes.

o Host B: RMBE receive buffer size is 10,000. Application has
already issued a recv for 10,000 bytes bytes.

Flow description:

1. Application The application issues a send() for 3,000 bytes, 3000 bytes; the SMC-R layer
copies data into a kernel send buffer. It then schedules an RDMA
write operation to move the data into the peer's RMBE receive
buffer, at relative position 4-3003. Note that no immediate data
or alert (i.e. (i.e., interrupt) is provided to host Host B for this RDMA
operation.

2. Host A sends a CDC message to update its Producer Cursor producer cursor to
byte 3003. This CDC message will deliver an interrupt to Host B.
At this point, the SMC-R layer can return control back to the
application.

3. Host B, once notified of the receipt of the previous CDC message,
locates the RMBE associated with the RMBE alert token and proceeds
to perform normal receive side receive-side processing, waking up the suspended
application read thread, copying the data into the application's
receive buffer, etc. After this processing is complete, the SMC-R
layer will also update its local Consumer
Cursor consumer cursor to match the Producer Cursor (i.e.
producer cursor (i.e., indicating that all data has been
consumed). It will not however not, however, update the partner with this information
information, as the window size is not constrained
(10000-3000=7000
(10,000 - 3000 = 7000 bytes of available space). The application
on Host B also issues a new recv() for 10,000. 10,000 bytes.

4. On Host A, the application issues a send() for 4,000 4000 bytes. The SMC-
R
SMC-R layer copies the data into a kernel buffer and schedules an
async RDMA write into the peer's RMBE receive buffer at relative
position 3003-7004. Note that no alert is provided to host Host B for
this flow.

5. Host A sends a CDC message to update the Producer Cursor producer cursor to
byte 7004. This CDC message will deliver an interrupt to Host B.
At this point, the SMC-R layer can return control back to the
application.

6. Host B, once notified of the receipt of the previous CDC message,
locates the RMBE associated with the RMBE alert token and proceeds
to perform normal receive side receive-side processing, waking up the suspended
application read thread, copying the data into the application's
receive buffer, etc. After this processing is complete, the SMC-R
layer will also update its local Consumer
Cursor consumer cursor to match the Producer Cursor (i.e.
producer cursor (i.e., indicating that all data has been
consumed). It will then determine whether or not it needs to
update the Consumer Cursor consumer cursor to the peer. The available window size
is now 3,000 3000 (10,000 - (Producer Cursor (producer cursor - Last Sent
Consumer Cursor)) last sent consumer
cursor)), which is < 1/2 of the receive buffer size
(10,000/2=5,000)
(10,000/2 = 5000), and the advance of the window size is > 10% of
the windows window size (1,000). Therefore (1000). Therefore, a CDC message is issued to
update the Consumer Cursor consumer cursor to peer Peer A.

4.7.4. Scenario 4: Large send, flow control, full window size writes Send, Flow Control, Full Window Size Writes

SMC Host A SMC HostB Host B
RMBE A Info RMBE B Info
(Consumer Cursors) (Producer Cursors)
Cursor Wrap Seq# Time Time Cursor Wrap Seq# Flags
1004 1 0 0 1004 1 0
1004 1 1 ---------------> 1 1004 1 0
RDMA-WR Data
(1004:9999)
1004 1 2 ---------------> 2 1004 1 0
RDMA-WR Data
(4:1003)
1004 1 3 ...............> 3 1004 2 Wrt
CDC Message Blk

1004 2 4 <............... 4 1004 2 Wrt
CDC Message Blk

1004 2 5 ---------------> 5 1004 2 Wrt
RDMA-WR Data Blk
(1004:9999)
1004 2 6 ---------------> 6 1004 2 Wrt
RDMA-WR Data Blk
(4:1003)
1004 2 7 ...............> 7 1004 3 Wrt
CDC Message Blk

1004 3 8 <............... 8 1004 3 Wrt
CDC Message Blk

Figure 19 19: Scenario 4: Large send, flow control, full window size
writes Send, Flow Control,
Full Window Size Writes
Scenario assumptions:

o Kernel implementation implementation.

o Existing SMC-R connection, Host B's receive window size is fully
open(Peer Consumer Cursor
open (peer consumer cursor = Peer Producer Cursor). peer producer cursor).

o Host A: Application issues send for 20,000 bytes to Host B B.

o Host B: RMB RMBE receive buffer size is 10,000, 10,000; application has issued
a recv for 10,000 bytes bytes.

Flow description:

1. Application The application issues a send() for 20,000 bytes, bytes; the SMC-R layer
copies data into a kernel send buffer (assumes that send buffer
space of 20,000 is available for this connection). It then
schedules an RDMA write operation to move the data into the peer's
RMBE receive buffer, at relative position 1004-9999. Note that no
immediate data or alert (i.e. (i.e., interrupt) is provided to host Host B
for this RDMA operation.

2. Host A then schedules an RDMA write operation to fill the
remaining 1000 bytes of available space in the peer's RMBE receive
buffer, at relative position 4-1003. Note that no immediate data
or alert (i.e. (i.e., interrupt) is provided to host Host B for this RDMA
operation. Also note that an implementation of SMC-R may optimize
this processing by combining step steps 1 and 2 into a single
RDMA Write write operation (with 2 two different data sources).

3. Host A sends a CDC message to update the Producer Cursor producer cursor to
byte 1004. Since the entire receive buffer space is filled, the
Producer Writer Blocked
producer writer blocked flag (WrtBlk (the "Wrt Blk" indicator above) (flag) in
Figure 19) is set and the Producer Window Wrap Sequence Number (Producer WrapSeq#
above) producer cursor wrap sequence number
(the producer "Wrap Seq#" in Figure 19) is incremented. This CDC
message will deliver an interrupt to Host B. At this point, the
SMC-R layer can return control back to the application.

4. Host B, once notified of the receipt of the previous CDC message,
locates the RMBE associated with the RMBE alert token and proceeds
to perform normal receive side receive-side processing, waking up the suspended
application read thread, copying the data into the application's
receive buffer, etc. In this scenario, Host B notices that the Producer Cursor
producer cursor has not been advanced (same value as Consumer Cursor), the consumer
cursor); however, it notices that the Producer
Window Wrap Size Sequence producer cursor wrap
sequence number is different from its local value (1) (1), indicating
that a full window of new data is available. All of the data in
the receive buffer can be processed, with the first segment
(1004-9999) followed by the second segment (4-
1003). (4-1003). Because the Producer Writer Blocked
producer writer blocked indicator was set, Host B schedules a CDC
message to update its latest information to the peer: Consumer Cursor consumer
cursor (1004), Consumer Window Wrap Size
Sequence Number (2: the consumer cursor wrap sequence number (the current Producer Window Wrap Sequence
Number
value of 2 is used).

5. Host A, upon receipt of the CDC message message, locates the TCP
connection associated with the alert token, and token and, upon examining the
control information provided provided, notices that Host B has consumed all
of the data (based on the Consumer Cursor consumer cursor and the
Consumer Window Wrap Size Sequence consumer cursor
wrap sequence number) and initiates the next RDMA write to fill
the receive buffer at offset 1003-9999.

6. Host A then moves the next 1000 bytes into the beginning of the
receive buffer (4-1003) by scheduling an RDMA write operation.
Note that at this point there are still 8 bytes remaining to be
written.

7. Host A then sends a CDC message to set the Producer Writer
Blocked producer writer blocked
indicator and to increment the Producer Window Wrap Size
Sequence Number producer cursor wrap sequence
number (3).

8. Host B, upon notification notification, completes the same processing as step 4
above, including sending a CDC message to update the peer to
indicate that all data has been consumed. At this point point, Host A
can write the final 8 utes bytes to host Host B's RMBE into
positions 1004-
1011 1004-1011 (not shown).

4.7.5. Scenario 5: Send flow, urgent data, window size unconstrained Flow, Urgent Data, Window Size Unconstrained

SMC Host A SMC HostB Host B
RMBE A Info RMBE B Info
(Consumer Cursors) (Producer Cursors)
Cursor Wrap Seq# Time Time Cursor Wrap Seq# Flag
1000 1 0 0 1000 1 0
1000 1 1 ---------------> 1 1000 1 0
RDMA-WR Data
(1000:1499)
1000 1 2 ...............> 2 1500 1 UrgP
CDC Message UrgA

1500 1 3 <............... 3 1500 1 UrgP
CDC Message UrgA

1500 1 4 ---------------> 4 1500 1 UrgP
RDMA-WR Data UrgA
(1500:2499)
1500 1 5 ...............> 5 2500 1 0
CDC Message

Figure 20 20: Scenario 5: send Send Flow, urgent data, window size open Urgent Data, Window Size Open

Scenario assumptions:

o Kernel implementation implementation.

o Existing SMC-R connection, connection; window size open, open (unconstrained); all
data has been consumed by receiver.

o Host A: Application issues send for 500 bytes with urgent data
indicator (OOB) (out of band) to Host B, then sends 1000 bytes of
normal data data.

o Host B: RMBE Receive receive buffer size is 10,000, 10,000; application has issued
a recv for 10,000 bytes and is also monitoring the socket for
urgent data data.

Flow description:

1. Application The application issues a send() for 500 bytes of urgent data. data; the
SMC-R layer copies data into a kernel send buffer. It then
schedules an RDMA write operation to move the data into the peer's
RMBE receive buffer, at relative position 1000-1499. Note that no
immediate data or alert (i.e. (i.e., interrupt) is provided to host Host B
for this RDMA operation.

2. Host A sends a CDC message to update its Producer Cursor producer cursor to
byte 1500 and to turn on the Producer producer Urgent Data Pending (UrgP)
and Urgent Data Present (UrgA) flags. This CDC message will
deliver an interrupt to Host B. At this point, the SMC-R layer
can return control back to the application.

3. Host B, once notified of the receipt of the previous CDC message,
locates the RMBE associated with the RMBE alert token, notices
that the Urgent Data Pending flag is on on, and proceeds with Out of Band out-of-
band socket API notification. For notification -- for example, satisfying any
outstanding select() or poll() requests on the socket by
indicating that urgent data is pending (i.e. (i.e., by setting the
exception bit on). The Urgent Data Present urgent data present indicator allows
Host B to also determine the position of the urgent data
(Producer (the
producer cursor points one 1 byte beyond the last byte of urgent
data). Host B can then perform normal receive side receive-side processing
(including specific urgent data processing), copying the data into
the application's receive buffer, etc. Host B then sends a CDC
message to update the partner's RMBE Control control area with its latest Consumer Cursor
consumer cursor (1500). Note that this CDC message must occur occur,
regardless of the current local window size that is available.
The partner host (Host A) cannot initiate any additional RDMA
writes until acknowledgement it receives acknowledgment that the urgent data has
been processed (or at least processed/remembered at the SMC-R
layer).

4. Upon receipt of the message, Host A wakes up, sees that the peer
consumed all data up to and including the last byte of Urgent
data urgent
data, and now resumes sending any pending data. In this case, the
application had previously issued a send for 1000 bytes of normal data
data, which would have been copied in the send buffer buffer, and control
would have been returned to the application. Host A now initiates a
an RDMA write to move that data to the Peer's peer's receive buffer at
position 1500-2499.

5. Host A then sends a CDC message with inline data to update its
Producer Cursor producer cursor
value (2500) and to turn off the Urgent Data Pending and Urgent
Data Present flags. Host B wakes up, processes the new data
(resumes application, copies data into the application receive buffer)
buffer), and then proceeds to update the
Local local current consumer
cursor (2500). Given that the window size is unconstrained unconstrained, there
is no need for Consumer Cursor a consumer cursor update in the peer's RMBE.

4.7.6. Scenario 6: Send flow, urgent data, window size closed Flow, Urgent Data, Window Size Closed

SMC Host A SMC HostB Host B
RMBE A Info RMBE B Info
(Consumer Cursors) (Producer Cursors)
Cursor Wrap Seq# Time Time Cursor Wrap Seq# Flag
1000 1 0 0 1000 2 Wrt
Blk

1000 1 1 ...............> 1 1000 2 Wrt
CDC Message Blk
UrgP

1000 2 2 <............... 2 1000 2 Wrt
CDC Message Blk
UrgP

1000 2 3 ---------------> 3 1000 2 Wrt
RDMA-WR data l Data Blk
(1000:1499) UrgP

1000 2 4 ...............> 4 1500 2 UrgP
CDC Message UrgA

1500 2 5 <............... 5 1500 2 UrgP
CDC Message UrgA

1500 2 6 ---------------> 6 1500 2 UrgP
RDMA-WR data l Data UrgA
(1500:2499)
1000 2 7 ...............> 7 2500 2 0
CDC Message

Figure 21 21: Scenario 6: Send flow, urgent data, window size closed Flow, Urgent Data, Window Size Closed

Scenario assumptions:

o Kernel implementation implementation.

o Existing SMC-R connection, connection; window size closed, closed; writer is blocked.

o Host A: Application issues send for 500 bytes with urgent data
indicator (OOB) (out of band) to Host B, then sends 1000 bytes of
normal data.

o Host B: RMBE Receive receive buffer size is 10,000, 10,000; application has no
outstanding recv() (for normal data) and is monitoring the socket
for urgent data.

Flow description:

1. Application The application issues a send() for 500 bytes of urgent data. data; the
SMC-R layer copies data into a kernel send buffer (if available).
Since the writer is blocked (window size closed) closed), it cannot send
the data immediately. It then sends a CDC message to notify the
peer of the Urgent Data Pending (UrgP)indicator (UrgP) indicator (the Writer
Blocked writer
blocked indicator remains on as well). This serves as a signal to
Host B that urgent data is pending in the stream. Control is also
returned to the application at this point.

2. Host B, once notified of the receipt of the previous CDC message,
locates the RMBE associated with the RMBE alert token, notices
that the Urgent Data Pending flag is on on, and proceeds with Out of Band out-of-
band socket API notification. For notification -- for example, satisfying any
outstanding select() or poll() requests on the socket by
indicating that urgent data is pending (i.e. (i.e., by setting the
exception bit on). At this point point, it is expected that the
application will enter urgent data mode processing, expeditiously
processing all normal data (by issuing recv API calls) so that it
can get to the urgent data byte. Whether the application has this
urgent mode processing or not, at some
point point, the application will
consume some or all of the pending data in the receive buffer.
When this occurs, Host B will also send a CDC message with inline data to update
its Consumer
Cursor consumer cursor and Consumer Window Wrap Sequence Number consumer cursor wrap sequence number to
the peer. In the example above, a full window window's worth of data was
consumed.

3. Host A, once awakened by the message message, will notice that the window
size is now open on this connection (based on the Consumer
Cursor consumer cursor
and the Consumer Window Wrap Sequence Number consumer cursor wrap sequence number, which now matches
the Producer Window Wrap Sequence Number) producer cursor wrap sequence number) and resume sending of
the urgent data segment by scheduling an RDMA write into relative
position 1000-1499.

4. Host A the then sends a CDC message to advance its Producer Cursor producer cursor
(1500) and to also notify Host B of the Urgent Data Present (UrgA)
indicator (and turn off the Writer Blocked writer blocked indicator). This
signals to Host B that the urgent data is now in the local receive
buffer and that the Producer Cursor producer cursor points to the last byte of
urgent data.

5. Host B wakes up, processes the urgent data and data, and, once the urgent
data is consumed consumed, sends a CDC message with inline data to update its Consumer Cursor consumer
cursor (1500).

6. Host A wakes up, sees that Host B has consumed the sequence number
associated with the urgent data data, and then initiates the next RDMA
write operation to move the 1000 bytes associated with the next
send() of normal data into the peer's receive buffer at
position (1500-2499). 1500-2499. Note that send() the send API would have likely
completed earlier in the process by copying the 1000 bytes into a
send buffer and returning back to the application application, even though we
could not send any new data until the urgent data was processed
and acknowledged by Host B.

7. Host A sends a CDC message to advance its Producer Cursor producer cursor to 2500
and to reset the Urgent Data Pending and Urgent Data Present
flags. Host B wakes up and processes the inbound data.

4.8. Connection termination Termination

Just as SMC-R connections are established using a combination of TCP
connection establishment flows and SMC-R protocol flows, the
termination of SMC-R connections also uses a similar combination of
SMC-R protocol termination flows and normal TCP protocol connection
termination flows. The following sections describe the SMC-R
protocol normal and abnormal connection termination flows.

4.8.1. Normal SMC-R connection termination flows Connection Termination Flows

Normal SMC-R connection flows are triggered via the normal stream
socket API semantics, namely by the application issuing a close() or
shutdown() API. Most applications, after consuming all incoming data
and after sending any outbound data data, will then issue a close() API to
indicate that they are done both sending and receiving data. Some
applications, typically a small percentage, make use of the
shutdown() API that allows then them to indicate that the application is
done sending data, receiving data data, or both sending and receiving
data. The main use of this API is scenarios where a TCP application
wants to alert its partner end point endpoint that it is done sending data, yet data but
is still receiving data on its socket (shutdown for Write). write). Issuing
shutdown
shutdown() for both sending and receiving data is really no different
than issuing a close() and can therefore be treated in a similar
fashion. Shutdown for read is typically not a very useful operation
and in normal circumstances does not trigger any network flows to
notify the partner TCP end point endpoint of this operation.

These same trigger points will be used by the SMC-R layer to initiate
SMC-R connections connection termination flows. The main design point for SMC-R
normal connection flows is to use the SMC-R protocol to first
shutdown shut
down the SMC-R connection and free up any SMC-R RDMA resources resources, and
then allow the normal TCP connection termination protocol (i.e. (i.e., FIN
processing) to drive cleanup of the TCP connection. This design
point is very important in ensuring that RDMA resources such as
the RMBEs are only freed and reused when both SMC-R end points endpoints
are completely done with their RDMA Write write operations to the
partner's RMBE.

1
+-----------------+
|-------------->| CLOSED |<-------------|
3D | | | | 4D
| +-----------------+ |
| | |
| 2 | |
| V |
+----------------+ +-----------------+ +----------------+
|AppFinCloseWait | | ACTIVE | |PeerFinCloseWait|
| | | | | |
+----------------+ +-----------------+ +----------------+
| | | |
| Active Close | 3A | 4A | Passive Close |
| V | V |
| +--------------+ | +-------------+ |
|--<----|PeerCloseWait1| | |AppCloseWait1|--->----|
3C | | | | | | | 4C
| +--------------+ | +-------------+ |
| | | | |
| | 3B | 4B | |
| V | V |
| +--------------+ | +-------------+ |
|--<----|PeerCloseWait2| | |AppCloseWait2|--->----|
| | | | |
+--------------+ | +-------------+
|
|

Figure 22 22: SMC-R connection states Connection States

Figure 23 22 describes the states that an SMC-R connection typically
goes through. Note that there are variations to these states that
can occur when an SMC-R connection is abnormally terminated, similar
in a way to when a TCP connection is reset. The following are the high
level
high-level state transitions for an SMC-R connection:

1. An SMC-R connection begins in the Closed state. This state is
meant to reflect an RMBE that is not currently in use (was
previously in use but no longer is is, or one that was never
allocated) allocated).

2. An SMC-R connection progresses to the Active state once the SMC-
R rendezvous SMC-R
Rendezvous processing has successfully completed, RMB element
indices have been exchanged exchanged, and SMC-R links have been activated.
In this state, the TCP connection is fully established, rendezvous
processing has been completed completed, and SMC-R peers can begin the
exchange of data via RDMA.

3. Active close processing (on the SMC-R peer that is initiating the
connection termination) termination).

A. When an application on one of the SMC-R connection peers issues
a close() or shutdown(write close(), a shutdown() for write, or both) a shutdown() for both
read and write, the SMC-R layer on that host will initiate
SMC-R connection termination processing. First First, if a close()
or shutdown(both) is issued issued, it will check to see that there's
no data in the local RMB element that has not been read by the
application. If unread data is detected, the SMC-R connection
must be abnormally reset - reset; for more detail details on this this, refer to "SMC-R connection reset".
Section 4.8.2 ("Abnormal SMC-R Connection Termination Flows").
If no unread data is pending, it then checks to see whether or
not any outstanding data is waiting to be written to the peer peer,
or if any outstanding RDMA writes for this SMC-R connection
have not yet completed. If either of these two scenarios are is
true, an indicator that this connection is in a pending close
state is saved in internal data structures representing this
SMC-R connection connection, and control is returned to the application.
If all data to be written to the partner has
completed completed, this
peer will send a CDC message to notify the peer of either the
PeerConnectionClosed indicator (close or shutdown for both was
issued) or the PeerDoneWriting indicator. This will provide stimulus an
interrupt to the inform that partner SMC-R peer that the connection
is terminating. At this point point, the local side of the SMC-R
connection transitions in the PeerCloseWait1 state state, and control
can be returned to the application. If this process could not
be completed synchronously (close (the pending close condition
mentioned above) above), it is completed when all RDMA writes for data
and control cursors have been completed.

B. At some point point, the SMC-R peer application (passive close) will
consume all incoming data, realize that that partner is done
sending data on this connection connection, and proceed to initiate its
own close of the connection once it has completed sending all
data from its end. The partner application can initiate this
connection termination processing via a close() or shutdown()
APIs. If the application does so by issuing a shutdown() for
write, then the partner SMC-R layer will send a CDC message to
notify the peer (active (the active close side) of the PeerDoneWriting
indicator. When the "active close" SMC-R peer wakes up as a
result of the previous CDC message, it will notice that the
PeerDoneWriting indicator is now on and transition to the
PeerCloseWait2 state. This state indicates that the peer is
done sending data and may still be reading data. The At this
point, the "active close" peer will also at this point need to ensure that
any outstanding recv() calls for this socket are woken up and
remember that that no more data is forthcoming on this connection
(in case the local connection was shutdown() for write only) only).

C. This flow is a common transition from 3a 3A or 3b 3B above. When the
SMC-R peer (passive close) consumes all data, data and updates all
necessary cursors to the peer peer, and the application closes its
socket (close or shutdown for both) both), it will send a CDC message
to the peer (the active close side) with the
PeerConnectionClosed indicator set. At this point point, the
connection can transition back to the Closed state if the local
application has already closed (or issued shutdown for both)
the socket. Once in the Closed state, the RMBE can now be
safely be reused for a new SMC-R connection. When the
PeerConnectionClosed indicator is turned on, the SMC-R peer is
indicating that it is done updating the partner's RMBE.

D. Conditional State: state: If the local application has not yet issued
a close() or shutdown(both) yet, shutdown(both), we need to wait until the
application does so (ApplFinWaitState). so. Once it does, the local host will send a
CDC message to notify the peer of the PeerConnectionClosed
indicator and then transition to the Closed state.

4. Passive close processing (on the SMC-R peer that receives an
indication that the partner is closing the connection) connection).

A. Upon receipt of an inbound RDMA write notice a CDC message, the SMC-R layer will detect that
the PeerConnectionClosed indicator or PeerDoneWriting indicator
is on. If any outstanding recv() calls are pending pending, they are
completed with an indicator that the partner has closed the
connection (zero length (zero-length data presented to the application). If
there is any pending data to be written and
PeerConnectionClosed is on on, then an SMC-R connection reset must
be performed. The connection then enters the ApplCloseWait1 AppCloseWait1
state on the passive close side waiting for the local
application to initiate its own close processing processing.

B. If the local application issues a shutdown() for writing writing, then
the SMC-R layer will send a CDC message to notify the partner
of the PeerDoneWriting indicator and then transition the local
side of the SMC-R connection to the ApplCloseWait2 AppCloseWait2 state.

C. When the application issues a close() or shutdown() for both,
the local SMC-R peer will send a message informing the peer of
the PeerConnectionClosed indicator and transition to the Closed
state if the remote peer has also sent the local peer the
PeerConnectionClosed indicator. If the peer has not sent the
PeerConnectionClosed indicator, we transition into the
PeerFinalCloseWait
PeerFinCloseWait state.

D. The local SMC-R connection stays in this state until the peer
sends the PeerConnectionClosed indicator in our RMBE. a CDC message.
When the indicator is sent sent, we transition to the Closed state
and are then free to reuse this RMBE.

Note that each SMC-R peer needs to provide some logic that will
prevent being stranded in a termination state indefinitely. For
example, if an Active Close SMC-R peer is in a PeerCloseWait (1 or 2)
state awaiting waiting for the remote SMC-R peer to update its connection
termination status status, it needs to provide a timer that will prevent it
from waiting in that state indefinitely should the remote SMC-R peer
not respond to this termination request. This could occur in error
scenarios;
scenarios -- for example, if the remote SMC-R peer suffered a failure
prior to being able to respond to the termination request or the
remote application is not responding to this connection termination
request by closing its own socket. This latter scenario is similar
to the TCP FINWAIT2 state that state, which has been known to sometimes cause
issues when remote TCP/IP hosts lose track of established connections
and neglect to close them. Even though the TCP standards do not
mandate a time out timeout from the TCP FINWAIT2 state, most TCP/IP
implementations implement assign a timeout for this state. A similar timeout
will be required for SMC-R connections. When this timeout occurs,
the local SMC-R peer performs TCP reset processing for this
connection. However, no additional RDMA writes to the partner RMBE
can occur at this point (we have already indicated that we are done
updating the peer's RMBE). After the TCP connection is Reset reset, the
RMBE can be returned to the free pool for reallocation. See section 3.2.5
Section 4.4.2 for more details.

Also note that it is possible to have two SMC-R end points endpoints initiate an
Active close concurrently. In that scenario scenario, the flows above still
apply,
apply; however, both end points endpoints follow the active close path (path 3).

4.8.1.1.

4.8.2. Abnormal SMC-R connection termination flows Connection Termination Flows

Abnormal SMC-R connection termination can occur for a variety of
reasons, including: including the following:

o The TCP connection associated with an SMC-R connection is reset.
In the TCP protocol TCP, either end point endpoint can send a RST segment to abort an
existing TCP connection when error conditions are detected for the
connection or the application overtly requests that the connection
be reset.

o Normal SMC-R connection termination processing has unexpectedly
stalled for a given connection. When the stall is detected
(connection termination timeout condition) condition), an abnormal SMC-R
connection termination flow is initiated.

In these scenarios scenarios, it is very important that resources associated
with the affected SMC-R connections are properly cleaned up to ensure
that there are no orphaned resources and that resources can reliably
be reused for new SMC-R connections. Given that SMC-R relies heavily
on the RDMA Write write processing, special care needs to be taken to
ensure that an RMBE is no longer being used by a an SMC-R peer before
logically reassigning that RMBE to a new SMC-R connection.

When an SMC-R peer initiates a TCP connection reset reset, it also
initiates an SMC-R abnormal connection flow at the same time. The
SMC-R peers explicitly signal their intent to abnormally terminate an
SMC-R connection and await explicit acknowledgement acknowledgment that the peer has
received this notification and has also completed abnormal connection
termination on its end. Note that TCP connection reset processing
can occur in parallel to these flows.

+-----------------+
|-------------->| CLOSED |<-------------|
| | | |
| +-----------------+ |
| |
| |
| |
| +-----------------+ +-----------------------+ |
| | Any State state | |
|1B | (before setting | 2B|
| | PeerConnClosed PeerConnectionClosed | |
| | Indicator indicator in | |
| | Peer's peer's RMBE) | |
| +-----------------+ +-----------------------+ |
| 1A | | 2A |
| Active Abort | | Passive Abort |
| V V |
| +--------------+ +--------------+ |
|-------|PeerAbortWait | | Process Abort|------|
| | | |
+--------------+ +--------------+

Figure 23 23: SMC-R abnormal connection termination state diagram Abnormal Connection Termination State Diagram

Figure 24 23 above shows the SMC-R abnormal connection termination state
diagram:

1. Active abort designates the SMC-R peer that is initiating the TCP
RST processing. At the time that the TCP RST is sent sent, the active
abort side must also do the following:

A. Send the PeerConnAbort indicator to the partner via RDMA
messaging with inline data in a CDC
message, and then transition to the PeerAbortWait state.
During this state state, it will monitor this SMC-
R SMC-R connection
waiting for the peer to send its corresponding PeerConnAbort
indicator but will ignore any other activity in this connection (i.e.
(i.e., new incoming data). It will also surface generate an
appropriate error to any socket API calls issued against this
socket (e.g. (e.g., ECONNABORTED, ECONNRESET, etc.) ECONNRESET).

B. Once the peer sends the PeerConnAbort indicator to the local
host, the local host can transition this SMC-R connection to
the Closed state and reuse this RMBE. Note that the SMC-R peer
that goes into the Active active abort state must provide some
protection against staying in that state indefinitely should
the remote SMC-
R SMC-R peer not respond by sending its own
PeerConnAbort indicator to the local host. While this should
be a rare scenario scenario, it could occur if the remote SMC-R peer
(passive abort) suffered a failure right after the local SMC-R
peer (active abort) sent the PeerConnAbort indicator. To
protect against these types of failures, a timer can be set
after entering the PeerAbortWait
state state, and when if that timer pops
before the peer has sent its local PeerConnAbort indicator (to
the active abort side) then side), this RMBE can be returned to the free
pool for possible re-
allocation. See section reallocation. See section 3.2.5 Section 4.4.2 for more
details.

2. Passive abort designates the SMC-R peer that is the recipient of
an SMC-R abort from the peer designated by the PeerConnAbort
indicator being sent by the peer in a CDC message. Upon receiving
this request, the local peer must do the following:

A. Indicate Using the appropriate error codes, indicate to the socket
application that this connection has been aborted using the appropriate error codes, aborted, and then
purge all in-
flight in-flight data for this connection that is waiting to
be read or waiting to be sent.

B. Send a CDC message to notify the peer of the PeerConnAbort
indicator and and, once that is completed completed, transition this RMBE to
the Closed state.

If an SMC-R peer receives a TCP RST for a given SMC-R connection connection, it
also initiates SMC-R abnormal connection termination processing if it
has not already been notified (via the PeerConnAbort indicator) that
the partner is severing the connection. It is possible to have two
SMC-R endpoints concurrently be in an Active active abort role for a given
connection. In that scenario scenario, the flows above still apply but both
end points
endpoints take the active abort path (path 1).

4.8.1.2.

4.8.3. Other SMC-R connection termination conditions SMC-R Connection Termination Conditions

The following are additional conditions that have implications of for
SMC-R connection termination:

o A An SMC-R peer being gracefully shut down. If an SMC-R peer
supports a graceful shutdown operation operation, it should attempt to
terminate all SMC-R connections as part of shutdown processing.
This could be accomplished via LLC Delete Link DELETE LINK requests on all
active SMC Links. SMC-R links.

o Abnormal termination of an SMC-R peer. In this example, there may
be no opportunity for the host to perform any SMC-R cleanup
processing. In this scenario scenario, it is up to the remote peer to
detect a RoCE communications failure with the failing host. This
could trigger an SMC SMC-R link switch switchover, but that would also surface generate
RoCE
errors errors, causing the remote host to eventually terminate all
existing SMC-R connections to this peer.

o Loss of RoCE connectivity between two SMC-R peers. If two peers
are no longer reachable across any links in their SMC Link group SMC-R link
group, then both peers perform a TCP reset for the connections, surface
generate an error to the local applications applications, and free up all QP
resources associated with the link group.

5. Security considerations Considerations

5.1. VLAN considerations Considerations

The concepts and access control of virtual LANs (VLANs) must be
extended to also cover the RoCE network traffic flowing across the
ethernet.
Ethernet.

The RoCE VLAN configuration and accesses access permissions must mirror the IP
VLAN configuration and accesses access permissions over the CEE Converged Enhanced
Ethernet fabric. This means that hosts, routers routers, and switches that
have access to specific VLANs on the IP fabric must also have the
same VLAN access across the RoCE fabric. In other words, the SMC-R
connectivity will follow the same virtual network access permissions
as normal TCP/IP traffic.

5.2. Firewall considerations Considerations

As mentioned above, the RoCE fabric inherits the same VLAN
topology/access as the IP fabric. RoCE is a layer Layer 2 protocol that
requires both end points endpoints to reside in the same layer Layer 2 network (i.e. (i.e.,
VLAN). RoCE traffic can not cannot traverse multiple VLANs VLANs, as there is no
support for routing RoCE traffic beyond a single VLAN. As a result,
SMC-R communications will also be confined to peers that are members
of the same VLAN. IP based IP-based firewalls are typically inserted between
VLANs (or physical lans) LANs) and rely on normal IP routing to insert
themselves in the data path. Since RoCE (and by extension SMC-R) is
not routable beyond the local VLAN, there is no ability to insert a
firewall in the network path of two SMC-R peers.

5.3. Host-based Host-Based IP Filters

Because SMC-R maintains the TCP three-way handshake for connection
setup before switching to RoCE out of band, existing IP filters that
control connection setup flows remain effective in an SMC-R
environment. IP filters that operate on traffic flowing in an active
TCP connection are not supported, because the connection data does
not flow over IP.

5.4. Intrusion Detection Services

Similar to IP filters, intrusion detection services that operate on
TCP connection setups are compatible with SMC-R with no changes
required. However However, once the TCP connection has switched to RoCE out
of band, packets are not available for examination.

5.5. IP Security (IPSec) (IPsec)

IP Security security is not compatible with SMC-R SMC-R, because there are no IP
packets on which to operate on. operate. TCP connections that require IP
security must opt out of SMC-R.

5.6. TLS/SSL

TLS/SSL

Transport Layer Security/Secure Socket Layer (TLS/SSL) is preserved
in an SMC-R environment. The TLS/SSL layer resides above the SMC-R layer
layer, and outgoing connection data is encrypted before being passed
down to the SMC-R layer for RMDA RDMA write. Similarly, incoming
connection data goes through the SMC-R layer encrypted and is
decrypted by the TLS/SSL layer as it is today.

The TLS/SSL handshake messages flow over the TCP connection after the
connection has switched to SMC-R, and so they are exchanged using
RDMA writes by the SMC-R layer, transparently to the TLS/SSL layer.

6. IANA considerations Considerations

The scarcity of TCP option codes available for assignment is
understood
understood, and this architecture uses experimental TCP options
following the conventions of RFC 6994 "Shared [RFC6994] ("Shared Use of Experimental
TCP
Options". Options").

TCP ExID 0xE2D4C3D9 has been registered with IANA as a TCP Experiment
Identifier. See Section 3.1.

If this protocol achieves wide acceptance acceptance, a discrete option code may
be requested by subsequent versions of this protocol.

7. References

7.1. Normative References

[ROCE] RDMA over Converged Ethernet specification, URL,
http://members.infinibandta.org/kwspub/spec/Annex_RoCE_fina
l.pdf

[IBTA] Infiniband Architecture specification, URL,
http://www.infinibandta.org/specs

[RFC793] University of Southern California Information Services
Institute, Postel, J., "Transmission Control Protocol", STD 7,
RFC 793, DOI 10.17487/RFC0793, September 1981.

[RFC4727] Fenner B., "Experimental Values in IPv4, IPv6, ICMPv4,
ICMPv6, UDP, and TCP Headers", RFC 4727, November 2006.

7.2. Informative References

[RFC 6994] 1981,
<http://www.rfc-editor.org/info/rfc793>.

[RFC6994] Touch, J., "Shared use Use of Experimental TCP Options",
draft URL, https://tools.ietf.org/html/rfc6994

8. Acknowledgments

This document was prepared using 2-Word-v2.0.template.dot.

9. Conventions used in this document

In the rendezvous flow diagrams, dashed lines (----) are used to
indicate flows over the TCP/IP fabric and dotted lines (....) are
used to indicate flows
RFC 6994, DOI 10.17487/RFC6994, August 2013,
<http://www.rfc-editor.org/info/rfc6994>.

[RoCE] InfiniBand, "RDMA over the RoCE fabric.

In the data transfer ladder diagrams, dashed lines (----) are used to
indicate RDMA write operations and dotted lines (....) are used to
indicate CDC messages, which are RDMA messages with inline data that
contain control information for the connection. Converged Ethernet specification",
<https://cw.infinibandta.org/wg/Members/documentRevision/
download/7149>.

Appendix A. Formats

A.1. TCP option Option

The SMC-R TCP option is formatted in accordance with RFC 6994 "Shared [RFC6994]
("Shared Use of Experimental TCP Options". Options"). The ExID value is
IBM-1047 (EBCDIC) encoding for 'SMCR' "SMCR".

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Kind = 254 | Length = 6 | x'E2' | x'D4' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| x'C3' | x'D9' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 24 24: SMC-R TCP option format Option Format

A.2. CLC messages Messages

The following rules apply to all CLC messages:

General rules on formats:

o Reserved fields must be set to zero and not validated validated.

o Each message has an eyecatcher eye catcher at the start and another eyecatcher
eye catcher at the end. These must both be validated by the
receiver.

o SMC version indicator: The only SMC-R version defined in this
architecture is version 1. In the future, if peers have a
mismatch of versions, the lowest common version number is used.

A.2.1. Peer ID format Format

All CLC messages contain a peer ID that uniquely identifies an
instance of a TCP/IP stack. This peer ID is required to be
universally unique across TCP/IP stacks and instances (including
restarts) of TCP/IP stacks.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Instance ID | RoCE MAC (first two 2 bytes) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RoCE MAC (last four 4 bytes) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 25 25: Peer ID format Format

Instance ID

A two-byte 2-byte instance count that ensures that if the same RNIC MAC is
later used in the peer ID for a different TCP/IP stack, stack -- for
example
example, if an RNIC is redeployed to another stack, stack -- the values
are unique. It also ensures that if a TCP/IP stack is restarted,
the instance ID changes. Value The value is implementation defined,
with one suggestion being two 2 bytes of the system clock.

RoCE MAC

The RoCE MAC address for one of the peer's RNICs. Note that in a
virtualized environment this will be the virtual MAC of one of the
peer's RNICs.

A.2.2. SMC Proposal CLC message format Message Format

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| x'E2' | x'D4' | x'C3' | x'D9' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type = 1 | Length |Version| Rsrvd |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- Client's Peer ID -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- -+
| |
+- Client's preferred GID -+
| |
+- -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Client's preferred RoCE |
+- MAC address +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |Offset to mask/prefix area (0) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
. .
. Area for future growth .
. .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| IPv4 Subnet Mask |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| IPv4 Mask Lgth| Reserved |Num IPv6 prfx |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
: :
: (Variable length) array Array of IPv6 Prefixes prefixes (variable length) :
: :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| x'E2' | x'D4' | x'C3' | x'D9' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 26 26: SMC Proposal CLC message format Message Format
The fields present in the SMC Proposal CLC message are:

Eyecatchers

Eye catchers

Like all CLC messages, the SMC Proposal has beginning and ending
eyecatchers
eye catchers to aid with verification and parsing. The hex digits
spell 'SMCR' "SMCR" in IBM-1047 (EBCDIC) (EBCDIC).

Type

CLC message type Type 1 indicates SMC Proposal Proposal.

Length

The length of this CLC message. If this is an IPv4 flow, this
value is 52. Otherwise Otherwise, it is variable variable, depending upon how many
prefixes are listed.

Version

Version of the SMC-R protocol. Version 1 is the only currently
defined value value.

Client's Peer ID

As described in A.2.1. above Appendix A.2.1 above.

Client's preferred RoCE GID

This is the

The IPv6 address of the client's preferred RNIC on the RoCE fabric
fabric.

Client's preferred RoCE MAC address

The MAC address of the client's preferred RNIC on the RoCE fabric.
It is required required, as some operating systems do not have neighbor
discovery or ARP support for RoCE RNICs.

Offset to mask/prefix area

Provides the number of bytes that must be skipped after this
field, to access the IPv4 Subnet Mask field and the fields that
follow it. Allows for future growth of this signal. In this
version of the architecture, this value is always zero.

Area for future growth

In this version of the architecture, this field does not exist.
This indicates where additional information may be inserted into
the signal in the future. "The Offset The "Offset to mask/prefix area" field
must be used to skip over this area.

IPv4 Subnet mask Mask

If this message is flowing over an IPv4 TCP connection, the value
of the subnet mask associated with the interface over which the
client sent this message over. message. If this is an IPv6 flow flow, this field is
all
zeroes. zeros.

This field, along with all fields that follow it in this signal,
must be accessed by skipping the number of bytes listed in the
"Offset to mask/prefix area" field after the end of that field.

IPv4 Mask Lgth

If this message is flowing over an IPv4 TCP connection, the number
of significant bits in the IPv4 subnet mask. Subnet Mask field. If this is an
IPv6 flow, this field is zero.

Num IPv6 prfx

If this message is flowing over an IPv6 TCP connection, the number
of IPv6 prefixes that follow, with a maximum value of 8.
if If this
is an IPv4 flow flow, this field is zero and is immediately followed by
the ending eyecatcher. eye catcher.

Array of IPv6 Prefixes prefixes

For IPv6 TCP connections, a list of the IPv6 prefixes associated
with the network over which the client sent this message over, message, up to a
maximum of 8 eight prefixes.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+ +
| |
+ IPv6 Prefix prefix value +
| |
+ +
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Prefix Length |
+-+-+-+-+-+-+-+-+

Figure 27 27: Format for IPv6 Prefix array element Array Element

A.2.3. SMC Accept CLC message format Message Format

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| x'E2' | x'D4' | x'C3' | x'D9' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type = 2 | Length = 68 |Version|F|Rsvd | |Version|F|Rsrvd|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- Server's Peer ID -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- -+
| |
+- Server's RoCE GID -+
| |
+- -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Server's RoCE |
+- MAC address +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | Server QP (bytes 1-2) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+
|Srvr QP byte 3 | Server RMB Rkey RKey (bytes 1-3) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Srvr RMB byte 4|Server RMB indx| Srvr RMB alert tkn (bytes 1-2)|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Srvr RMB alert tkn (bytes 3-4)|Bsize | MTU | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- Server's RMB virtual address -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved | Server's initial packet sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| x'E2' | x'D4' | x'C3' | x'D9' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 28 28: SMC Accept CLC message format Message Format
The fields present on in the SMC Accept CLC message are:

Eyecatchers

Eye catchers

Like all CLC messages, the SMC Accept has beginning and ending
eyecatchers
eye catchers to aid with verification and parsing. The hex digits
spell 'SMCR' "SMCR" in IBM-1047 (EBCDIC) (EBCDIC).

Type

CLC message type Type 2 indicates SMC Accept Accept.

Length

The SMC Accept CLC message is 68 bytes long long.

Version

Version of the SMC-R protocol. Version 1 is the only currently
defined value.

F-bit

First Contact contact flag: A 1-bit flag that indicates that the server
believes this TCP connection is the first SMC-R contact for this
link group group.

Server's Peer ID

As described in A.2.1. above Appendix A.2.1 above.

Server's RoCE GID

This is the

The IPv6 address of the RNIC that the server chose for this SMC Link SMC-R
link.

Server's RoCE MAC address

The MAC address of the server's RNIC for the SMC SMC-R link. It is
required
required, as some operating systems do not have neighbor discovery
or ARP support for RoCE RNICs.

Server's QP number

The number for the reliably connected queue pair that the server
created for this SMC link SMC-R link.

Server's RMB Rkey RKey

The RDMA Rkey RKey for the RMB that the server created or chose for
this TCP connection connection.

Server's RMB element index

This indexes

Indexes which element within the server's RMB will represent this
TCP connection connection.

Server's RMB element alert token

A platform defined, platform-defined, architecturally opaque token that identifies
this TCP connection. Added by the client as immediate data on
RDMA writes from the client to the server to inform the server
that there is data for this connection to retrieve from the
RMB
element element.

Bsize:

Server's RMB element buffer size in four bits 4-bit compressed notation: x=4
x = 4 bits. Actual buffer size value is (2^(x+4)) (2^(x + 4)) * 1K.
Smallest possible value is 16K. Largest size supported by this
architecture is 512K.

MTU

An enumerated value indicating this peer's QP MTU size. The two
peers exchange this value their MTU values, and the minimum of the peer's whichever value is smaller
will be used for the QP. This field should only be validated on a in
the first contact exchange.

The enumerated MTU values are:

0: reserved

1: 256

2: 512

3: 1024

4: 2048

5: 4096

6-15: reserved
Server's RMB virtual address

The virtual address of the server's RMB as assigned by the
server's RNIC.

Server's initial packet sequence number

The starting packet sequence number that this peer will use when
sending to the other peer, so that the other peer can prepare its
QP for the sequence number to expect.

A.2.4. SMC Confirm CLC message format Message Format

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| x'E2' | x'D4' | x'C3' | x'D9' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type = 3 | Length = 68 |Version| Rsrvd |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- Client's Peer ID -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- -+
| |
+- Client's RoCE GID -+
| |
+- -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Client's RoCE |
+- MAC address +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | Client QP (bytes 1-2) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+---+
|Clnt QP byte 3 | Client RMB Rkey RKey (bytes 1-3) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Clnt RMB byte 4|Client RMB indx| Clnt RMB alert tkn (bytes 1-2)|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Clnt RMB alert tkn (bytes 3-4)|Bsize | MTU | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- Client's RMB Virtual Address -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved | Client's initial packet sequence number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| x'E2' | x'D4' | x'C3' | x'D9' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 29 29: SMC Confirm CLC message format Message Format

The SMC Confirm CLC message is nearly identical to the SMC Accept Accept,
except that it contains client information and lacks a first contact
flag.

The fields present on in the SMC Confirm CLC message are:

Eyecatchers

Eye catchers

Like all CLC messages, the SMC Confirm has beginning and ending
eyecatchers
eye catchers to aid with verification and parsing. The hex digits
spell 'SMCR' "SMCR" in IBM-1047 (EBCDIC) (EBCDIC).

Type

CLC message type Type 3 indicates SMC Confirm Confirm.

Length

The SMC Confirm CLC message is 68 bytes long long.

Version

Version of the SMC-R protocol. Version 1 is the only currently
defined value.

Client's Peer ID

As described in A.2.1. above

Clients's Appendix A.2.1 above.

Client's RoCE GID

This is the

The IPv6 address of the RNIC that the client chose for this SMC Link SMC-R
link.

Client's RoCE MAC address

The MAC address of the client's RNIC for the SMC SMC-R link. It is
required
required, as some operating systems do not have neighbor discovery
or ARP support for RoCE RNICs.

Client's QP number

The number for the reliably connected queue pair that the client
created for this SMC link SMC-R link.

Client's RMB Rkey RKey

The RDMA Rkey RKey for the RMB that the client created or chose for
this TCP connection connection.

Client's RMB element index

This indexes

Indexes which element within the client's RMB will represent this
TCP connection connection.

Client's RMB element alert token

A platform defined, platform-defined, architecturally opaque token that identifies
this TCP connection. Added by the server as immediate data on
RDMA writes from the server to the client to inform the client
that there is data for this connection to retrieve from the
RMB
element element.

Bsize:

Client's RMB element buffer size in four bits 4-bit compressed notation: x=4
x = 4 bits. Actual buffer size value is (2^(x+4)) (2^(x + 4)) * 1K.
Smallest possible value is 16K. Largest size supported by this
architecture is 512K.

MTU

An enumerated value indicating this peer's QP MTU size. The two
peers exchange this value their MTU values, and the minimum of the peer's whichever value is smaller
will be used for the QP. The values are enumerated in
Appendix A.2.3. This value should only be validated on in the first
contact exchange.

Client's RMB virtual address Virtual Address

The virtual address of the server's client's RMB as assigned by the
server's RNIC.

Client's initial packet sequence number

The starting packet sequence number that this peer will use when
sending to the other peer, so that the other peer can prepare its
QP for the sequence number to expect

. expect.

A.2.5. SMC Decline CLC message format Message Format

Figure 30 30: SMC Decline CLC message format Message Format

The fields present on in the SMC Decline CLC message are:

Eyecatchers

Eye catchers

Like all CLC messages, the SMC Decline has beginning and ending
eyecatchers
eye catchers to aid with verification and parsing. The hex digits
spell 'SMCR' "SMCR" in IBM-1047 (EBCDIC) (EBCDIC).

Type

CLC message type Type 4 indicates SMC Decline Decline.

Length

The SMC Decline CLC message is 28 bytes long long.

Version

Version of the SMC-R protocol. Version 1 is the only currently
defined value.

S-bit

Synch

Sync Bit. Indicates that the link group is out of synch sync and the
receiving peer must clean up its representation of the link group group.

Sender's Peer ID

As described in A.2.1. above Appendix A.2.1 above.

Peer Diagnosis Information

Four

4 bytes of diagnosis information provided by the peer. These
values are defined by the individual peers peers, and it is necessary to
consult the peer's system documentation to interpret the results.

A.3. LLC messages Messages

LLC messages are sent over an existing SMC-R link using RoCE message
passing SendMsg
and are always 44 bytes long so that they fit into the space
available in a single WQE without requiring the receiver to post
receive buffers. If all 44 bytes are not needed, they are padded out
with zeroes. zeros. LLC messages are in a request/response format. The
message type is the same for request and response, and a flag
indicates whether a message is flowing as a request or a response.

The two high order high-order bits of an LLC message opcode indicate how it is
to be handled by a peer that does not support the opcode.

If the high order high-order bits of the opcode are b'00' b'00', then the peer must
support the LLC message and indicate a protocol error if it does not.

If the high order high-order bits of the opcode are b'10' b'10', then the peer must
silently discard the LLC message if it does not support the opcode.
This requirement is inserted included to allow for toleration of advanced, but
optional function.

High order
optional, functionality.

High-order bits of b'11' indicate a Connection Data Control (CDC)
message as described in Appendix A.4.

A.3.1. CONFIRM LINK LLC message format Message Format

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| type Type = 1 | length Length = 44 | Reserved |R| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sender's RoCE |
+- MAC address +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
| |
+- -+
| Sender's RoCE GID |
+- -+
| |
+- +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |Sender's QP number, bytes 1-2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Sender QP byte3| Link number |Sender's link userid, userID, bytes 1-2|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Sender's link userid bytes, userID, bytes 3-4| Max links | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- Reserved -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 31 31: CONFIRM LINK LLC message format Message Format

The CONFIRM LINK LLC message is required to be exchanged between the
server and client over a newly created SMC-R link to complete the
setup of an SMC SMC-R link. Its purpose is to confirm that the RoCE path
is actually usable.

On first contact contact, this message flows after the server receives the
SMC Confirm CLC message from the client over the IP connection. For
additional links added to an SMC SMC-R link group, it flows after the
ADD LINK and ADD LINK CONTINUATION exchange. This flow provides
confirmation that the queue pair is in fact usable. Each peer echoes
its RoCE information back to the other.

The contents of the CONFIRM LINK LLC message are:

Type

Type 1 indicates CONFIRM LINK LINK.

Length
All

The CONFIRM LINK LLC messages are message is 44 bytes long long.

Reply flag. When set set, indicates that this is a CONFIRM LINK REPLY
reply.

Sender's RoCE MAC address

The MAC address of the sender's RNIC for the SMC SMC-R link. It is
required
required, as some operating systems do not have neighbor discovery
or ARP support for RoCE RNICs.

Sender's RoCE GID

This is the

The IPv6 address of the RNIC that the sender is using for this
SMC-R Link link.

Sender's QP number

The number for the reliably connected queue pair that the sender
created for this SMC-R link link.

Link number

An identifier assigned by the server that uniquely identifies the
link within the link group. This identifier is ONLY unique within
a link group. Provided by the server and echoed back by the client
client.

Link User user ID

An opaque, implementation defined implementation-defined identifier assigned by the
sender and provided to the receiver solely for purposes of
display, diagnosis, network management, etc. The link user ID
should be unique across the sender's entire software space,
including all link other link groups.

Max Links links

The maximum number of links the sender can support in a link
group. The maximum for this link group is the the smaller of the
values provided by the two peers.

A.3.2. ADD LINK LLC message format Message Format

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| type Type = 2 | length Length = 44 | Rsrvd |RsnCode|R|Z| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Sender's RoCE |
+- MAC address +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
| |
+- -+
| Sender's RoCE GID |
+- -+
| |
+- +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |Sender's QP number, bytes 1-2 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Sender QP byte3| Link number |Rsrvd | MTU |Initial PSN |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Initial PSN, continued PSN (continued) | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -+
| Reserved |
+- -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 32 32: ADD LINK LLC message format Message Format

The ADD LINK LLC message is sent over an existing link in the link
group when a peer wishes to add an SMC-R link to an existing SMC-R
link group. It is sent by the server to add a new SMC-R link to the
group, or by the client to request that the server add a new link, link --
for example example, when a new RNIC becomes active. When sent from the
client to the server, it represents a request that the server
initiate an ADD LINK exchange.

This message is sent immediately after the initial SMC SMC-R link in the
group completes, as described in 3.5.1. First contact. Section 3.5.1 ("First Contact"). It
can also be sent over an existing SMC-R link group at any time as new
RNICs are added and become available. Therefore Therefore, there can be as few
as 1 one new RMB RTokens RToken to communicate, be communicated, or several. Rtokens RTokens will
be communicated using ADD LINK CONTINUATION messages.

The contents of the ADD LINK LLC message are:

Type

Type 2 indicates ADD LINK LINK.

Length

All

The ADD LINK LLC messages are message is 44 bytes long long.

RsnCode

If the Z (rejection) flag is set, this field provides the reason
code. Values can be:

X'1' - no alternate path available: set when the server
provides the same MAC/GID as an existing SMC-R link in
the group, and the client does not have any additional
RNICs available (i.e., the server is attempting to set
up an asymmetric link but none is available) available).

X'2' - Invalid MTU value specified specified.

Reply flag. When set set, indicates that this is an ADD LINK REPLY reply.

Rejection flag. When set on reply reply, indicates that the server's
ADD LINK was rejected by the client. When this flag is set, the
reason code will also be set.

Sender's RoCE MAC address

The MAC address of the sender's RNIC for the new SMC-R link. It
is required required, as some operating systems do not have neighbor
discovery or ARP support for RoCE RNICs.

Sender's RoCE GID

The IPv6 address of the RNIC that the sender is using for the new
SMC-R Link link.

Sender's QP number

The number for the reliably connected queue pair that the sender
created for the new SMC-R link link.

Link number

An identifier for the new SMC-R link. This is assigned by the
server and uniquely identifies the link within the link group.
This identifier is ONLY unique within a link group. Provided by
the server and echoed back by the client client.

MTU

Initial PSN

The starting packet sequence number (PSN) that this peer will use
when sending to the other peer, so that the other peer can prepare
its QP for the sequence number to expect.

A.3.3. ADD LINK CONTINUATION LLC message format Message Format

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| type Type = 3 | length Length = 44 | Reserved |R| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Linknum | NumRTokens | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- -+
| |
+- Rkey/Rtoken Pair RKey/RToken pair -+
| |
+- -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- -+
| |
+- Rkey/Rtoken Pair RKey/RToken pair or zeroes zeros -+
| |
+- -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 33 33: ADD LINK CONTINUATION LLC message format Message Format

When a new SMC-R link is added to an SMC-R link group, it is
necessary to communicate the new link's RTokens for the RMBs that the
SMC-r
SMC-R link group can access. This message follows the ADD LINK and
provides the RTokens.

The server kicks off this exchange by sending the first ADD LINK
CONTINUATION LLC message, and the server controls the exchange as
described below.

o If the client and the server require the same number of ADD LINK
CONTINUATION messages to communicate their RTokens, the server
starts the exchange by sending the client the first ADD LINK CONTINUATION
request to the client with its RTokens, then the (the server's) RTokens. The client
then responds with an ADD LINK CONTINUATION response with its
RTokens, and so on until the exchange is completed.

o If the server requires more ADD LINK CONTINUATION messages than
the client, then after the client has communicated all of its
RTokens, the server continues to send ADD LINK CONTINUATION
request messages to the client. The client continues to respond,
using empty (number of RTokens to be communicated = 0) ADD LINK
CONTINUATION response messages.

o If the client requires more ADD LINK CONTINUATION messages than
the server, then after communicating all of its RTokens RTokens, the
server will continue to send empty ADD LINK CONTINUATION messages
to the client to solicit replies with the client's RTokens, until
all have been communicated.

The contents of this the ADD LINK CONTINUATION LLC message are:

Type

Type 3 indicates ADD LINK CONTINUATION CONTINUATION.

Length

All

The ADD LINK CONTINUATION LLC messages are message is 44 bytes long long.

Reply flag. When set set, indicates that this is an ADD LINK
CONTINUATION
REPLY reply.

LinkNum

The link number of the new link within the SMC SMC-R link group that
Rkeys for
which RKeys are being communicated for communicated.

NumRTokens

Number of RTokens remaining to be communicated (including the ones
in this message). If the value is less than or equal to 2, this
is the last message. If it is greater than 2, another
continuation message will be required, and its value will be the
value in this message minus 2, and so on until all Rkeys RKeys are
communicated. The maximum value for this field is 255.

Up to 2 Rkey/RToken

RKey/RToken pairs (two or less)

These consist of an Rkey RKey for an RMB that is known on the SMC-R
link that over which this message was sent over (the reference Rkey), RKey), paired
with the same RMB's RToken over the new SMC SMC-R link. A full RToken
is not required for the reference reference, because it is only being used
to distinguish which RMB it applies to, not address it.

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Reference Rkey RKey |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| New Rkey RKey |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
+- New Virtual Address -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 34 Rkey/Rtoken pair format 34: RKey/RToken Pair Format

The contents of the RKey/RToken pair are:

Reference Rkey RKey

The Rkey RKey of the RMB as it is already known on the SMC-R link over
which this message is being sent. Required so that the peer knows
with which RMB to associate the new Rtoken with. RToken.

New Rkey RKey

The Rkey RKey of this RMB as it is known over the new SMC-R link link.

New Virtual Address

The virtual address of this RMB as it is known over the new
SMC-R link.

A.3.4. DELETE LINK LLC message format Message Format

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| type Type = 4 | length Length = 44 | Reserved |R|A|O| Rsrvd |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Linknum | Reason reason code (bytes 1-3) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|RsnCode byte 4 | |
+-+-+-+-+-+-+-+-+ -+
| |
+- -+
| |
+- -+
| |
+- Reserved -+
| |
+- -+
| |
+- -+
| |
+- -+
| |
+- -+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 35 35: DELETE LINK LLC message format Message Format

When the client or server detects that a QP or SMC-R link goes down
or needs to come down, it sends this message over one of the other
links in the link group.

When the DELETE Link LINK is sent from the client client, it only serves as a
notification, and the client expects the server to send respond by sending
a DELETE LINK
Request in response. request. To avoid races, only the server will initiate
the actual DELETE LINK Request request and Response response sequence that results
from notification from the client.

The server can also initiate the DELETE Link LINK without notification
from the client if it detects an error or if orderly link termination
was initiated.

The client may also request termination of the entire link group group, and
the server may terminate the entire link group using this message.

The contents of this the DELETE LINK LLC message are:

Type

Type 4 indicates DELETE LINK LINK.

Length

All

The DELETE LINK LLC messages are message is 44 bytes long long.

Reply flag. When set set, indicates that this is an a DELETE LINK REPLY reply.

All

"All" flag. When set set, indicates that all links in the link group
are to be terminated. This terminates the link group.

Orderly flag. Indicates orderly termination. Orderly termination
is generally caused by an operator command rather than an error on
the link. When the client requests orderly termination, the
server may wait to complete other work before terminating.

LinkNum

The link number of the link to be terminated. If the A flag is
set, this field has no meaning and is set to 0.

RsnCode

The termination reason code. Currently defined reason codes are:

Request Reason Codes:

o reason codes:

X'00010000' = lost Lost path

X'00020000' = operator Operator initiated termination
o

X'00030000' = Program initiated termination (link inactivity)

X'00040000' = LLC protocol violation

X'00050000' = Asymmetric link no longer needed
Response Reason Codes:

o reason code:

X'00100000' = Unknown Link link ID (no link)

o Others TBD

A.3.5. CONFIRM RKEY LLC message format Message Format

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| type Type = 6 | length Length = 44 | Reserved |R|0|Z|C|Rsrvd |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| NumTkns | New RMB Rkey RKey for this link (bytes 1-3) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|ThisLink byte 4| |
+-+-+-+-+-+-+-+-+ -+
| New RMB virtual address for this link |
+- +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | |
+-+-+-+-+-+-+-+-+ -+
| |
+- Other link RMB specification or zeros -+
| |
+- +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -+
| |
+- -+
| Other link RMB specification or zeroes zeros |
+- +-+-+-+-+-+-+-+-+
| | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 36 36: CONFIRM RKEY LLC message format Message Format

The CONFIRM_RKEY CONFIRM RKEY flow can be sent at any time from either the client
or the server, to inform the peer that an RMB has been created or
deleted. The creator of a new RMB must inform its peer of the new
RMB's RToken for all SMC-R links in the SMC-R link group. The
deleter of an RMB must inform its peer of the deleted RMB's RToken
for all SMC-R links.

For RMB creation, the creator sends this message over the SMC SMC-R link
that the first TCP connection that uses the new RMB is using. This
message contains the new RMB RToken for the SMC SMC-R link that over which
the message is sent over, sent. It then it lists the sender's SMC SMC-R links in the
link group paired with the new RToken for the new RMB for that link.
This message can communicate the new RTokens for 3 three QPs: the QP
for the link over which this message is sent over, sent, and 2 two others. If
there are more than
3 three links in the SMC-R link group, CONFIRM_RKEY_CONTINUATION a
CONFIRM RKEY CONTINUATION will be required.

For RMB deletion, the creator sends the same format of message with a
delete flag set, to inform the peer that the RMB's RTokens on all
links in the group are deleted.

In both cases, the

The peer responds by simply echoing the message with the response
flag set. If the response is a negative response, the sender must
recalculate the RToken set and start a new CONFIRM_RKEY CONFIRM RKEY exchange from
the beginning. The timing of this retry is controlled by the C flag flag,
as described below.

The contents of this the CONFIRM RKEY LLC message are:

Type

Type 6 indicates CONFIRM RKEY RKEY.

Length

All

The CONFIRM RKEY LLC messages are message is 44 bytes long long.

Reply flag. When set set, indicates that this is a CONFIRM RKEY REPLY
reply.

Reserved bit bit.

Negative response flag flag.

Configuration Retry bit. If this is a negative response and this
flag is set, the originator should recalculate the Rkey RKey set and
retry this exchange as soon as the current configuration change is
completed. If this flag is not set on a negative response, the
originator must wait for the next natural stimulus (for example, a
new TCP connection started that requires a new RMB) before
retrying.

NumTkns

The number of other link/RToken pairs, including those provided in
this message, to be communicated. Note that this value does not
include the Rtoken RToken for the link on which this message was sent on
(i.e., the maximum value is 2). If this value is three 3 or fewer less, this
is the only message in the exchange. If this value is greater
than three, 3, a CONFIRM RKEY CONTINUATION message will be required.

Note: in In this version of the architecture, 8 eight is the maximum
number of links supported in a link group.

New RMB Rkey RKey for this link

The new RMB's Rkey RKey as assigned on the link over which this message
is being
sent over. sent.

New RMB virtual address for this link

The new RMB's virtual address as assigned on the link over which
this
messages message is being sent over. sent.