<?xml version="1.0" encoding="US-ASCII"?> version='1.0' encoding='utf-8'?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd" >
<?xml-stylesheet type="text/xsl" href="rfc2629.xslt" ?>

<?rfc strict="yes" ?>
<?rfc toc="yes"?>
<?rfc symrefs="yes"?>
<?rfc sortrefs="yes" ?>
<?rfc compact="yes" ?>
<?rfc subcompact="no" ?> "rfc2629-xhtml.ent">

<rfc xmlns:xi="http://www.w3.org/2001/XInclude" category="std"
     docName="draft-ietf-nfsv4-rpcrdma-cm-pvt-data-08"
  number="8797" ipr="trust200902" submissionType="IETF" consensus="true"
  updates="8166"
 xml:lang="en"> xml:lang="en" obsoletes="" tocInclude="true" symRefs="true"
  sortRefs="true" version="3">
  <!-- xml2rfc v2v3 conversion 2.41.0 -->
  <front>
    <title abbrev="RPC-Over-RDMA abbrev="RPC-over-RDMA CM Private Data">
RDMA
Remote Direct Memory Access - Connection Manager (RDMA-CM) Private Data For RPC-Over-RDMA for
    RPC-over-RDMA Version 1
</title> 1</title>
    <seriesInfo name="RFC" value="8797"/>
    <author initials="C.L." initials="C." surname="Lever" fullname="Charles Lever">
      <organization abbrev="Oracle">Oracle Corporation</organization>
      <address>
        <postal>
<street></street>
<city></city>
<region></region>
<code></code>
          <street/>
          <city/>
          <region/>
          <code/>
          <country>United States of America</country>
        </postal>
        <email>chuck.lever@oracle.com</email>
      </address>
    </author>
    <date />

<area>Transport</area>
<workgroup>Network File System Version 4</workgroup> month="June" year="2020"/>

    <keyword>NFS-over-RDMA</keyword>
    <abstract>
      <t>
This document specifies the format of
Remote Direct Memory Access - Connection Manager (RDMA-CM) Private Data
exchanged between RPC-over-RDMA version 1 peers
as part of establishing a connection.
The addition of the private data Private Data payload specified in this document
is an optional extension
that does not alter the RPC-over-RDMA version 1 protocol.
This document updates RFC 8166.
</t>
    </abstract>
  </front>
  <middle>
    <section
 title="Introduction"
 anchor="section:A4741DA2-FF86-4A9D-907B-8A73D53C7148"> anchor="section_A4741DA2-FF86-4A9D-907B-8A73D53C7148" numbered="true" toc="default">
      <name>Introduction</name>
      <t>
The RPC-over-RDMA version 1 transport protocol
<xref target="RFC8166"/> target="RFC8166" format="default"/>
enables payload data transfer using
Remote Direct Memory Access (RDMA)
for upper-layer protocols based on Remote Procedure Calls (RPC) (RPCs)
<xref target="RFC5531"/>. target="RFC5531" format="default"/>.
The terms "Remote Direct Memory Access" (RDMA) and
"Direct Data Placement" (DDP) are introduced in
<xref target="RFC5040"/>. target="RFC5040" format="default"/>.
</t>
      <t>
The two most immediate shortcomings
of RPC-over-RDMA version 1 are:
<list style="symbols"> are as follows:
</t>
      <ol spacing="normal">
        <li>
          <t>
Setting up an RDMA data transfer (via RDMA Read or Write) can be costly.
The small default size of messages transmitted using RDMA Send
forces the use of RDMA Read or Write operations
even for relatively small messages and data payloads.
<vspace/>
</t>
          <t>
The original specification of RPC-over-RDMA version 1 provided
an out-of-band protocol for passing inline threshold values
between connected peers
<xref target="RFC5666"/>. target="RFC5666" format="default"/>.
However,
<xref target="RFC8166"/> target="RFC8166" format="default"/>
eliminated support for this protocol protocol, making it unavailable for this purpose.
</t>
<t>
        </li>
        <li>
Unlike most other contemporary RDMA-enabled storage protocols,
there is no facility in RPC-over-RDMA version 1
that enables the use of remote invalidation
<xref target="RFC5042"/>.
</t>
</list>
</t> target="RFC5042" format="default"/>.
</li>
      </ol>
      <t>
Each RPC-over-RDMA version 1 transport header Transport Header follows the
External Data Representation (XDR)
<xref target="RFC4506"/> definition
<xref target="RFC4506" format="default"/>
specified in <xref target="RFC8166"/>. target="RFC8166" format="default"/>.
However, RPC-over-RDMA version 1
has no means of extending this definition
in such a way that interoperability with existing implementations is preserved.
As a result, an out-of-band mechanism is needed
to help relieve these constraints
for existing RPC-over-RDMA version 1 implementations.
</t>
      <t>
This document specifies a simple, non-XDR-based message format
designed to be passed between RPC-over-RDMA version 1 peers
at the time each RDMA transport connection is first established.
The mechanism assumes that the underlying RDMA transport has a
private data
Private Data field that is passed between peers at connection time,
such as is present in the iWARP Marker PDU Aligned Framing (MPA) protocol
(described in Section 7.1 of <xref target="RFC5044"/>) target="RFC5044" sectionFormat="of" section="7.1"/>
and extended in <xref target="RFC6581"/>) or the InfiniBand Connection Manager
<xref target="IBA"/>. target="IBA" format="default"/>.
</t>
      <t>
To enable current RPC-over-RDMA version 1 implementations
to interoperate with implementations that support
the private message format described in this document,
implementation of the private data message Private Data exchange is OPTIONAL. <bcp14>OPTIONAL</bcp14>.
When the private data message Private Data has been successfully exchanged,
peers may choose to perform extended RDMA semantics.
However, the private message format this exchange
does not alter the XDR definition specified in
<xref target="RFC8166"/>. target="RFC8166" format="default"/>.
</t>
      <t>
The message format is intended to be further extensible
within the normal scope of such IETF work
(see
<xref target="section:96A18D5A-2AE4-490D-B140-B964302BF1DF"/> target="section_96A18D5A-2AE4-490D-B140-B964302BF1DF" format="default"/>
for further details).
<xref target="section:8701C122-F206-4F87-8FA6-62F1CD648A14"/> target="section_8701C122-F206-4F87-8FA6-62F1CD648A14" format="default"/>
of this document defines an IANA registry for this purpose.
In addition, interoperation between
implementations of RPC-over-RDMA version 1 that present this message format to peers
and those that do not recognize this message format is guaranteed.
</t>
    </section>
    <section
 title="Requirements Language"
 anchor="section:6E4898D3-A2A9-47FA-9CBD-069328E610E4">
<t>
The anchor="section_6E4898D3-A2A9-47FA-9CBD-069328E610E4" numbered="true" toc="default">
      <name>Requirements Language</name>
       <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED",
"MAY", "<bcp14>MUST</bcp14>", "<bcp14>MUST NOT</bcp14>",
       "<bcp14>REQUIRED</bcp14>", "<bcp14>SHALL</bcp14>",
       "<bcp14>SHALL NOT</bcp14>", "<bcp14>SHOULD</bcp14>",
       "<bcp14>SHOULD NOT</bcp14>",
       "<bcp14>RECOMMENDED</bcp14>", "<bcp14>NOT RECOMMENDED</bcp14>",
       "<bcp14>MAY</bcp14>", and "OPTIONAL" "<bcp14>OPTIONAL</bcp14>" in this document
       are to be interpreted as described in BCP 14 BCP&nbsp;14
       <xref target="RFC2119"/> <xref target="RFC8174"/> when, and only
       when, they appear in all capitals, as shown here.
</t> here.</t>
    </section>
    <section
 title="Advertised anchor="section_D6ED5D4A-E123-4DDD-B65F-733F2B377679" numbered="true" toc="default">
      <name>Advertised Transport Properties"
 anchor="section:D6ED5D4A-E123-4DDD-B65F-733F2B377679"> Properties</name>
      <section
 title="Inline anchor="section_0F60CBDD-8A7E-4B41-B193-18C04201DB3E" numbered="true" toc="default">
        <name>Inline Threshold Size"
 anchor="section:0F60CBDD-8A7E-4B41-B193-18C04201DB3E"> Size</name>
        <t>
Section 3.3.2 of
<xref target="RFC8166"/> target="RFC8166" sectionFormat="of" section="3.3.2"/>
defines the term "inline threshold." threshold".
An inline threshold is the maximum number of bytes that
can be transmitted using one RDMA Send and one RDMA Receive.
There are a pair of inline thresholds for a connection:
a client-to-server threshold and a server-to-client threshold.
</t>
        <t>
If an incoming RDMA message exceeds the size
of a receiver's inline threshold,
the receive Receive operation fails
and
the RDMA provider typically terminates the connection.
To convey an RPC message larger than the receiver's inline threshold
without risking receive failure,
a sender must use explicit RDMA data transfer operations,
which are more expensive than an RDMA Send.
See Sections 3.3 Sections&nbsp;<xref target="RFC8166" section="3.3"
 sectionFormat="bare"/> and 3.5 <xref target="RFC8166" section="3.5"
 sectionFormat="bare"/> of <xref target="RFC8166"/>
for a complete discussion.
</t>
        <t>
The default value of inline thresholds for RPC-over-RDMA version 1
connections is 1024 bytes (as defined in Section 3.3.3 of <xref target="RFC8166"/>). target="RFC8166"
sectionFormat="of" section="3.3.3"/>).
This value is adequate for nearly all NFS version 3 procedures.
</t>
        <t>
NFS version 4 COMPOUND operations
<xref target="RFC7530"/> target="RFC7530" format="default"/>
are larger on average
than NFS version 3 procedures
<xref target="RFC1813"/>, target="RFC1813" format="default"/>,
forcing clients to use explicit RDMA operations
for frequently-issued frequently issued requests such as LOOKUP and GETATTR.
The use of RPCSEC_GSS security also increases the average size
of RPC messages,
due to the larger size of RPCSEC_GSS credential material
included in RPC headers
<xref target="RFC7861"/>. target="RFC7861" format="default"/>.
</t>
        <t>
If a sender and receiver could somehow agree on larger inline thresholds,
frequently-used
frequently used RPC transactions avoid the cost of explicit RDMA operations.
</t>
      </section>
      <section
 title="Remote Invalidation"
 anchor="section:F82BDC05-F8A6-478C-A02C-BC308EE75A18"> anchor="section_F82BDC05-F8A6-478C-A02C-BC308EE75A18" numbered="true" toc="default">
        <name>Remote Invalidation</name>
        <t>
After an RDMA data transfer operation completes,
an RDMA consumer can request
that its peer's RDMA network interface card Network Interface Card (RNIC)
invalidate the Steering Tag (STag)
associated with the data transfer
<xref target="RFC5042"/>. target="RFC5042" format="default"/>.
</t>
        <t>
An RDMA consumer requests remote invalidation by posting
an RDMA Send With with Invalidate Work Request operation
in place of an RDMA Send Work Request. operation.
Each RDMA Send With with Invalidate carries one STag to invalidate.
The receiver of an RDMA Send With with Invalidate performs the
requested invalidation and then reports that invalidation
as part of the completion of a waiting Receive Work Request. operation.
</t>
        <t>
If both peers support remote invalidation,
an RPC-over-RDMA responder might use remote invalidation
when replying to an RPC request that provided chunks.
Because one of the chunks has already been invalidated,
finalizing the results of the RPC is made simpler and faster.
</t>
        <t>
However, there are some important caveats which that contraindicate
the blanket use of remote invalidation:
<list style="symbols">
<t>
</t>
        <ul spacing="normal">
          <li>
Remote invalidation is not supported by all RNICs.
</t>
<t>
</li>
          <li>
Not all RPC-over-RDMA responder implementations can generate
RDMA Send With with Invalidate Work Requests.
</t>
<t> operations.
</li>
          <li>
Not all RPC-over-RDMA requester implementations can recognize
when remote invalidation has occurred.
</t>
<t>
</li>
          <li>
On one connection in different RPC-over-RDMA transactions,
or in a single RPC-over-RDMA transaction,
an RPC-over-RDMA requester can expose a mixture of STags
that may be invalidated remotely
and some that must not be.
No indication is provided at the RDMA layer as to which is which.
</t>
</list>
</li>
        </ul>
        <t>
A responder therefore must not employ remote invalidation unless it is
aware of support for it in its own RDMA stack, and on the requester.
And, without altering the XDR structure of RPC-over-RDMA version 1 messages,
it is not possible to support remote invalidation with requesters
that mix STags include an STag that may and must not be invalidated
remotely in a single an RPC or with STags that may be invalidated. Likewise, it
is not possible to support remote invalidation with requesters that
mix RPCs with STags that may be invalidated with RPCs with STags that
must not be invalidated on the same connection.
</t>
        <t>
There are some NFS/RDMA client implementations whose STags
are always safe to invalidate remotely.
For such clients, indicating to the responder that remote
invalidation is always safe can enable such invalidation
without the need for additional protocol elements to be defined.
</t>
      </section>
    </section>
    <section
 title="Private anchor="section_99541DF3-E579-42B6-9C7A-D020926798F6" numbered="true" toc="default">
      <name>Private Data Message Format"
 anchor="section:99541DF3-E579-42B6-9C7A-D020926798F6"> Format</name>
      <t>
With an InfiniBand lower layer, for example,
RDMA connection setup uses a Connection Manager (CM)
when establishing a Reliable Connection
<xref target="IBA"/>. target="IBA" format="default"/>.
When an RPC-over-RDMA version 1 transport connection is established,
the client (which actively establishes connections)
and the server (which passively accepts connections)
populate the CM Private Data field exchanged
as part of CM connection establishment.
</t>
      <t>
The transport properties exchanged via this mechanism
are fixed for the life of the connection.
Each new connection presents an opportunity
for a fresh exchange.
An implementation of the extension described in this document
MUST
<bcp14>MUST</bcp14> be prepared for the settings to change upon a reconnection.
</t>
      <t>
For RPC-over-RDMA version 1, the CM Private Data field
is formatted as described in the following subsection. below. RPC clients and servers use the same format.
If the capacity of the Private Data field is too small
to contain this message format
or
the underlying RDMA transport is not managed by a Connection Manager, CM,
the CM Private Data field cannot be used on behalf of RPC-over-RDMA version 1.
</t>
      <t>
The first 8 eight octets of the CM Private Data field
is
are to be formatted as follows:
</t>
<figure align="left">
      <artwork xml:space="preserve" align="left"> align="left" name="" type="" alt=""><![CDATA[
  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                       Format Identifier                       |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |    Version    |  Reserved   |R|   Send Size   | Receive Size  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
</artwork>
</figure>
<t>
<list style="hanging">
<t hangText="Format Identifier:">
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+]]></artwork>
      <dl newline="false" spacing="normal">
        <dt>Format Identifier:</dt>
        <dd>
This field contains a fixed 32-bit value that identifies
the content of the Private Data field as an RPC-over-RDMA
version 1 CM Private Data message.
In RPC-over-RDMA version 1 version&nbsp;1 Private Data,
the value of this field is always 0xf6ab0e18, in network byte order.
The use of this field is further expanded upon in
<xref target="section:0BD80E15-13FF-4210-A1D8-E0AE531821A4"/>.
</t>
<t hangText="Version:"> target="section_0BD80E15-13FF-4210-A1D8-E0AE531821A4" format="default"/>.
</dd>
        <dt>Version:</dt>
        <dd>
This 8-bit field contains a message format version number.
The value "1" in this field indicates that exactly eight octets are present,
that they appear in the order described in this section,
and that each has the meaning defined in this section.
Further considerations about the use of this field are discussed in
<xref target="section:96A18D5A-2AE4-490D-B140-B964302BF1DF"/>.
</t>
<t hangText="Reserved:"> target="section_96A18D5A-2AE4-490D-B140-B964302BF1DF" format="default"/>.
</dd>
        <dt>Reserved:</dt>
        <dd>
This 7-bit field is unused.
Senders MUST <bcp14>MUST</bcp14> set these bits to zero zero,
and
receivers MUST <bcp14>MUST</bcp14> ignore their value.
</t>
<t hangText="R:">
</dd>
        <dt>R:</dt>
        <dd>
This 1-bit field indicates that
the sender supports remote invalidation.
The field is set and interpreted as described in
<xref target="section:1D624044-49CC-4464-9BEF-59EA2473ACBF"/>.
</t>
<t hangText="Send Size:"> target="section_1D624044-49CC-4464-9BEF-59EA2473ACBF" format="default"/>.
</dd>
        <dt>Send Size:</dt>
        <dd>
This 8-bit field contains an encoded value
corresponding to the maximum number of bytes
this peer is prepared to transmit in a single RDMA Send
on this connection.
The value is encoded as described in
<xref target="section:689BD9D4-9C3C-41C7-81F3-8BBE283D252B"/>.
</t>
<t hangText="Receive Size:"> target="section_689BD9D4-9C3C-41C7-81F3-8BBE283D252B" format="default"/>.
</dd>
        <dt>Receive Size:</dt>
        <dd>
This 8-bit field contains an encoded value
corresponding to the maximum number of bytes
this peer is prepared to receive with a single RDMA Receive
on this connection.
The value is encoded as described in
<xref target="section:689BD9D4-9C3C-41C7-81F3-8BBE283D252B"/>.
</t>
</list>
</t> target="section_689BD9D4-9C3C-41C7-81F3-8BBE283D252B" format="default"/>.
</dd>
      </dl>
      <section
 title="Using anchor="section_1D624044-49CC-4464-9BEF-59EA2473ACBF" numbered="true" toc="default">
        <name>Using the R Field"
 anchor="section:1D624044-49CC-4464-9BEF-59EA2473ACBF"> Field</name>
        <t>
The R field indicates limited support for remote invalidate invalidation
as described in
<xref target="section:F82BDC05-F8A6-478C-A02C-BC308EE75A18"/>. target="section_F82BDC05-F8A6-478C-A02C-BC308EE75A18"
      format="default"/>.
When both connection peers have set this bit flag in their CM Private Data,
the responder MAY <bcp14>MAY</bcp14> use RDMA Send With with Invalidate operations
when transmitting RPC Replies.
Each RDMA Send With with Invalidate MUST <bcp14>MUST</bcp14> invalidate an STag
associated only with the XID Transaction ID (XID) in the rdma_xid field
of the RPC-over-RDMA Transport Header it carries.
</t>
        <t>
When either peer on a connection clears this flag,
the responder MUST <bcp14>MUST</bcp14> use only RDMA Send when transmitting RPC Replies.
</t>
      </section>
      <section
 title="Send anchor="section_689BD9D4-9C3C-41C7-81F3-8BBE283D252B" numbered="true" toc="default">
        <name>Send and Receive Size Values"
 anchor="section:689BD9D4-9C3C-41C7-81F3-8BBE283D252B"> Values</name>
        <t>
Inline threshold sizes from 1024 to 262144 octets
can be represented in the Send Size and Receive Size fields.
The inline threshold values provide a pair of
1024-octet-aligned maximum message lengths that
guarantee that Send and Receive operations
do not fail due to length errors.
</t>
        <t>
The minimum inline threshold for RPC-over-RDMA version 1
is 1024 octets (see Section 3.3.3 of <xref target="RFC8166"/>). target="RFC8166" sectionFormat="of" section="3.3.3"/>).
The values in the Send Size and Receive Size fields represent
the unsigned number of additional kilo-octets of length
beyond the first 1024 octets.
Thus, a sender computes the encoded value by
dividing its actual buffer size, in octets, by 1024
and
subtracting one from the result.
A receiver decodes an incoming Size value by performing
the inverse set of operations:
it adds one to the encoded value
and then
multiplies that result by 1024.
</t>
        <t>
The client uses the smaller of its own send size and
the server's reported receive size
as the client-to-server inline threshold.
The server uses the smaller of its own send size and
the clients's client's reported receive size
as the server-to-client inline threshold.
</t>
      </section>
    </section>
    <section
 title="Interoperability Considerations"
 anchor="section:58229D50-C3E7-48D1-A82F-7720AC8A9F8B"> anchor="section_58229D50-C3E7-48D1-A82F-7720AC8A9F8B" numbered="true" toc="default">
      <name>Interoperability Considerations</name>
      <t>
The extension described in this document is designed to allow
RPC-over-RDMA version implementations that use CM Private Data
to interoperate fully with
RPC-over-RDMA version 1 implementations that do not exchange this information.
Implementations that use this extension must also interoperate
fully with RDMA implementations that use CM Private Data for other purposes.
Realizing these goals requires that implementations of this extension
follow the practices described in the rest of this section.
</t>
      <section
 title="Interoperability anchor="section_9A3B285D-D557-4028-ACB0-60EA1C9C93BE" numbered="true" toc="default">
        <name>Interoperability with RPC-over-RDMA Version 1 Implementations"
 anchor="section:9A3B285D-D557-4028-ACB0-60EA1C9C93BE"> Implementations</name>
        <t>
When a peer does not receive a CM Private Data message
which
that conforms to
<xref target="section:99541DF3-E579-42B6-9C7A-D020926798F6"/>, target="section_99541DF3-E579-42B6-9C7A-D020926798F6" format="default"/>,
it needs to act as if the remote peer supports only the
default RPC-over-RDMA version 1 settings,
as defined in
<xref target="RFC8166"/>. target="RFC8166" format="default"/>.
In other words, the peer MUST <bcp14>MUST</bcp14> behave as if a Private Data
message was received in which bit (1)&nbsp;bit 15 of the Flags field is zero, zero
and both (2)&nbsp;both Size fields contain the value zero.
</t>
      </section>
      <section
 title="Interoperability Amongst anchor="section_0BD80E15-13FF-4210-A1D8-E0AE531821A4" numbered="true" toc="default">
        <name>Interoperability amongst RDMA Transports"
 anchor="section:0BD80E15-13FF-4210-A1D8-E0AE531821A4"> Transports</name>
        <t>
The Format Identifier field defined in
<xref target="section:99541DF3-E579-42B6-9C7A-D020926798F6"/> target="section_99541DF3-E579-42B6-9C7A-D020926798F6" format="default"/>
is provided to enable implementations to distinguish RPC-over-RDMA version 1 the Private Data defined
in this document from private data Private Data inserted at other layers, such as the private data inserted
additional Private Data defined by the iWARP MPAv2 enhancement protocol described in
<xref target="RFC6581"/>. target="RFC6581"/>, and others.
</t>
        <t>
As part of connection establishment,
the received private data buffer containing the received Private Data is searched for the Format Identifier word.
The offset of the Format Identifier is not restricted to any alignment.
If the RPC-over-RDMA version 1 CM Private Data Format Identifier
is not present,
an RPC-over-RDMA version 1 receiver MUST <bcp14>MUST</bcp14>
behave as if no RPC-over-RDMA version 1 CM Private Data
has been provided.
</t>
        <t>
Once the RPC-over-RDMA version 1 CM Private Data Format Identifier
is found,
the receiver parses the subsequent octets as
RPC-over-RDMA version 1 CM Private Data.
As additional assurance that the private data content is valid
RPC-over-RDMA version 1 CM Private Data,
the receiver should check that
the format version number field contains a valid and recognized version number
and
the size of the private data content does not overrun the length of the buffer.
</t>
      </section>
    </section>
    <section
 title="Updating anchor="section_96A18D5A-2AE4-490D-B140-B964302BF1DF" numbered="true" toc="default">
      <name>Updating the Message Format"
 anchor="section:96A18D5A-2AE4-490D-B140-B964302BF1DF"> Format</name>
      <t>
Although the message format described in this document
provides the ability for the client and server
to exchange particular information about
the local RPC-over-RDMA implementation,
it is possible that there will be a future need
to exchange additional properties.
This would make it necessary to extend or otherwise modify
the format described in this document.
</t>
      <t>
Any modification faces the problem of interoperating properly
with implementations of RPC-over-RDMA version 1
that are unaware of the existence of the new format.
These include implementations that that do not recognize
the exchange of CM Private Data
as well as
those that recognize only the format described in this document.
</t>
      <t>
Given the message format described in this document,
these interoperability constraints could be met by the following
sorts of new message formats:
<list style="symbols">
<t>
</t>
      <ul spacing="normal">
        <li>
A format which that uses a different value for the first four bytes of the format,
as provided for in the registry described in
<xref target="section:8701C122-F206-4F87-8FA6-62F1CD648A14"/>.
</t>
<t> target="section_8701C122-F206-4F87-8FA6-62F1CD648A14" format="default"/>.
</li>
        <li>
A format which that uses the same value for the Format Identifier field
and a value other than one (1) in the Version field.
</t>
</list>
</li>
      </ul>
      <t>
Although it is possible to reorganize
the last three of the eight bytes eight&nbsp;bytes in the existing format,
extended formats are unlikely to do so.
New formats would take the form of extensions
of the format described in this document with added fields
starting at byte eight of the format
or changes to the definition of bits in the Reserved field.
</t>
    </section>
    <section
 title="Security Considerations"
 anchor="section:19F962AD-CDE5-4216-BE3A-44BD238458EC"> anchor="section_19F962AD-CDE5-4216-BE3A-44BD238458EC" numbered="true" toc="default">
      <name>Security Considerations</name>
      <t>
The reader is directed to the Security Considerations section of
<xref target="RFC8166"/> target="RFC8166" format="default"/>
for background and further discussion.
</t>
      <t>
The RPC-over-RDMA version 1 protocol framework depends
on the semantics of the Reliable Connected (RC) queue pair (QP)
type, as defined in Section 9.7.7
Section&nbsp;9.7.7 of <xref target="IBA"/>.
The integrity of CM Private Data
and
the authenticity of its source
are ensured by the exclusive use of RC queue pairs. QPs.
Any attempt to interfere with or hijack data in transit
on an RC connection
results in the RDMA provider terminating the connection.
</t>
      <t>
The Security Considerations section of
<xref target="RFC5042"/> target="RFC5042" format="default"/>
refers the reader to further relevant discussion
of generic RDMA transport security.
That document recommends IPsec as
the default transport layer transport-layer security solution.
When deployed with iWARP, the Remote Direct Memory Access Protocol (RDMAP) <xref
target="RFC5040"/>, DDP <xref target="RFC5041"/>, and MPA <xref
target="RFC5044"/>, IPsec establishes a protected channel before any iWARP
operations are exchanged,
thus exchanged; thus, it protects the exchange of Private Data
that occurs as each QP is established. Data.
However, IPsec is not available for InfiniBand or RoCE RDMA over Converged Ethernet (RoCE) deployments.
Those fabrics rely on
physical security
and
cyclic redundancy checks
to protect network traffic.
</t>
      <t>
Exchanging the information
contained in the Private Message message format defined in this document
does not expose upper-layer payloads to an attacker.
Furthermore, the behavior changes that occur
as a result of processing exchanging the CM Private Data
format
described in the current document
do not introduce any new risk of exposure
of upper-layer payload data.
</t>
      <t>
Improperly setting one of the fields in a version 1
Private Message Data can result in an increased risk of disconnection
(i.e., self-imposed Denial of Service).
A similar risk can arise
if non-RPC-over-RDMA CM Private Data
inadvertently contains the Format Identifier that
identifies this protocol's data structure.
Additional checking of incoming Private Data,
as described in
<xref target="section:0BD80E15-13FF-4210-A1D8-E0AE531821A4"/>, target="section_0BD80E15-13FF-4210-A1D8-E0AE531821A4" format="default"/>,
can help reduce this risk.
</t>
      <t>
In addition to describing the structure of a new format version,
any document that extends the Private Data format described
in the current document must discuss security considerations
of new data items exchanged between connection peers.
Such documents should also explore the risks
of erroneously identifying non-RPC-over-RDMA CM Private Data
as the new format.
</t>
    </section>
    <section
 title="IANA Considerations"
 anchor="section:8701C122-F206-4F87-8FA6-62F1CD648A14"> anchor="section_8701C122-F206-4F87-8FA6-62F1CD648A14" numbered="true" toc="default">
      <name>IANA Considerations</name>
      <t>
In accordance with
<xref target="RFC8126"/>,
the author requests that
IANA create a new registry in has created the "RDMA-CM Private Data Identifiers" subregistry within the
"Remote Direct Data Placement"
Protocol Category Group.
The new registry is to be called the
"RDMA-CM Private Data Identifier Registry". protocol category group.
This is a registry subregistry of 32-bit numbers that identify
the upper-layer protocol associated with data that appears in
the application-specific RDMA-CM Private Data area.
The fields in this registry include: subregistry include the following:
Format Identifier,
Format
Length (in (format length, in octets),
Description,
and
Reference.
</t>
      <t>
The initial contents of this registry are a single entry:
</t>
<texttable
      <table align="left"
 style="full"
 title="RDMA-CM anchor="table_336556D8-B52A-476E-8E9E-FDF4A4C7ED66">
        <name>New &quot;RDMA-CM Private Data Identifier Registry"
 anchor="table:336556D8-B52A-476E-8E9E-FDF4A4C7ED66">
<ttcol Identifiers&quot; Registry</name>
        <thead>
          <tr>
            <th align="left">Format Identifier</ttcol>
<ttcol align="left">Length</ttcol>
<ttcol align="left">Description</ttcol>
<ttcol align="left">Reference</ttcol>
<c>0xf6ab0e18</c>
<c>8</c>
<c>RPC-over-RDMA Identifier</th>
            <th align="left">Length</th>
            <th align="left">Description</th>
            <th align="left">Reference</th>
          </tr>
        </thead>
        <tbody>
          <tr>
            <td align="left">0xf6ab0e18</td>
            <td align="left">8</td>
            <td align="left">RPC-over-RDMA version 1 CM Private Data</c>
<c>[RFC-TBD]</c>
</texttable>

<!-- RFC Editor:
Please replace RFC-TBD with the RFC number assigned to this document
--> Data</td>
            <td align="left">RFC 8797</td>
          </tr>
        </tbody>
      </table>

<t>
IANA is to assign subsequent new entries in this registry using
the Specification Required policy as defined in Section 4.6 of <xref target="RFC8126"/>. target="RFC8126" sectionFormat="of" section="4.6"/>.
</t>
      <section
 title="Guidance anchor="section_DB1C2628-A2D2-4EE7-8CD6-592F832C8C86" numbered="true" toc="default">
        <name>Guidance for Designated Experts"
 anchor="section:DB1C2628-A2D2-4EE7-8CD6-592F832C8C86"> Experts</name>
        <t>
The Designated Expert (DE), appointed by the IESG,
should ascertain the existence of suitable documentation that
defines the semantics and format of the private data, Private Data,
and verify that the document is permanently and publicly available.
Documentation produced outside the IETF must not conflict
with work that is active or already published within the IETF.
The new Reference field should contain
a reference to that documentation.
</t>
        <t>
The Description field should contain the name of the
upper-layer protocol
that generates and uses the private data. Private Data.
</t>
        <t>
The DE should assign a new Format Identifier so that
it does not conflict with existing entries in this registry, registry
and so that
it is not likely to be mistaken
as part of the payload of other registered formats.
</t>
        <t>
The DE shall post the request to the nfsv4 WG NFSV4 Working Group mailing list
(or a successor to that list, if such a list exists), exists)
for comment and review.
The DE shall approve or deny the request and publish notice
of the decision within 30 days.
</t>
      </section>
    </section>
  </middle>
  <back>

<references title="Normative References">
    <references>
      <name>References</name>
      <references>
        <name>Normative References</name>
        <reference
 anchor="IBA"> anchor="IBA" target="https://www.infinibandta.org/">
          <front>
            <title>InfiniBand Architecture Specification Volume 1</title>
            <seriesInfo name="Release" value="1.3"/>
            <author>
              <organization>InfiniBand Trade Association</organization>
            </author>
            <date month="March" year="2015"/>
          </front>
<seriesInfo name="Release" value="1.3"/>
<annotation>
Available from https://www.infinibandta.org/
</annotation>
        </reference>

<?rfc include="reference.RFC.2119.xml"?>
<?rfc include="reference.RFC.4506.xml"?>
<?rfc include="reference.RFC.5040.xml"?>
<?rfc include="reference.RFC.5042.xml"?>
<?rfc include="reference.RFC.8126.xml"?>
<?rfc include="reference.RFC.8166.xml"?>
<?rfc include="reference.RFC.8174.xml"?>

<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.2119.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.4506.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.5040.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.5042.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.8126.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.8166.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.8174.xml"/>
      </references>
      <references>
        <name>Informative References</name>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.1813.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.5041.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.5044.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.5531.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.5666.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.6581.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.7530.xml"/>
<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.7861.xml"/>
      </references>

<references title="Informative References">

<?rfc include="reference.RFC.1813.xml"?>
<?rfc include="reference.RFC.5044.xml"?>
<?rfc include="reference.RFC.5531.xml"?>
<?rfc include="reference.RFC.5666.xml"?>
<?rfc include="reference.RFC.6581.xml"?>
<?rfc include="reference.RFC.7530.xml"?>
<?rfc include="reference.RFC.7861.xml"?>
    </references>

    <section
 title="Acknowledgments"
 anchor="section:3910182B-831B-4CE3-9A51-B23F0C26A43B"
 numbered="no"> anchor="section_3910182B-831B-4CE3-9A51-B23F0C26A43B" numbered="false" toc="default">
      <name>Acknowledgments</name>
      <t>
Thanks to
Christoph Hellwig
and
Devesh Sharma
<contact fullname="Christoph Hellwig"/>
and
<contact fullname="Devesh Sharma"/>
for suggesting this approach,
and to
Tom Talpey
and
Dave Noveck
<contact fullname="Tom Talpey"/>
and
<contact fullname="David Noveck"/>
for their expert comments and review.
The author also wishes to thank
Bill Baker
and
Greg Marsden
<contact fullname="Bill Baker"/>
and
<contact fullname="Greg Marsden"/>
for their support of this work.
Also, thanks to expert reviewers
Sean Hefty
and
Dave Minturn.
<contact fullname="Sean Hefty"/>
and
<contact fullname="Dave Minturn"/>.
</t>
      <t>
Special thanks go to
document shepherd
Brian Pawlowski,
<contact fullname="Brian Pawlowski"/>,
Transport Area Director
Magnus Westerlund,
<contact fullname="Magnus Westerlund"/>,
NFSV4 Working Group Chairs
David Noveck
and
Spencer Shepler,
<contact fullname="David Noveck"/>
and
<contact fullname="Spencer Shepler"/>,
and
NFSV4 Working Group Secretary Thomas Haynes. <contact fullname="Thomas Haynes"/>.
</t>
    </section>
  </back>
</rfc>