DetNet

Internet Engineering Task Force (IETF)                     B. Varga, Ed.
Internet-Draft
Request for Comments: 9550                                     J. Farkas
Intended status:
Category: Informational                                         Ericsson
Expires: 18 July 2024
ISSN: 2070-1721                                                S. Kehrer
                                                                 T. Heer
                                  Hirschmann Automation and Control GmbH
                                                         15 January
                                                              March 2024

      Deterministic Networking (DetNet): Packet Ordering Function
                        draft-ietf-detnet-pof-11

Abstract

   Replication

   The replication and Elimination elimination functions of DetNet Architecture the Deterministic
   Networking (DetNet) architecture can result in out-of-order packets,
   which is not acceptable for some time-sensitive applications.  The
   Packet Ordering Function (POF)
   algorithm algorithms described herein enables to restore in this document
   enable restoration of the correct packet order when the replication
   and elimination functions are used in DetNet networks.  The POF only
   provides ordering within the latency bound of a DetNet flow, and flow; it does
   not provide any additional reliability.

Status of This Memo

   This Internet-Draft document is submitted in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents not an Internet Standards Track specification; it is
   published for informational purposes.

   This document is a product of the Internet Engineering Task Force
   (IETF).  Note that other groups may also distribute
   working documents as Internet-Drafts.  The list  It represents the consensus of current Internet-
   Drafts is at https://datatracker.ietf.org/drafts/current/.

   Internet-Drafts are draft the IETF community.  It has
   received public review and has been approved for publication by the
   Internet Engineering Steering Group (IESG).  Not all documents valid
   approved by the IESG are candidates for a maximum any level of Internet
   Standard; see Section 2 of RFC 7841.

   Information about the current status of six months this document, any errata,
   and how to provide feedback on it may be updated, replaced, or obsoleted by other documents obtained at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   This Internet-Draft will expire on 18 July 2024.
   https://www.rfc-editor.org/info/rfc9550.

Copyright Notice

   Copyright (c) 2024 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents (https://trustee.ietf.org/
   license-info)
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Revised BSD License text as described in Section 4.e of the
   Trust Legal Provisions and are provided without warranty as described
   in the Revised BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
   2.  Terminology . . . . . . . . . . . . . . . . . . . . . . . . .   3
     2.1.  Terms Used in This Document . . . . . . . . . . . . . . .   4
     2.2.  Abbreviations . . . . . . . . . . . . . . . . . . . . . .   4
   3.  Requirements on for POF Implementations . . . . . . . . . . . . .   4
   4.  POF Algorithms  . . . . . . . . . . . . . . . . . . . . . . .   5
     4.1.  Prerequisites and Assumptions . . . . . . . . . . . . . .   5
     4.2.  POF building blocks . . . . . . . . . . . . . . . . . . .   5 Building Blocks
     4.3.  The Basic POF Algorithm . . . . . . . . . . . . . . . . .   6
     4.4.  The Advanced POF Algorithm  . . . . . . . . . . . . . . .   8
     4.5.  Further enhancements Enhancements of the POF algorithms  . . . . . . . . .   9 Algorithms
     4.6.  Selecting and using Using the POF algorithm . . . . . . . . . .  10 Algorithms
   5.  Control and Management Plane Parameters for POF . . . . . . .  10
   6.  Security Considerations . . . . . . . . . . . . . . . . . . .  11
   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .  11
   8.  Acknowledgements  . . . . . . . . . . . . . . . . . . . . . .  11
   9.  References  . . . . . . . . . . . . . . . . . . . . . . . . .  11
     9.1.
     8.1.  Normative References  . . . . . . . . . . . . . . . . . .  11
     9.2.
     8.2.  Informative References  . . . . . . . . . . . . . . . . .  11
   Acknowledgements
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .  12

1.  Introduction

   The DetNet Working Group has defined packet replication

   [RFC8655] defines the Packet Replication Function (PRF) and
   packet elimination Packet
   Elimination Function (PEF) functions in DetNet for achieving extremely low
   packet loss.  The PRF and PEF are described in [RFC8655] and provide service protection for DetNet
   flows.  This service protection method relies on copies of the same
   packet sent over multiple maximally disjoint paths and uses
   sequencing information to eliminate duplicates.  A possible
   implementation of the PRF and PEF functions is described in
   [IEEE8021CB] [IEEE8021CB], and
   the related YANG model is defined in [IEEEP8021CBcv].

   In general, use of per packet per-packet replication and elimination functions
   can result in out-of-order delivery of packets, which is not
   acceptable for some deterministic applications.  Correcting packet
   order is not a trivial task, therefore task; therefore, details of a Packet Ordering
   Function (POF) are specified herein.  The IETF DetNet WG has defined in this document.  [RFC8655] defines the
   external observable result of a POF function, i.e., (i.e., that packets are reordered,
   reordered) but without does not specify any implementation details.

   So far in packet networks, out-of-order delivery situations were have been
   handled at higher OSI layers at the end-points/hosts endpoints/hosts (e.g., in the TCP
   stack when packets are sent to the application layer) and not within
   a network in nodes acting at the Layer-2 Layer 2 or Layer-3 Layer 3 OSI layers.

   Figure 1 shows a DetNet flow on which packet replication, elimination Packet Replication,
   Elimination, and ordering Ordering Functions (PREOF) functions are applied during
   forwarding from source to destination.

                                        +------------+
                 +-----------E1----+    |            |
    +----+       |            |    +---R3---+        |          +----+
    |src |------R1        +---+             |        E3----O1---+ dst|
    +----+       |        |                 E2-------+          +----+
                 +-------R2                 |
                          +-----------------+

    R: replication point (PRF)
    E: elimination point (PEF)
    O: ordering function (POF)

                Figure 1: PREOF scenario Scenario in a DetNet network Network

   In general, the use of PREOF functions require requires sequencing information to be
   included in the packets of a DetNet compound flow.  This can be done
   by adding a sequence number as part of DetNet encapsulation
   [RFC8655].  Sequencing information is typically added once, at or
   close to the source.

   Important

   It is important to note that different applications can react
   differently to out-of-order delivery.  A single out-of-order packet (E.g.,
   (e.g., packet
   order: order #1, #3, #2, #4, #5) is interpreted by some
   application as a single error, but some other applications treat it as 3
   three errors in-
   a-row situation. in a row.  For example, in industrial scenarios 3 scenarios, three
   errors in-
   a-row in a row is a usual typical error threshold and can cause the
   application to stop (e.g., to go to a fail-safe state).

   The POF ensures in-order delivery for packets being within the latency
   bound of the (DetNet) DetNet flow.  The POF does not correct errors in the
   packet flow e.g., (e.g., duplicate packets, packets or packets that are too late packets. late).

2.  Terminology

2.1.  Terms Used in This Document

   This document uses the terminology established in the DetNet
   architecture [RFC8655], and [RFC8655]; the reader is assumed to be familiar with
   that document and its terminology.

2.2.  Abbreviations

   The following abbreviations are used in this document:

   DetNet   Deterministic Networking. Networking

   PEF      Packet Elimination Function. Function

   POF      Packet Ordering Function. Function

   PREOF    Packet Replication, Elimination Elimination, and Ordering Functions. Functions

   PRF      Packet Replication Function. Function

3.  Requirements on for POF Implementations

   The requirements on a for POF function implementations are:

   *  to  To solve the out-of-order delivery problem of the Replication replication and
      Elimination
      elimination functions of DetNet networks.

   *  to  To consider the delay bound requirement of a DetNet Flow. flow.

   *  to  To be simple and to require in network nodes only a minimum set of
      states/configuration parameters states,
      configuration parameters, and resources per DetNet Flow. flow in network
      nodes.

   *  to  To add only minimal or no delay to the forwarding process of packets.

   *  To not to require synchronization between PREOF nodes.

   Some aspects are explicitly out-of-scope out of scope for a POF function: POF:

   *  to  To eliminate the delay variation caused by the packet ordering.
      Dealing with delay variation is a DetNet forwarding sub-layer
      target
      target, and it can be achieved achieved, for example example, by placing a de-jitter de-
      jitter buffer or flow regulator (e.g., shaping) function after the POF
      functionality.
      POF.

4.  POF Algorithms

4.1.  Prerequisites and Assumptions

   The POF Algorithm algorithms discussed in this document makes make some assumptions
   and tradeoffs trade-offs regarding the characteristics of the sequence of
   received packets.  In particular, the algorithm assumes algorithms assume that a Packet
   Elimination Function (PEF) PEF is
   performed on the incoming packets before they are handed to the POF function. POF.
   Hence, the sequence of incoming packets can be out of order out-of-order or
   incomplete but cannot contain duplicate packets.  However, the PREOF functions
   run independently without any state exchange required between the PEF
   and the POF or the PRF and the POF.  Error cases in which duplicate
   packets are presented to the POF can lead to out of order out-of-order delivery of
   duplicate packets as well as and to increased delays.

   The algorithm algorithms further requires require that the delay difference between two
   replicated packets that arrive at the PEF before the POF is bounded
   and known.  Error cases that violate this condition (e.g., a packet
   that arrives later than this bound) will result in out-of order out-of-order
   packets.

   The algorithm algorithms also makes make some tradeoffs. trade-offs.  For simplicity, it is
   designed in a way that allows to allow for some out of order out-of-order packets directly after
   initialization.  If this is not acceptable, Section 4.5 provides an
   alternative initialization scheme that prevents out-of-
   order out-of-order packets
   in the initialization phase.

4.2.  POF building blocks Building Blocks

   The method described herein in this document provides a POF for DetNet
   networks.  The configuration parameters of the POF can be derived during
   when engineering the DetNet flow through the network.

   The POF method is provided via:

   1. via the following:

   Delay calculator: calculates  Calculates buffering time for out-of-order
      packets.  Buffering time considers (i) the delay difference of
      paths used for forwarding the replicated packets and (ii) the
      bounded delay requirement of the given DetNet flow.

   2.

   Conditional delay buffer:  Used for buffering the out-of-order
      packets of a DetNet flow for a given time.

   Note: the The conditional delay buffer of the POF increases the
   burstiness of the traffic as it only adds delay only for some of the
   packets.

   Figure 2 shows the building blocks of a possible POF implementation.

                    +------------+        +--------------+
                    | Delay calc |        | Conditional  |
                 +--| for packet >--->>---| Delay Buffer >--+
                 |  +------------+        +--------------+  |
                 |                                          |
          +------^--------+                                 |
     ->>--| POF selector  >---------------------------------+-->>----
          | (Flow ident.) |
          +---------------+

     ->>- packet flow

                       Figure 2: POF Building Blocks

4.3.  The Basic POF Algorithm

   The basic POF algorithm delays all out-of-order packets until all
   previous packet arrives packets arrive or a given time (POFMaxDelay) ("POFMaxDelay") elapses.  The
   basic POF algorithm works as follows:

   *  The sequence number of the last forwarded packet (POFLastSent) ("POFLastSent")
      is stored for each DetNet Flow. flow.

   *  The sequence number (seq_num) of a received packet is compared to
      that of the last forwarded one (POFLastSent). ("POFLastSent").

   *  If (seq_num <= POFLastSent + 1)

      -  Then the packet is forwarded without buffering buffering, and
         "POFLastSent" is updated (POFLastSent = seq_num).

      -  Else  Else, the received packet is buffered.

   *  A buffered packet is forwarded from the buffer when its seq_num
      becomes equal to "POFLastSent +1," + 1" OR a predefined time
      ("POFMaxDelay") elapses.

   *  When a packet is forwarded from the buffer buffer, "POFLastSent" is
      updated with its seq_num (POFLastSent = seq_num).

   Note: the

   Notes:

   *  The difference of between sequence number numbers in consecutive packets is
      bounded due to the history window of the Elimination elimination function
      before the POF.  Therefore  Therefore, "<=" can be evaluated despite of the
      circular sequence number space.  A possible implementation of the
      PEF function and the impact of the history window is are described in
      [IEEE8021CB].

   Note2:

   *  The basic POF algorithm can be extended to cope with multiple
      failure scenarios (i.e., simultaneous packet loss and out-of-order packets),
      packets) when the expiration of the timer for a packet with
      sequence number N
   trigger triggers the release of some number of packets with a
      sequence number smaller than N.

   The state used by the basic POF algorithm (i.e., "POFLastSent") needs
   initialization and maintenance.  This works as follows:

   *  The next received packet is forwarded and the POFLastSent "POFLastSent"
      updated when the POF function was is reset OR no packet was is received for a
      predefined time ("POFTakeAnyTime").

   *  The reset of the POF erases all packets from the time-based buffer
      used by the POF.

   The basic POF algorithm has two parameters to engineer:

   *  "POFMaxDelay", which cannot be smaller than the delay difference
      of the paths used by the flow.

   *  "POFTakeAnyTime", which is calculated based on several factors,
      for example example, the RECOVERY_TIMEOUT related settings of the
      Elimination elimination function(s) relating
      to RECOVERY_TIMEOUT before the POF, the flow characteristics
      (e.g., inter packet inter-packet time), and the delay difference of the paths
      used by the flow.

   Design of these parameters is out-of-scope in out of scope for this document.

   Note: multiple Multiple network failures can impact the POF function (e.g., complete
   outage of all redundant paths).

   The basic POF algorithm increases the delay of packets with maximum
   "POFMaxDelay" time.  Packets being in order  In-order packets are not delayed.  This basic
   POF method can be applied in all network scenarios where the
   remaining delay budget of a flow at the POF point is larger than
   "POFMaxDelay" time.

   Figure 3 shows the delay budget relations situation at the POF point.

                             Path delay
                             difference
                           /-------------/
   <- path with min delay ->             /-- remaining delay budget --/

   |-----------------------|-------------|----------------------------|
   0                       t1            t2                           T

   <-------- path with max delay -------->

   /-------------------- delay budget at POF point -------------------/

             Figure 3: Delay Budget Relations Situation at the POF Point

4.4.  The Advanced POF Algorithm

   In network scenario scenarios where the remaining delay budget of a flow at
   the POF point is smaller than "POFMaxDelay" time time, the basic method
   needs extensions.

   The issue is that packets on the longest path cannot be buffered in
   order to keep the delay budget of the flow.  It must be noted that
   such a packet (i.e., forwarded over the longest path) needs no
   buffering as it is the "last chance" last chance to deliver a packet with a given
   sequence number.  This is because all replicas are already arrived via a
   shorter path(s).

   The advanced POF algorithm needs two requires extensions of the basic POF
   algorithm:

   *  to identify the received packet's path at the POF location and

   *  to make the value of "POFMaxDelay" for buffered packets path
      dependent ("POFMaxDelay_i", where "i" notes the path the packet
      has used).

   By identifying

   The advanced POF algorithm identifies the path of a given packet, the POF algorithm can use packet and
   uses this information to select what the predefined time "POFMaxDelay_i" ("POFMaxDelay_i")
   to apply for the buffered packet.  So, in the advanced POF algorithm algorithm,
   "POFMaxDelay" is an array, array that contains the predefined and path path-
   specific buffering time for each redundant path of a flow.  Values in
   the "POFMaxDelay" array are engineered to fulfill the delay budget
   requirement.

   Design of these parameters is out-of-scope in out of scope for this document.

   Note: for For the "Advanced advanced POF Algorithm" algorithm, the path dependent path-dependent delays might
   result in multiple packets triggering the "maximum delay reached" at
   exactly the same time.  The transmission order of these packets then
   should be done in their seq_num order.

   The method for identification of identifying the packet's path at the POF location
   depends on the network scenario.  It can be implemented via various
   techniques, for example example, using ingress interface information,
   encoding the path in the packet itself (e.g., replication functions
   can
   set a different FlowID per member flow at their egress what can be and such a
   FlowID is used as to identify the path of a PathID), packet at the POF), or
   in other
   means.  Method  Methods for identification of identifying the packet's path is are out of scope in
   for this document.

   Note: in case of When using the advanced POF algorithm algorithm, it might be advantageous
   to combine PEF and POF locations in the DetNet network, as it this can
   simplify the method used for identification of identifying the packet's path at the POF
   location.

4.5.  Further enhancements Enhancements of the POF algorithms Algorithms

   POF algorithms can be further enhanced by distinguishing the case of
   initialization from normal operation at the price of more states and
   more sophisticated implementation.  Such enhancements could could, for
   example
   example, react better after some failure scenarios (e.g., complete
   outage of all paths of a DetNet flow) and can be dependent on the PEF
   implementation.

   The challenge for POF initialization is that for example after a
   reset it is not known whether
   the first received packet is in-order or out-of-order. out-of-order (for example,
   after a reset).  The original initialization (see before) Section 4.3)
   considers the first packet as in-order, so out-of-order packet(s)
   during "POFMaxTime"/"POFMaxTime_path_i" time - -- after the first
   packet was is received - -- cannot be corrected.  Motivation  The motivation behind
   such an initialization is simplicity of POF implementation simplicity. implementation.

   A possible enhancement of POF initialization works as follows:

   *  After a reset reset, all received packets are buffered with their
      predefined timer ("POFMaxTime"/"POFMaxTime_path_i").

   *  No basic/advanced basic or advanced POF rules are applied until the first timer
      expires.

   *  When the first timer expires expires, the packet with lowest seq_num in
      the buffer is selected, selected and forwarded, and "POFLastSent" is set
      with its seq_num.

   *  The basic/advanced basic or advanced POF rules are applied for the packet(s) in
      the buffer and the subsequently received packets.

4.6.  Selecting and using Using the POF algorithm Algorithms

   The selection of the POF algorithm depends on the network scenario
   and the remaining delay budget of a flow.  Using the POF algorithms
   and calculating
   its their parameters require proper design.  Knowing the
   path delay difference is essential for the POF algorithms described
   here.  Failure scenarios breaking the design assumptions can impact
   the result of the POF (e.g., packet received out of the expected
   worst-case delay window - -- calculated based on the path delay
   difference - -- can result in unwanted out-of-order delivery).

   In DetNet scenarios scenarios, there is always an Elimination elimination function before
   the POF (therefore (therefore, duplicates are not considered by the POF).
   Implementing them together in the same node allows the POF to
   consider PEF events/states during the re-ordering. reordering.  For example, under
   normal
   circumstances circumstances, the difference of between sequence number numbers in
   consecutive packets is bounded due to the history window of the PEF.
   However, in some scenarios (e.g., reset of sequence number) number), the
   difference can be much larger than the size of the history window size. window.

5.  Control and Management Plane Parameters for POF

   POF algorithms needs setting of require the following parameters: parameters to be set:

   *  Basic POF

      -  "POFMaxDelay"

      -  "POFTakeAnyTime"

   *  Advanced POF

      -  "POFMaxDelay_i" for each possible path i

      -  "POFTakeAnyTime"

      -  Network  Configuration(s) related to network path identification related configuration(s)

   Note, that in

   Note: In a proper design design, "POFTakeAnyTime" is always larger than
   "POFMaxDelay".

6.  Security Considerations

   PREOF related

   PREOF-related security considerations (including POF) are described
   in section Section 3.3 of [RFC9055].  There are no additional POF related POF-related
   security considerations originating from this document.

7.  IANA Considerations

   This document makes has no IANA requests. actions.

8.  Acknowledgements

   Authors extend their appreciation to Gyorgy Miklos, Mohammadpour
   Ehsan, Ludovic Thomas, Greg Mirsky, Jeong-dong Ryoo, Shirley Yangfan,
   Toerless Eckert, Norman Finn and Ethan Grossman for their insightful
   comments and productive discussion that helped to improve the
   document.

9.  References

9.1.

8.1.  Normative References

   [RFC8655]  Finn, N., Thubert, P., Varga, B., and J. Farkas,
              "Deterministic Networking Architecture", RFC 8655,
              DOI 10.17487/RFC8655, October 2019,
              <https://www.rfc-editor.org/info/rfc8655>.

   [RFC9055]  Grossman, E., Ed., Mizrahi, T., and A. Hacker,
              "Deterministic Networking (DetNet) Security
              Considerations", RFC 9055, DOI 10.17487/RFC9055, June
              2021, <https://www.rfc-editor.org/info/rfc9055>.

9.2.

8.2.  Informative References

   [IEEE8021CB]
              IEEE, "IEEE Standard for Local and metropolitan area
              networks -- Frame Replication and Elimination for
              Reliability", IEEE Std 802.1CB-2017,
              DOI 10.1109/IEEESTD.2017.8091139, October 2017,
              <https://standards.ieee.org/standard/802_1CB-2017.html>.

   [IEEEP8021CBcv]
              Kehrer, S., "FRER
              IEEE, "IEEE Standard for Local and metropolitan area
              networks -- Frame Replication and Elimination for
              Reliability - Amendment 1: Information Model, YANG Data Model
              Model, and Management Information Base Module", IEEE P802.1CBcv
              /D1.2 P802.1CBcv, March 2021,
              <https://www.ieee802.org/1/files/private/cv-drafts/d1/802-
              1CBcv-d1-2.pdf>. Std
              802.1CBcv-2001, DOI 10.1109/IEEESTD.2022.9715061, February
              2022, <https://standards.ieee.org/ieee/802.1CBcv/7285/>.

Acknowledgements

   Authors extend their appreciation to Gyorgy Miklos, Ehsan
   Mohammadpour, Ludovic Thomas, Greg Mirsky, Jeong-dong Ryoo, Fan Yang,
   Toerless Eckert, Norman Finn, and Ethan Grossman for their insightful
   comments and productive discussion that helped to improve the
   document.

Authors' Addresses

   Balazs Varga (editor)
   Ericsson
   Budapest
   Magyar Tudosok krt. 11.
   1117
   Hungary
   Email: balazs.a.varga@ericsson.com

   Janos Farkas
   Ericsson
   Budapest
   Magyar Tudosok krt. 11.
   1117
   Hungary
   Email: janos.farkas@ericsson.com

   Stephan Kehrer
   Hirschmann Automation and Control GmbH
   Stuttgarter Strasse 45-51.
   72654 Neckartenzlingen
   Germany
   Email: Stephan.Kehrer@belden.com

   Tobias Heer
   Hirschmann Automation and Control GmbH
   Stuttgarter Strasse 45-51.
   72654 Neckartenzlingen
   Germany
   Email: Tobias.Heer@belden.com