<?xml version="1.0" encoding="US-ASCII"?> encoding="UTF-8"?>

<!-- updated by Chris 01/08/21 -->

<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
<?rfc toc="yes"?>
<?rfc tocompact="yes"?>
<?rfc tocdepth="3"?>
<?rfc tocindent="yes"?>
<?rfc symrefs="yes"?>
<?rfc sortrefs="yes"?>
<?rfc comments="yes"?>
<?rfc inline="yes"?>
<?rfc compact="yes"?>
<?rfc subcompact="no"?> "rfc2629-xhtml.ent">

<rfc category="info" xmlns:xi="http://www.w3.org/2001/XInclude" docName="draft-ietf-bmwg-b2b-frame-04"
number="9004" ipr="trust200902"
     updates="2544"> updates="2544" obsoletes="" submissionType="IETF"
category="info" consensus="true" xml:lang="en" tocInclude="true" tocDepth="3"
symRefs="true" sortRefs="true" version="3">

  <!-- xml2rfc v2v3 conversion 3.5.0 -->
  <front>
    <title abbrev="B2B Frame Update">Updates for the Back-to-back Back-to-Back Frame
    Benchmark in RFC 2544</title>
    <seriesInfo name="RFC" value="9004"/>
    <author fullname="Al Morton" initials="A." surname="Morton">
      <organization>AT&amp;T Labs</organization>
      <address>
        <postal>
          <street>200 Laurel Avenue South</street>

          <city>Middletown,</city>
          <city>Middletown</city>
          <region>NJ</region>
          <code>07748</code>

          <country>USA</country>
          <country>United States of America</country>
        </postal>
        <phone>+1 732 420 1571</phone>

        <facsimile>+1 732 368 1192</facsimile>
        <email>acmorton@att.com</email>
        <uri/>
      </address>
    </author>
    <date day="18" month="December" year="2020"/> month="May" year="2021"/>

<keyword>Buffer size</keyword>
<keyword>Buffer delay</keyword>
<keyword>Correction Factor</keyword>
    <abstract>
      <t>Fundamental Benchmarking Methodologies benchmarking methodologies for Network Interconnect
      Devices network interconnect
      devices of interest to the IETF are defined in RFC 2544. This memo
      updates the procedures of the test to measure the Back-to-back frames
      Benchmark Back-to-Back Frames
      benchmark of RFC 2544, based on further experience.</t>
      <t>This memo updates Section 26.4 of RFC 2544.</t>
    </abstract>

    <note title="Requirements Language">
      <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
      "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
      "OPTIONAL" in this document are to be interpreted as described in BCP
      14<xref target="RFC2119"/> <xref target="RFC8174"/> when, and only when,
      they appear in all capitals, as shown here.</t>

      <t/>
    </note>
  </front>
  <middle>
    <section title="Introduction"> numbered="true" toc="default">
      <name>Introduction</name>
      <t>The IETF's fundamental Benchmarking Methodologies benchmarking methodologies are defined in
      <xref target="RFC2544"> target="RFC2544" format="default"> </xref>, supported by the terms and definitions
      in <xref target="RFC1242"/>, and target="RFC1242" format="default"/>.  <xref target="RFC2544"/> target="RFC2544" format="default"/> actually
      obsoletes an earlier specification, <xref target="RFC1944"/>. target="RFC1944" format="default"/>. Over time,
      the benchmarking community has updated <xref target="RFC2544"/> target="RFC2544" format="default"/> several
      times, including the Device Reset Benchmark benchmark <xref target="RFC6201"/>, target="RFC6201" format="default"/>
      and the important Applicability Statement <xref target="RFC6815"/> target="RFC6815" format="default"/>
      concerning use outside the Isolated Test Environment (ITE) required for
      accurate benchmarking. Other specifications implicitly update <xref
      target="RFC2544"/>, target="RFC2544" format="default"/>, such as the IPv6 Benchmarking Methodologies benchmarking methodologies in <xref
      target="RFC5180"/>.</t> target="RFC5180" format="default"/>.</t>
      <t>Recent testing experience with the Back-to-back Back-to-Back Frame test and
      Benchmark
      benchmark in Section 26.4 of <xref target="RFC2544"/> target="RFC2544" sectionFormat="of" section="26.4"/> indicates that an
      update is warranted <xref target="OPNFV-2017"/> target="OPNFV-2017" format="default"/> <xref
      target="VSPERF-b2b"/>. target="VSPERF-b2b" format="default"/>. In particular, analysis of the results indicates
      that buffer size matters when compensating for interruptions of software
      packet software-packet processing, and this finding increases the importance of the
      Back-to-back frame
      Back-to-Back Frame characterization described here. This memo describes provides
      additional rationale and provides the updated method.</t>

      <t><xref target="RFC2544"/> (which obsoletes <xref target="RFC1944"/>) target="RFC2544" format="default"/>
      provides its own Requirements Language requirements language consistent with <xref
      target="RFC2119"/>, target="RFC2119" format="default"/>, since <xref target="RFC1944"/> pre-dates target="RFC1944" format="default"/> (which it obsoletes) predates <xref
      target="RFC2119"/> and all target="RFC2119" format="default"/>.  All three memos share common authorship.
      Today,<xref target="RFC8174"/>
      Today, <xref target="RFC8174" format="default"/> clarifies the usage of Requirements
      Language, requirements
      language, so the requirements language presented in this memo are expressed in accordance with
      <xref target="RFC8174"/> terms, and target="RFC8174" format="default"/>. They are intended for those
      performing/reporting laboratory tests to improve clarity and
      repeatability, and for those designing devices that facilitate these
      tests.</t>
    </section>
    <section>
      <name>Requirements Language</name>
        <t>
    The key words "<bcp14>MUST</bcp14>", "<bcp14>MUST NOT</bcp14>", "<bcp14>REQUIRED</bcp14>", "<bcp14>SHALL</bcp14>", "<bcp14>SHALL
    NOT</bcp14>", "<bcp14>SHOULD</bcp14>", "<bcp14>SHOULD NOT</bcp14>", "<bcp14>RECOMMENDED</bcp14>", "<bcp14>NOT RECOMMENDED</bcp14>",
    "<bcp14>MAY</bcp14>", and "<bcp14>OPTIONAL</bcp14>" in this document are to be interpreted as
    described in BCP&nbsp;14 <xref target="RFC2119"/> <xref target="RFC8174"/>
    when, and only when, they appear in all capitals, as shown here.
        </t>
    </section>

    <section title="Scope numbered="true" toc="default">
      <name>Scope and Goals"> Goals</name>

      <t>The scope of this memo is to define an updated method to
      unambiguously perform tests, measure the benchmark(s), and report the
      results for Back-to-back Back-to-Back Frames (presently (as described in Section 26.4 of
      <xref target="RFC2544"/>).</t> target="RFC2544" sectionFormat="of" section="26.4"/>).</t>
      <t>The goal is to provide more efficient test procedures where possible, possible
      and to expand reporting with additional interpretation of the results.
      The tests described in this memo address the cases in which the maximum
      frame rate of a single ingress port cannot be transferred loss-free to
      an egress port without loss (for some frame sizes of interest).</t>

      <t><xref target="RFC2544"/> Benchmarks
      <t>Benchmarks as described in <xref target="RFC2544" format="default"/> rely on test conditions with
      constant frame sizes, with the goal of understanding what network device network-device
      capability has been tested. Tests with the smallest size stress the
      header processing
      header-processing capacity, and tests with the largest size stress the
      overall bit processing bit-processing capacity. Tests with sizes in-between in between may
      determine the transition between these two capacities.
      However,
      conditions simultaneously sending a mixture of Internet (IMIX) frame sizes
      (IMIX), sizes, such as those described in <xref target="RFC6985"/>, MUST NOT target="RFC6985" format="default"/>, <bcp14>MUST NOT</bcp14> be
      used in Back-to-back Back-to-Back Frame testing.</t>

      <t>Section 3 of <xref target="RFC8239"/>
      <t><xref target="RFC8239" sectionFormat="of" section="3"/> describes buffer size buffer-size testing
      for physical networking devices in a data center. The <xref
      target="RFC8239"/> Those methods measure buffer latency directly with traffic
      on multiple ingress ports that overload an egress port on the Device
      Under Test (DUT) and are not subject to the revised calculations
      presented in this memo. Likewise, the methods of <xref
      target="RFC8239"/> SHOULD target="RFC8239" format="default"/> <bcp14>SHOULD</bcp14> be used for test cases where the egress port egress-port
      buffer is the known point of overload.</t>
    </section>
    <section title="Motivation">
      <t>Section 3.1 of <xref target="RFC1242"/> numbered="true" toc="default" anchor="motivation">
      <name>Motivation</name>
      <t><xref target="RFC1242" sectionFormat="of" section="3.1"/> describes the rationale for
      the Back-to-back Back-to-Back Frames Benchmark. benchmark. To summarize, there are several
      reasons that devices on a network produce bursts of frames at the
      minimum allowed spacing; and it is, therefore, worthwhile to understand
      the Device Under Test (DUT) DUT limit on the length of such bursts in
      practice. Also, <xref target="RFC1242"/> states: <figure>
          <artwork><![CDATA[       "Tests The same document also states:</t>

      <blockquote>
	Tests of this parameter are intended to determine the extent
      of data buffering in the device."]]></artwork>
        </figure></t>

      <t>After device.</blockquote>

      <t>Since this test was defined, there have been occasional discussions
      of the stability and repeatability of the results, both over time and
      across labs. Fortunately, the Open Platform for Network Function
      Virtualization (OPNFV) VSPERF project's project on Virtual Switch Performance (VSPERF) Continuous Integration (CI)
      <xref target="VSPERF-CI"/> target="VSPERF-CI" format="default"/> testing routinely repeats Back-to-back Back-to-Back Frame
      tests to verify that test functionality has been maintained through
      development of the test control test-control programs. These tests were used as a
      basis to evaluate stability and repeatability, even across lab set-ups setups
      when the test platform was migrated to new DUT hardware at the end of
      2016.</t>
      <t>When the VSPERF CI results were examined <xref target="VSPERF-b2b"/>, target="VSPERF-b2b" format="default"/>,
      several aspects of the results were considered notable:<list
          style="numbers">
          <t>Back-to-back notable:</t>
      <ol spacing="normal" type="1"><li>Back-to-Back Frame Benchmark benchmark was very consistent for some fixed
          frame sizes, and somewhat variable for other frame sizes.</t>

          <t>The sizes.</li>
        <li>The number of Back-to-back Back-to-Back Frames with zero loss reported for
          large frame sizes was unexpectedly long (translating to 30 seconds
          of buffer time), and no explanation or measurement limit condition
          was indicated. It was important that the buffering time calculations
          were part of the referenced testing and analysis<xref
          target="VSPERF-b2b"> analysis <xref target="VSPERF-b2b" format="default"> </xref>, because the calculated buffer times time of
          30 seconds for some frame sizes were was clearly wrong or highly
          suspect. On the other hand, a result expressed only as a large
          number of Back-to-back Back-to-Back Frames does not permit such an easy
          comparison with reality.</t>

          <t>Calculation reality.</li>
        <li>Calculation of the extent of buffer time in the DUT helped to
          explain the results observed with all frame sizes (for sizes. For example,
          tests with some frame sizes cannot exceed the frame header
          processing frame-header-processing rate of the DUT and thus DUT, thus, no buffering occurs; therefore, occurs.  Therefore,
          the results depended on the test equipment and not the DUT).</t>

          <t>It DUT.</li>
        <li>It was found that a better estimate of the DUT buffer time could
          be calculated using measurements of both the longest burst in frames
          without loss and results from the Throughput tests conducted
          according to Section 26.1 of <xref target="RFC2544"/>. target="RFC2544" sectionFormat="of" section="26.1"/>. It is
          apparent that the DUT's frame processing frame-processing rate empties the buffer
          during a trial and tends to increase the "implied" buffer size buffer-size
          estimate (measured according to Section 26.4 of <xref
          target="RFC2544"/> target="RFC2544" sectionFormat="of" section="26.4"/> because many frames have departed the buffer when
          the burst of frames ends). A calculation using the Throughput
          measurement can reveal a "corrected" buffer size estimate.</t>
        </list></t> buffer-size estimate.</li>
      </ol>
      <t>Further, if the Throughput tests of Section 26.1 of <xref
      target="RFC2544"/> target="RFC2544" sectionFormat="of" section="26.1"/> are conducted as a prerequisite test, prerequisite, the number of
      frame sizes required for Back-to-back Back-to-Back Frame Benchmarking benchmarking can be reduced
      to one or more of the small frame sizes, or the results for large frame
      sizes can be noted as invalid in the results if tested anyway (these anyway. These are
      the larger frame sizes for which the back-to-back frame Back-to-Back Frame rate cannot
      exceed the frame header processing frame-header-processing rate of the DUT and little or no
      buffering occurs).</t> occurs.</t>
      <t>The material below provides the details of the calculation to
      estimate the actual buffer storage available in the DUT, using results
      from the Throughput tests for each frame size, size and the maximum
      theoretical frame rate Max
      Theoretical Frame Rate for the DUT links (which constrain the minimum
      frame spacing).</t>
      <t>In reality, there are many buffers and packet header processing packet-header-processing steps
      in a typical DUT. The simplified model used in these calculations for
      the DUT includes a packet header processing packet-header-processing function with limited rate
      of operation, as shown below:</t>

      <t><figure>
          <artwork><![CDATA[ in <xref target="simplified-model"/>.</t>
      <figure anchor="simplified-model">
	<name>Simplified Model for DUT Testing</name>
      <artwork name="" type="" align="left" alt="" ><![CDATA[
                     |------------ DUT --------|
Generator -> Ingress -> Buffer -> HeaderProc -> Egress -> Receiver
]]></artwork>
        </figure></t>
      </figure>
      <t>So, in the Back-to-back Back-to-Back Frame testing:<list style="numbers">
          <t>The testing:</t>
      <ol spacing="normal" type="1"><li>The ingress burst arrives at Max Theoretical Frame Rate, and
          initially the frames are buffered.</t>

          <t>The packet header processing buffered.</li>
        <li>The packet-header-processing function (HeaderProc) operates at
          the &ldquo;Measured Throughput&rdquo; (Section 26.1 of <xref
          target="RFC2544"/>), "Measured Throughput" (<xref target="RFC2544" sectionFormat="of" section="26.1"/>), removing frames from the buffer (this is the
          best approximation we have).</t>

          <t>Frames have, another acceptable approximation is the received frame rate
      during Back-to-back Frame testing, if Measured Throughput is
      not available).  </li>
        <li>Frames that have been processed are clearly not in the buffer, so
          the Corrected DUT buffer time Buffer Time equation (Section 5.4) (<xref target="bench-calc" />) estimates and
          removes the frames that the DUT forwarded on egress during the
          burst. We define buffer time as the number of frames occupying the
          buffer divided by the Maximum Max Theoretical Frame Rate (on ingress)
          for the frame size under test.</t>

          <t>A test.</li>
        <li>A helpful concept is the buffer filling buffer-filling rate, which is the
          difference between the Max Theoretical Frame Rate (ingress) and the
          Measured Throughput (HeaderProc on egress). If the actual buffer
          size in frames was is known, the time to fill the buffer during a
          measurement can be calculated using the filling rate rate, as a check on
          measurements. However, the buffer in the model represents many
          buffers of different sizes in the DUT data path.</t>
        </list></t> path.</li>
      </ol>
      <t>Knowledge of approximate buffer storage size (in time or bytes) may
      be useful to estimate in estimating whether frame losses will occur if DUT forwarding
      is temporarily suspended in a production deployment, deployment due to an
      unexpected interruption of frame processing (an interruption of duration
      greater than the estimated buffer would certainly cause lost frames). In
      Section 5,
      <xref target="b2b"/>, the calculations for the correct buffer time use the
      combination of offered load at Max Theoretical Frame Rate and header
      processing header-processing speed at 100% of Measured Throughput. Other combinations are
      possible, such as changing the percent of measured Measured Throughput to account
      for other processes reducing the header processing rate.</t>

      <t>The presentation of OPNFV VSPERF evaluation and development of
      enhanced search algorithms <xref target="VSPERF-BSLV"/> target="VSPERF-BSLV" format="default"/> was given and discussed at
      IETF-102.
      IETF 102.  The enhancements are intended to compensate for transient
      processor interrupts that may cause loss at near-Throughput levels of offered
      load. Subsequent analysis of the results indicates that buffers within
      the DUT can compensate for some interrupts, and this finding increases
      the importance of the Back-to-back frame Back-to-Back Frame characterization described
      here.</t>
    </section>
    <section title="Prerequisites"> numbered="true" toc="default">
      <name>Prerequisites</name>
      <t>The Test Setup MUST test setup <bcp14>MUST</bcp14> be consistent with Figure 1 of <xref
      target="RFC2544"/>, target="RFC2544" format="default"/>, or Figure 2 of that document when the tester's sender and receiver
      are different devices. Other mandatory testing aspects described in
      <xref target="RFC2544"/> MUST target="RFC2544" format="default"/> <bcp14>MUST</bcp14> be included, unless explicitly modified in
      the next section.</t>
      <t>The ingress and egress link speeds and link layer link-layer protocols MUST <bcp14>MUST</bcp14> be
      specified and used to compute the maximum theoretical frame rate Max Theoretical Frame Rate when
      respecting the minimum inter-frame interframe gap.</t>
      <t>The test results for the Throughput Benchmark benchmark conducted according to
      Section 26.1 of
      <xref target="RFC2544"/> target="RFC2544" sectionFormat="of" section="26.1"/> for all <xref
      target="RFC2544"/>-RECOMMENDED frame sizes MUST <bcp14>RECOMMENDED</bcp14> by <xref target="RFC2544" format="default"/> <bcp14>MUST</bcp14> be available to reduce
      the tested frame size list, tested-frame-size list or to note invalid results for individual
      frame sizes (because the burst length may be essentially infinite for
      large frame sizes).</t>
      <t>Note that:<list style="symbols">
          <t>the that:</t>
      <ul spacing="normal">
        <li>the Throughput and the Back-to-back Back-to-Back Frame measurement
          configuration measurement-configuration traffic characteristics (unidirectional or
          bi-directional,
          bidirectional, and number of flows generated) MUST match.</t>

          <t>the <bcp14>MUST</bcp14> match.</li>
        <li>the Throughput measurement MUST <bcp14>MUST</bcp14> be taken under zero-loss conditions,
          according to Section 26.1 of <xref target="RFC2544"/>.</t>
        </list>The Back-to-back target="RFC2544" sectionFormat="of" section="26.1"/>.</li>
      </ul>
      <t>The Back-to-Back Benchmark described in Section 3.1 of <xref
      target="RFC1242"/> MUST target="RFC1242" sectionFormat="of" section="3.1"/> <bcp14>MUST</bcp14> be measured directly by the tester, where buffer
      size is inferred from Back-to-back Back-to-Back Frame bursts and associated packet
      loss packet-loss measurements. Therefore, sources of packet frame loss that are unrelated
      to consistent evaluation of buffer size SHOULD <bcp14>SHOULD</bcp14> be identified and removed
      or mitigated. Example sources include:<list style="symbols">
          <t>On-path include:</t>
      <ul spacing="normal">
        <li>On-path active components that are external to the DUT</t>

          <t>Operating system DUT</li>
        <li>Operating-system environment interrupting DUT operation</t>

          <t>Shared resource operation</li>
        <li>Shared-resource contention between the DUT and other off-path
          component(s) impacting DUT's behaviour, behavior, sometimes called the "noisy
          neighbour"
          neighbor" problem with virtualized network functions.</t>
        </list></t> functions.</li>
      </ul>
      <t>Mitigations applicable to some of the sources above are discussed in
      Section 5.2,
      <xref target="frame-size"/>, with the other measurement requirements described below in
      Section 5.</t>
      <xref target="b2b"/>.</t>
    </section>
    <section title="Back-to-back Frames"> numbered="true" toc="default" anchor="b2b">
      <name>Back-to-Back Frames</name>
      <t>Objective: To characterize the ability of a DUT to process
      back-to-back frames
      Back-to-Back Frames as defined in <xref target="RFC1242"/>.</t> target="RFC1242" format="default"/>.</t>
      <t>The Procedure procedure follows.</t>
      <section title="Preparing numbered="true" toc="default">
        <name>Preparing the list List of Frame sizes"> Sizes</name>
        <t>From the list of RECOMMENDED <bcp14>RECOMMENDED</bcp14> frame sizes (Section 9 of <xref
        target="RFC2544"/>), (<xref target="RFC2544" sectionFormat="of" section="9"/>), select the subset of frame sizes whose measured Measured
        Throughput (during prerequisite testing) was less than the Maximum Max
        Theoretical Frame Rate of the DUT/test-set-up. DUT/test setup. These are the only
        frame sizes where it is possible to produce a burst of frames that
        cause the DUT buffers to fill and eventually overflow, producing one
        or more discarded frames.</t>
      </section>
      <section title="Test numbered="true" toc="default" anchor="frame-size">
        <name>Test for a Single Frame Size"> Size</name>
        <t>Each trial in the test requires the tester to send a burst of
        frames (after idle time) with the minimum inter-frame gap, interframe gap and to
        count the corresponding frames forwarded by the DUT.</t>
        <t>The duration of the trial includes three REQUIRED <bcp14>REQUIRED</bcp14> components: <list
            style="numbers">
            <t>The </t>
        <ol spacing="normal" type="1"><li>The time to send the burst of frames (at the back-to-back
            rate), determined by the search algorithm.</t>

            <t>The algorithm.</li>
          <li>The time to receive the transferred burst of frames (at the
            <xref target="RFC2544"/> target="RFC2544" format="default"/> Throughput rate), possibly truncated by
            buffer overflow, and certainly including the latency of the
            DUT.</t>

            <t>At
            DUT.</li>
          <li>At least 2 seconds not overlapping the time to receive the
            burst (2.), (Component 2, above), to ensure that DUT buffers have depleted. Longer times
            MUST
            <bcp14>MUST</bcp14> be used when conditions warrant, such as when buffer times
            &gt;2 seconds are measured or when burst sending times are &gt;2
            seconds, but care is needed needed, since this time component directly
            increases trial duration duration, and many trials and tests comprise a
            complete benchmarking study.</t>
          </list>The study.</li>
        </ol>
        <t>The upper search limit for the time to send each burst MUST <bcp14>MUST</bcp14>
        be configurable, configurable to values as high as 30 seconds (buffer time results
        reported at or near the configured upper limit are likely invalid, and
        the test MUST <bcp14>MUST</bcp14> be repeated with a higher search limit).</t>
        <t>If all frames have been received, the tester increases the length
        of the burst according to the search algorithm and performs another
        trial.</t>
        <t>If the received frame count is less than the number of frames in
        the burst, then the limit of DUT processing and buffering may have
        been exceeded, and the burst length for the next trial is determined by the search
        algorithm for the next trial (the burst length is typically reduced,
        but see below).</t>
        <t>Classic search algorithms have been adapted for use in
        benchmarking, where the search requires discovery of a pair of
        outcomes, one with no loss and another with loss, at load conditions
        within the acceptable tolerance or accuracy. Conditions encountered
        when benchmarking the Infrastructure infrastructure for Network Function
        Virtualization network function
        virtualization require algorithm enhancement. Fortunately, the
        adaptation of Binary Search, and an enhanced Binary Search with Loss
        Verification
        Verification, have been specified in clause Clause 12.3 of <xref
        target="TST009"/>. target="TST009" format="default"/>. These algorithms can easily be used for
        Back-to-back
        Back-to-Back Frame benchmarking by replacing the Offered Load offered load level
        with burst length in frames. <xref target="TST009"/> target="TST009" format="default"/>, Annex B describes
        the theory behind the enhanced Binary Search with Loss Verification
        algorithm.</t>
        <t>There is are also promising work-in-progress works in progress that may prove useful in
        Back-to-back
        Back-to-Back Frame benchmarking. <xref
        target="I-D.vpolak-mkonstan-bmwg-mlrsearch"/> target="I-D.vpolak-mkonstan-bmwg-mlrsearch" format="default"/> and <xref
        target="I-D.vpolak-bmwg-plrsearch"/> target="I-D.vpolak-bmwg-plrsearch" format="default"/> are two such examples.</t>
        <t>Either the <xref target="TST009"/> target="TST009" format="default"/> Binary Search or Binary Search
        with Loss Verification algorithms MUST <bcp14>MUST</bcp14> be used, and input parameters
        to the algorithm(s) MUST <bcp14>MUST</bcp14> be reported.</t>
        <t>The tester usually imposes a (configurable) minimum step size for
        burst length, and the step size MUST <bcp14>MUST</bcp14> be reported with the results (as
        this influences the accuracy and variation of test results).</t>
        <t>The original Section 26.4 of <xref target="RFC2544"/> target="RFC2544" sectionFormat="of" section="26.4"/> definition is
        stated below:<list style="empty">
            <t>The Back-to-back Frame below:</t>
	<blockquote>
	  The back-to-back value is the longest burst number of frames in the longest burst that the DUT can successfully process and buffer will handle without frame
            loss, as determined from the series loss of trials.</t>
          </list></t> any frames.
</blockquote>
      </section>
      <section title="Test numbered="true" toc="default" anchor="test-rep">
        <name>Test Repetition and Benchmark"> Benchmark</name>
        <t>On this topic, Section 26.4 of <xref target="RFC2544"/>
        requires:<list style="empty">
            <t>The target="RFC2544" sectionFormat="of" section="26.4"/>
        requires:</t>
	<blockquote>
          The trial length MUST <bcp14>MUST</bcp14> be at least 2 seconds and SHOULD <bcp14>SHOULD</bcp14> be
            repeated at least 50 times with the average of the recorded values
            being reported.</t>
          </list></t> reported.</blockquote>
        <t>Therefore, the Back-to-back Back-to-Back Frame Benchmark benchmark is the average of burst
        length burst-length values over repeated tests to determine the longest burst of
        frames that the DUT can successfully process and buffer without frame
        loss. Each of the repeated tests completes an independent search
        process.</t>

        <t>In
<!--[rfced] *AD - FYI, we edited the following sentence for clarity. Please check that the meaning of the requirements language has not changed.  AD approval is necessary for changes to requirements language.

Original:
In this update, the test MUST be repeated N times (the number of
repetitions is now a variable that must be reported),for each frame
size in the subset list, and each Back-to-back Frame value made
available for further processing (below).

Changed to:
In this update, the test MUST be repeated N times (the number of
repetitions is now a variable that must be reported) for each frame
size in the subset list, and each back-to-back frame value MUST be
made available for further processing (below).
-->

        <t>In this update, the test <bcp14>MUST</bcp14> be repeated N times (the number of
        repetitions is now a variable that must be reported) for each frame
        size in the subset list, and each Back-to-Back Frame value <bcp14>MUST</bcp14> be made
        available for further processing (below).</t>
      </section>
      <section title="Benchmark Calculations"> numbered="true" toc="default" anchor="bench-calc">
        <name>Benchmark Calculations</name>
        <t>For each frame size, calculate the following summary statistics for
        longest Back-to-back Back-to-Back Frame values over the N tests:<list
            style="symbols">
            <t>Average (Benchmark)</t>

            <t>Minimum</t>

            <t>Maximum</t>

            <t>Standard Deviation</t>
          </list></t> tests:</t>
        <ul spacing="normal">
          <li>Average (Benchmark)</li>
          <li>Minimum</li>
          <li>Maximum</li>
          <li>Standard Deviation</li>
        </ul>
        <t>Further, calculate the Implied DUT Buffer Time and the Corrected
        DUT Buffer Time in seconds, as follows:<figure>
            <artwork><![CDATA[Implied follows:</t>
	<sourcecode>
Implied DUT Buffer Time buffer time =

   Average num of Back-to-back Frames / Max Theoretical Frame Rate
]]></artwork>
          </figure>The
	</sourcecode>
        <t>The formula above is simply expressing the burst of frames
        in units of time.</t>
        <t>The next step is to apply a correction factor that accounts for the
        DUT's frame forwarding operation during the test (assuming the simple
        model of the DUT composed of a buffer and a forwarding function,
        described in Section 3).</t>

        <t><figure>
            <artwork><![CDATA[Corrected <xref target="motivation"/>).</t>
        <artwork name="" type="" align="left" alt=""><![CDATA[Corrected DUT Buffer Time =
                  /                                         \
   Implied DUT    |Implied DUT       Measured Throughput    |
=  Buffer Time -  |Buffer Time * -------------------------- |
                  |              Max Theoretical Frame Rate |
                  \                                         /]]></artwork>
          </figure></t>

        <t>where:<list style="numbers">
            <t>The &ldquo;Measured Throughput&rdquo;
        <t>where:</t>
        <ol spacing="normal" type="1"><li>The "Measured Throughput" is the <xref
            target="RFC2544"/> target="RFC2544" format="default"/> Throughput Benchmark for the frame size tested,
            as augmented by methods including the Binary Search with Loss
            Verification algorithm in <xref target="TST009"/> target="TST009" format="default"/> where
            applicable,
            applicable and MUST <bcp14>MUST</bcp14> be expressed in frames per second in this
            equation.</t>

            <t>The &ldquo;Max
            equation.</li>
          <li>The "Max Theoretical Frame Rate&rdquo; Rate" is a calculated
            value for the interface speed and link layer link-layer technology used, and
            MUST it
            <bcp14>MUST</bcp14> be expressed in frames per second in this equation.</t>
          </list></t> equation.</li>
        </ol>

        <t>The term on the far right in the formula for Corrected DUT Buffer
        Time accounts for all the frames in the Burst burst that were transmitted by
        the DUT *while <strong>while the Burst burst of frames were was sent in*. in</strong>.&nbsp; So, these frames are
        not in the buffer buffer, and the buffer size is more accurately estimated by
        excluding them.</t> them.  If Measured Throughput is not available,
  an acceptable approximation is the received frame rate (see Forwarding
  Rate in <xref target="RFC2889" format="default"/> measured during Back-to-back Frame testing).</t>
      </section>
    </section>
    <section title="Reporting"> numbered="true" toc="default">
      <name>Reporting</name>
      <t>The back-to-back frame Back-to-Back Frame results SHOULD <bcp14>SHOULD</bcp14> be reported in the format of a
      table with a row for each of the tested frame sizes. There SHOULD <bcp14>SHOULD</bcp14> be
      columns for the frame size and for the resultant average frame count for
      each type of data stream tested.</t>
      <t>The number of tests Averaged averaged for the Benchmark, benchmark, N, MUST <bcp14>MUST</bcp14> be
      reported.</t>
      <t>The Minimum, Maximum, minimum, maximum, and Standard Deviation standard deviation across all complete
      tests SHOULD <bcp14>SHOULD</bcp14> also be reported (they are referred to as "Min,Max,StdDev"
      in the table below).</t> <xref target="frame-results"/>).</t>
      <t>The Corrected DUT Buffer Time SHOULD <bcp14>SHOULD</bcp14> also be reported.</t>
      <t>If the tester operates using a limited maximum burst length in
      frames, then this maximum length SHOULD <bcp14>SHOULD</bcp14> be reported.</t>

      <texttable style="full" title="Back-to-Back
      <table align="center" anchor="frame-results">
        <name>Back-to-Back Frame Results">
        <ttcol>Frame Results</name>
        <thead>
          <tr>
            <th align="left">Frame Size, octets</ttcol>

        <ttcol>Ave octets</th>
            <th align="left">Ave B2B Length, frames</ttcol>

        <ttcol>Min,Max,StdDev</ttcol>

        <ttcol>Corrected frames</th>
            <th align="left">Min,Max,StdDev</th>
            <th align="left">Corrected Buff Time, Sec</ttcol>

        <c>64</c>

        <c>26000</c>

        <c>25500,27000,20</c>

        <c>0.00004</c>
      </texttable> Sec</th>
          </tr>
        </thead>
        <tbody>
          <tr>
            <td align="left">64</td>
            <td align="left">26000</td>
            <td align="left">25500,27000,20</td>
            <td align="left">0.00004</td>
          </tr>
        </tbody>
      </table>

      <t>Static and configuration parameters (reported with the table
      above):</t>

      <t>Number <xref target="frame-results"/>):</t>
      <ul>
      <li>Number of test repetitions, N</t>

      <t>Minimum N</li>
      <li>Minimum Step Size (during searches), in frames.</t> frames.</li>
      </ul>
      <t/>
      <t>If the tester has a specific (actual) frame rate of interest (less
      than the Throughput rate), it is useful to estimate the buffer time at
      that actual frame rate:</t>

      <figure>
        <artwork><![CDATA[Actual
      <artwork name="" type="" align="left" alt=""><![CDATA[Actual Buffer Time =
                                   Max Theoretical Frame Rate
     = Corrected DUT Buffer Time * --------------------------
                                       Actual Frame Rate
 ]]></artwork>
      </figure>
      <t>and report this value, properly labeled.</t>
    </section>
    <section title="Security Considerations"> numbered="true" toc="default">
      <name>Security Considerations</name>
      <t>Benchmarking activities as described in this memo are limited to
      technology characterization using controlled stimuli in a laboratory
      environment, with dedicated address space and the other constraints
      of<xref target="RFC2544"/>.</t>
      of <xref target="RFC2544" format="default"/>.</t>
      <t>The benchmarking network topology will be an independent test setup
      and MUST NOT <bcp14>MUST NOT</bcp14> be connected to devices that may forward the test traffic
      into a production network, network or misroute traffic to the test management
      network. See <xref target="RFC6815"/>.</t> target="RFC6815" format="default"/>.</t>
      <t>Further, benchmarking is performed on an "opaque-box" (a.k.a.
      "black-box") basis, relying solely on measurements observable external
      to the DUT/SUT.</t> Device or System Under Test (SUT).</t>
      <t>The DUT developers are commonly independent from the personnel and
      institutions conducting benchmarking studies. DUT developers might have
      incentives to alter the performance of the DUT if the test conditions
      can be detected. Special capabilities SHOULD NOT <bcp14>SHOULD NOT</bcp14> exist in the DUT/SUT
      specifically for benchmarking purposes. Procedures described in this
      document are not designed to detect such activity. Additional testing
      outside of the scope of this document would be needed and has been used
      successfully in the past to discover such malpractices.</t>
      <t>Any implications for network security arising from the DUT/SUT SHOULD <bcp14>SHOULD</bcp14>
      be identical in the lab and in production networks.</t>
    </section>
    <section anchor="IANA" title="IANA Considerations"> numbered="true" toc="default">
      <name>IANA Considerations</name>
      <t>This memo makes document has no requests of IANA.</t> IANA actions.</t>
    </section>

    <section title="Acknowledgments">
      <t>Thanks to Trevor Cooper, Sridhar Rao, and Martin Klozik of the VSPERF
      project for many contributions to the early testing <xref
      target="VSPERF-b2b"/>. Yoshiaki Itou has also investigated the topic,
      and made useful suggestions. Maciek Konstantyowicz and Vratko Polak also
      provided many comments and suggestions based on extensive integration
      testing and resulting search algorithm proposals
  </middle>
  <back>

<displayreference target="I-D.vpolak-bmwg-plrsearch" to="BMWG-PLRSEARCH"/>
<displayreference target="I-D.vpolak-mkonstan-bmwg-mlrsearch" to="BMWG-MLRSEARCH"/>

    <references>
      <name>References</name>
      <references>
        <name>Normative References</name>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.2544.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.1242.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.2119.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.6985.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.8174.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.8239.xml"/>

<!--[rfced] FYI - when reviewing the most up-to-date
      feedback possible. Tim Carlin also provided comments documents listed below, we do not see Al Morton listed as an author.

[TST009]   Morton, A., "ETSI GS NFV-TST 009 V3.4.1 (2020-12),
              "Network Functions Virtualisation (NFV) Release 3;
              Testing; Specification of Networking Benchmarks and support
              Measurement Methods for the
      draft. Warren Kumari's review improved readability in several key
      passages. David Black, Martin Duke, and Scott Bradner's comments
      improved the clarity and configuration advice on trial duration. Malisa
      Vucinic suggested additional text NFVI"", December 2020,
              <https://www.etsi.org/deliver/etsi_gs/NFV-
              TST/001_099/009/03.04.01_60/gs_NFV-TST009v030401p.pdf>.

[VSPERF-b2b]
              Morton, A., "Back2Back Testing Time Series (from CI)",
              June 2017, <https://wiki.opnfv.org/display/vsperf/
              Traffic+Generator+Testing#TrafficGeneratorTesting-
              AppendixB:Back2BackTestingTimeSeries(fromCI)>.

Linked wiki page states:
Traffic Generator Testing
Created by Trevor Cooper, last modified by Bob Fubel on DUT design cautions in the Security
      Considerations section.</t>
    </section>
  </middle>

  <back>
    <references title="Normative References">
      <?rfc include='reference.RFC.2544'?>

      <?rfc include='reference.RFC.1242'?>

      <?rfc include="reference.RFC.2119"?>

      <?rfc include='reference.RFC.6985'?>

      <?rfc include='reference.RFC.8174'?>

      <?rfc include='reference.RFC.8239'?> Oct 02, 2018
-->

        <reference anchor="TST009" target="https://www.etsi.org/deliver/etsi_gs/NFV-TST/001_099/009/03.04.01_60/gs_NFV-TST009v030401p.pdf">
          <front>
          <title>ETSI GS NFV-TST 009 V3.4.1 (2020-12), "Network
            <title>Network Functions
          Virtualisation (NFV) Release 3; Testing; Specification of Networking
          Benchmarks and Measurement Methods for NFVI"</title>

          <author fullname="Rapporteur: Al Morton" initials="A."
                  surname="Morton">
            <organization>ETSI Network Function Virtualization
            ISG</organization> NFVI</title>
            <author>
              <organization>ETSI</organization>
            </author>
            <date month="December" year="2020"/>
          </front>
	  <seriesInfo name="ETSI GS NFV-TST 009" value="v3.4.1"/>
	  <refcontent>Rapporteur: A. Morton</refcontent>
        </reference>

      <?rfc ?>

      <?rfc ?>

      <?rfc ?>

      <?rfc ?>

      <?rfc ?>

      <?rfc ?>

      <?rfc ?>

      </references>

    <references title="Informative References">
      <references>
        <name>Informative References</name>

        <reference anchor="OPNFV-2017"
                 target="https://wiki.opnfv.org/download/attachments/10293193/VSPERF-Dataplane-Perf-Cap-Bench.pptx?api=v2"> target="https://wiki.anuket.io/download/attachments/4404001/VSPERF-Dataplane-Perf-Cap-Bench.pdf?version=1&amp;modificationDate=1621191833500&amp;api=v2">
          <front>
            <title>Dataplane Performance, Capacity, and Benchmarking in
          OPNFV</title>
            <author fullname="Trevor Cooper" initials="T." surname="Cooper">
              <organization>Intel Corp.</organization>
            </author>
            <author fullname="Al Morton" initials="A." surname="Morton">
            <organization>AT&amp;T Labs</organization>
          </author>

          <author fullname="Sridhar Rao" initials="S." surname="Rao">
              <organization>Spirent Communications</organization>
            </author>
            <author fullname="Al Morton" initials="A." surname="Morton">
              <organization>AT&amp;T Labs</organization>
            </author>
            <date day="15" month="June" year="2017"/>
          </front>
        </reference>

        <reference anchor="VSPERF-b2b"
                 target="https://wiki.opnfv.org/display/vsperf/Traffic+Generator+Testing#TrafficGeneratorTesting-AppendixB:Back2BackTestingTimeSeries(fromCI)"> target="https://wiki.anuket.io/display/HOME/Traffic+Generator+Testing#TrafficGeneratorTesting-AppendixB:Back2BackTestingTimeSeries(fromCI)">
          <front>
            <title>Back2Back Testing Time Series (from CI)</title>
            <author fullname="Al Morton" initials="A." surname="Morton">
              <organization>AT&amp;T Labs</organization>
            </author>
            <date month="June" year="2017"/>
          </front>
        </reference>

        <reference anchor="VSPERF-CI"
                 target="https://wiki.opnfv.org/display/vsperf/VSPERF+CI"> target="https://wiki.anuket.io/display/HOME/VSPERF+CI">
          <front>
            <title>OPNFV VSPERF CI</title>
            <author fullname="Maryam Tahhan" initials="M." surname="Tahhan">
              <organization>Intel Corporation</organization>
            </author>
            <date month="June" year="2019"/>
          </front>
        </reference>

        <reference anchor="VSPERF-BSLV" target="https://datatracker.ietf.org/meeting/102/materials/slides-102-bmwg-evolution-of-repeatability-in-benchmarking-fraser-plugfest-summary-for-ietf-bmwg-00">
          <front>
            <title>Evolution of Repeatability in Benchmarking: Fraser Plugfest
          (Summary for IETF BMWG)</title>
            <author fullname="Al Morton" initials="A." surname="Morton">
            <organization>AT&amp;T Labs</organization>
          </author>

          <author fullname="Sridhar Rao" initials="S." surname="Rao">
              <organization>Spirent Communications</organization>
            </author>
            <author fullname="Al Morton" initials="A." surname="Morton">
              <organization>AT&amp;T Labs</organization>
            </author>
            <date month="July" year="2018"/>
          </front>
        </reference>

      <?rfc include='reference.RFC.1944'?>

      <?rfc include='reference.RFC.6201'?>

      <?rfc include='reference.RFC.6815'?>

      <?rfc include='reference.RFC.5180'?>

      <?rfc ?>

      <?rfc include='reference.I-D.vpolak-bmwg-plrsearch'?>

      <?rfc include='reference.I-D.vpolak-mkonstan-bmwg-mlrsearch'?>

      <?rfc ?>

        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.1944.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.6201.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.6815.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.5180.xml"/>

<xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.2889.xml"/>

<!-- [I-D.vpolak-bmwg-plrsearch] IESG state Expired -->

<reference anchor='I-D.vpolak-bmwg-plrsearch'>
<front>
<title>Probabilistic Loss Ratio Search for Packet Throughput (PLRsearch)</title>

<author role="editor" initials='M' surname='Konstantynowicz' fullname='Maciek Konstantynowicz'>
    <organization />
</author>

<author role="editor" initials='V' surname='Polak' fullname='Vratko Polak'>
    <organization />
</author>

<date month='March' day='6' year='2020' />

</front>
<seriesInfo name='Internet-Draft' value='draft-vpolak-bmwg-plrsearch-03' />
<format type='TXT'
        target='http://www.ietf.org/internet-drafts/draft-vpolak-bmwg-plrsearch-03.txt' />
</reference>

<!-- [I-D.vpolak-mkonstan-bmwg-mlrsearch] IESG state Expired -->

<reference anchor='I-D.vpolak-mkonstan-bmwg-mlrsearch'>
<front>
<title>Multiple Loss Ratio Search for Packet Throughput (MLRsearch)</title>

<author role="editor" initials='M' surname='Konstantynowicz' fullname='Maciek Konstantynowicz'>
    <organization />
</author>

<author role="editor" initials='V' surname='Polak' fullname='Vratko Polak'>
    <organization />
</author>

<date month='March' day='6' year='2020' />

</front>

<seriesInfo name='Internet-Draft' value='draft-vpolak-mkonstan-bmwg-mlrsearch-03' />
<format type='TXT'
        target='http://www.ietf.org/internet-drafts/draft-vpolak-mkonstan-bmwg-mlrsearch-03.txt' />
</reference>

      </references>
    </references>
    <section numbered="false" toc="default">
      <name>Acknowledgments</name>
      <t>Thanks to <contact fullname="Trevor Cooper"/>, <contact fullname="Sridhar Rao"/>, and <contact fullname="Martin Klozik"/> of the VSPERF
      project for many contributions to the early testing <xref target="VSPERF-b2b" format="default"/>. <contact fullname="Yoshiaki Itou"/> has also investigated the topic
      and made useful suggestions. <contact fullname="Maciek Konstantyowicz"/> and <contact fullname="Vratko Polak"/> also
      provided many comments and suggestions based on extensive integration
      testing and resulting search-algorithm proposals -- the most up-to-date
      feedback possible. <contact fullname="Tim Carlin"/> also provided comments and support for the
      document. <contact fullname="Warren Kumari"/>'s review improved readability in several key
      passages. <contact fullname="David Black"/>, <contact fullname="Martin Duke"/>, and <contact fullname="Scott Bradner"/>'s comments
      improved the clarity and configuration advice on trial duration. <contact fullname="Malisa Vucinic"/> suggested additional text on DUT design cautions in the Security
      Considerations section.</t>
    </section>

  </back>
</rfc>