<?xml version="1.0" encoding="US-ASCII"?> version='1.0' encoding='utf-8'?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
<?rfc toc="yes"?>
<?rfc tocompact="yes"?>
<?rfc tocdepth="3"?>
<?rfc tocindent="yes"?>
<?rfc symrefs="yes"?>
<?rfc sortrefs="yes"?>
<?rfc comments="yes"?>
<?rfc inline="yes"?>
<?rfc compact="yes"?>
<?rfc subcompact="no"?> "rfc2629-xhtml.ent">

<rfc xmlns:xi="http://www.w3.org/2001/XInclude" category="std"
     docName="draft-ietf-avtcore-multi-media-rtp-session-13" number="8860"
     ipr="trust200902" updates="3550, 3551"> 3551" obsoletes="" submissionType="IETF"
     consensus="true" xml:lang="en" tocInclude="true" tocDepth="3" symRefs="true" sortRefs="true" version="3">
  <!-- xml2rfc v2v3 conversion 2.35.0 -->
  <front>
    <title abbrev="Multiple Media Types in an RTP Session">Sending Multiple
    Types of Media in a Single RTP Session</title>
    <seriesInfo name="RFC" value="8860"/>
    <author fullname="Magnus Westerlund" initials="M." surname="Westerlund">
      <organization>Ericsson</organization>
      <address>
        <postal>
          <street>Farogatan 6</street>

          <city>SE-164 80 Kista</city>
          <street>Torshamnsgatan 23</street>
          <city>Stockholm</city>
	  <code>164 80</code>
          <country>Sweden</country>
        </postal>

        <phone>+46 10 714 82 87</phone>
        <email>magnus.westerlund@ericsson.com</email>
      </address>
    </author>
    <author fullname="Colin Perkins" initials="C. " surname="Perkins">
      <organization>University of Glasgow</organization>
      <address>
        <postal>
          <street>School of Computing Science</street>
          <city>Glasgow</city>
          <code>G12 8QQ</code>
          <country>United Kingdom</country>
        </postal>
        <email>csp@csperkins.org</email>
      </address>
    </author>

    <author fullname="Jonathan Lennox" initials="J." surname="Lennox">
      <organization abbrev="Vidyo">Vidyo, Inc.</organization> abbrev="8x8 / Jitsi">8x8, Inc. / Jitsi</organization>
      <address>
        <postal>
          <street>433 Hackensack Avenue</street>

          <street>Seventh Floor</street>

          <city>Hackensack</city>
          <city>Jersey City</city>
          <region>NJ</region>

          <code>07601</code>

          <country>US</country>
          <code>07302</code>
          <country>United States of America</country>
        </postal>

        <email>jonathan@vidyo.com</email>
        <email>jonathan.lennox@8x8.com</email>
      </address>
    </author>
    <date day="18" month="December" year="2015"/>

    <workgroup>AVTCORE WG</workgroup> month="January" year="2021"/>

<keyword>Real-time</keyword>
<keyword>Multiplexing</keyword>
<keyword>Bundle</keyword>

    <abstract>
      <t>This document specifies how an RTP session can contain RTP Streams streams
      with media from multiple media types such as audio, video, and text.
      This has been restricted by the RTP Specification, specifications (RFCs 3550 and 3551), and thus this
      document updates RFC RFCs 3550 and RFC 3551 to enable this behaviour for
      applications that satisfy the applicability for using multiple media
      types in a single RTP session.</t>
    </abstract>
  </front>
  <middle>
    <section title="Introduction"> numbered="true" toc="default">
      <name>Introduction</name>
      <t>The Real-time Transport Protocol <xref target="RFC3550"/> target="RFC3550" format="default"/> was
      designed to use separate RTP sessions to transport different types of
      media. This implies that different transport layer transport-layer flows are used for
      different RTP streams. For example, a video conferencing application
      might send audio and video traffic RTP flows on separate UDP ports. With
      increased use of network address/port translation, firewalls, and other
      middleboxes
      middleboxes, it is, however, becoming difficult to establish multiple
      transport layer
      transport-layer flows between endpoints. Hence, there is pressure to
      reduce the number of concurrent transport flows used by RTP
      applications.</t>
      <t>This memo updates <xref target="RFC3550"/> target="RFC3550" format="default"/> and <xref
      target="RFC3551"/> target="RFC3551" format="default"/> to allow multiple media types to be sent in a single
      RTP session in certain cases, thereby reducing the number of transport
      layer transport-layer flows that are needed. It makes no changes to RTP behaviour when
      using multiple RTP streams containing media of the same type (e.g.,
      multiple audio streams or multiple video streams) in a single RTP
      session. However However, <xref target="I-D.ietf-avtcore-rtp-multi-stream"/> target="RFC8108" format="default"/>
      provides important clarifications to RTP behaviour in that case.</t>
      <t>This memo is structured as follows. <xref target="sec:defn"/> target="sec_defn" format="default"/> defines
      terminology. <xref target="sec:background"/> target="sec_background" format="default"/> further describes the
      background to, and motivation for, this memo and memo; <xref
      target="sec:applicability"/> target="sec_applicability" format="default"/> describes the scenarios where this memo is
      applicable. <xref target="sec:base"/> target="sec_base" format="default"/> discusses issues arising from the
      base RTP and RTCP specification RTP Control Protocol (RTCP) specifications <xref
      target="RFC3550"/> <xref target="RFC3551"/> when using multiple types of media in a
      single RTP session, while <xref target="sec:extn"/> target="sec_extn" format="default"/> considers the impact
      of RTP extensions. We discuss signalling in <xref target="sec:sig"/>. target="sec_sig" format="default"/>.
      Finally, security considerations are discussed in <xref
      target="Security"/>.</t> target="Security" format="default"/>.</t>
    </section>
    <section anchor="sec:defn" title="Terminology"> anchor="sec_defn" numbered="true" toc="default">
      <name>Terminology</name>
      <t>The terms Encoded Stream, Endpoint, Media Source, RTP Session, "encoded stream", "endpoint", "media source", "RTP session", and
      RTP Stream
      "RTP stream" are used as defined in <xref target="RFC7656"/>. target="RFC7656" format="default"/>. We also
      define the following terms:</t>

      <t><list style="hanging">
          <t hangText="Media Type:">The
      <dl newline="false" spacing="normal">
        <dt>Media Type:</dt>
        <dd>The general type of media data used by a
          real-time application. The media type corresponds to the value used
          in the &lt;media&gt; field of an SDP m= a Session Description Protocol (SDP)
	  "m=" line. The media types
          defined at the time of this writing are "audio", "video", "text",
          "image", "application", and "message". "message" <xref target="RFC4566"/> target="RFC4566" format="default"/>
          <xref target="RFC6466"/></t>

          <t hangText="Quality target="RFC6466" format="default"/>.</dd>
        <dt>Quality of Service (QoS):">Network (QoS):</dt>
        <dd>Network mechanisms that are
          intended to ensure that the packets within a flow or with a specific
          marking are transported with certain properties.</t>
        </list></t>

      <t>The properties.</dd>
      </dl>
        <t>
    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
      "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", "<bcp14>MUST</bcp14>", "<bcp14>MUST NOT</bcp14>",
    "<bcp14>REQUIRED</bcp14>", "<bcp14>SHALL</bcp14>", "<bcp14>SHALL NOT</bcp14>",
    "<bcp14>SHOULD</bcp14>", "<bcp14>SHOULD NOT</bcp14>",
    "<bcp14>RECOMMENDED</bcp14>", "<bcp14>NOT RECOMMENDED</bcp14>",
    "<bcp14>MAY</bcp14>", and
      "OPTIONAL" "<bcp14>OPTIONAL</bcp14>" in this document are to be
    interpreted as described in BCP 14 <xref target="RFC2119"/> <xref
      target="RFC2119"/>.</t>
    target="RFC8174"/> when, and only when, they appear in all capitals, as
    shown here.
        </t>
    </section>
    <section anchor="sec:background" title="Background anchor="sec_background" numbered="true" toc="default">
      <name>Background and Motivation"> Motivation</name>
      <t>RTP was designed to support multimedia sessions, containing multiple
      types of media sent simultaneously, by using multiple transport layer transport-layer
      flows. The existence of network address translators, firewalls, and
      other middleboxes complicates this, however, since a mechanism is needed
      to ensure that all the transport layer transport-layer flows needed by the application
      can be established. This has three consequences: <list style="numbers">
          <t>increased </t>
      <ol spacing="normal" type="1">
        <li>increased delay to establish a complete session, since each of
          the transport layer transport-layer flows needs to be negotiated and
          established;</t>

          <t>increased
          established;</li>
        <li>increased state and resource consumption in the middleboxes that
          can lead to unexpected behaviour when middlebox resource limits are
          reached; and</t>

          <t>increased and</li>
        <li>increased risk that a subset of the transport layer transport-layer flows will
          fail to be established, thus preventing the application from
          communicating.</t>
        </list></t>
          communicating.</li>
      </ol>
      <t>Using fewer transport layer transport-layer flows can hence be seen to reduce the
      risk of communication failure, failure and can lead to improved reliability and
      performance.</t>
      <t>One of the benefits of using multiple transport layer transport-layer flows is that
      it makes it easy to use network layer quality of service (QoS) network-layer QoS
      mechanisms to give differentiated performance for different flows.
      However, we note that many RTP-using application applications that use RTP don't use network QoS
      features,
      features and don't expect or desire any separation in network treatment
      of their media packets, independent of whether they are audio, video video, or
      text. When an application has no such desire, it doesn't need to provide
      a transport flow structure that simplifies flow based flow-based QoS.</t>
      <t>Given the above issues, it might seem appropriate for RTP-based
      applications to send all their RTP streams bundled into one RTP session,
      running over a single transport layer transport-layer flow. However, this is prohibited
      by the RTP specification, specifications <xref target="RFC3550"/> <xref target="RFC3551"/>, because the design of RTP makes certain
      assumptions that can be incompatible with sending multiple media types
      in a single RTP session. Specifically, the RTP control protocol (RTCP) RTCP
      timing rules assume that all RTP media flows in a single RTP session
      have broadly similar RTCP reporting and feedback requirements, which can
      be problematic when different types of media are multiplexed together.
      Various RTP extensions also make assumptions about SSRC Synchronisation
      Source (SSRC) use and RTCP
      reporting that are incompatible with sending different media types in a
      single RTP session.</t>
      <t>This memo updates <xref target="RFC3550"/> target="RFC3550" format="default"/> and <xref
      target="RFC3551"/> target="RFC3551" format="default"/> to allow RTP sessions to contain more than one media
      type in certain circumstances, circumstances and gives guidance on when it is safe to
      send multiple media types in a single RTP session.</t>
    </section>
    <section anchor="sec:applicability" title="Applicability"> anchor="sec_applicability" numbered="true" toc="default">
      <name>Applicability</name>
      <t>This specification has limited applicability, and anyone intending to
      use it needs to ensure that their application and use case meets meet the
      following criteria: <list style="hanging">
          <t hangText="Equal </t>
      <dl newline="false" spacing="normal">
        <dt>Equal treatment of media:">The media:</dt>
        <dd>The use of a single RTP
          session normally results in similar network treatment for all types
          of media used within the session. Applications that require
          significantly different network quality of service (QoS) QoS or RTCP
          configuration for different RTP streams are better suited by to sending
          those RTP streams in separate RTP session, sessions, using separate transport
          layer
	  transport-layer flows for each, since that gives method provides greater flexibility. Further
          guidance on how to provide differential treatment for some media streams is
          given in <xref target="I-D.ietf-avtcore-multiplex-guidelines"/> target="RFC8872" format="default"/> and
          <xref target="RFC7657"/>.</t>

          <t hangText="Compatible target="RFC7657" format="default"/>.</dd>
        <dt>Compatible RTCP Behaviour:">The behaviour:</dt>
        <dd>The RTCP timing rules
          enforce a single RTCP reporting interval for all participants in an
          RTP session. Flows with very different media sending rate rates or RTCP
          feedback requirements cannot be multiplexed together, since this
          leads to either excessive or insufficient RTCP for some flows,
          depending on how the RTCP session bandwidth, and hence the reporting
          interval, is are configured. For example, it is likely infeasible to
          find a single RTCP configuration that simultaneously suits both a
          low-rate audio flow with no feedback, feedback and a high-quality video flow
          with sophisticated RTCP-based feedback. Thus, combining these into a
          single RTP session is difficult and/or inadvisable.</t>

          <t hangText="Signalled Support:">The inadvisable.</dd>
        <dt>Signalled support:</dt>
        <dd>The extensions defined in this memo
          are not compatible with unmodified endpoints that are compatible
	  with <xref
          target="RFC3550"/>-compatible endpoints. target="RFC3550" format="default"/>. Their use requires
          signalling and mutual agreement by all participants within an RTP
          session. This requirement can be a problem for signalling solutions
          that can't negotiate with all participants. For declarative
          signalling solutions, mandating that the session is using use multiple
          media types in one RTP session can be a way of attempting to ensure
          that all participants in the RTP session follow the requirement.
          However, for signalling solutions that lack methods for enforcing
          a requirement that a receiver supports support a specific feature, this can still cause
          issues.</t>

          <t hangText="Consistent
          issues.</dd>
        <dt>Consistent support for multiparty RTP sessions:">If sessions:</dt>
        <dd><t>If it
          is desired to send multiple types of media in a multiparty RTP
          session, then all participants in that session need to support
          sending multiple type types of media in a single RTP session. It is not
          possible, in the general case, to implement a gateway that can
          interconnect an endpoint using that uses multiple types of media sent using
          separate RTP sessions, sessions with one or more endpoints that send multiple
          types of media in a single RTP session.</t>

        <t>One reason for this is that the same SSRC value can safely be
          used for different streams in multiple RTP sessions, but when
          collapsed to a single RTP session there is an SSRC collision. This
          would not be an issue, since SSRC collision detection will resolve
          the conflict, except that some RTP payload formats and extensions
          use matching SSRCs to identify related flows, flows and will break when a
          single RTP session is used.</t>

        <t>A middlebox that remaps SSRC values when combining multiple RTP
          sessions into one also needs to be aware of all possible RTCP packet
          types that might be used, so that it can remap the SSRC values in
          those packets. This is impossible to do without restricting the set
          of RTCP packet types that can be used to those that are known by the
          middlebox. Such a middlebox might also have difficulty due to
          differences in configured RTCP bandwidth and other parameters
          between the RTP sessions.</t>

        <t>Finally, the use of a middlebox that translates SSRC values can
          negatively impact the possibility for of loop detection, as SSRC/CSRC
          (Contributing Source) can't be used to detect the loops; instead instead, some other RTP stream or
          media source identity name space namespace that is common across all
          interconnect
          interconnected parts is needed.</t>

          <t hangText="Ability
	</dd>
        <dt>Ability to operate with limited payload type space:">An space:</dt>
        <dd>An
          RTP session has only a single 7-bit payload type space for all its
          payload type numbers. Some applications might find this space to be
          limiting (i.e., overly restrictive) when using different media types and RTP payload formats
          within a single RTP session.</t>

          <t hangText="Avoids session.</dd>
        <dt>Avoidance of incompatible Extensions:">Some extensions:</dt>
        <dd>Some RTP and RTCP
          extensions rely on the existence of multiple RTP sessions and relate
          RTP streams between sessions. Others report on particular media
          types,
          types and cannot be used with other media types. Applications that
          send multiple types of media into a single RTP session need to avoid
          such extensions.</t>
        </list></t> extensions.</dd>
      </dl>
    </section>
    <section anchor="sec:base"
             title="Using anchor="sec_base" numbered="true" toc="default">
      <name>Using Multiple Media Types in a Single RTP Session"> Session</name>
      <t>This section defines what needs to be done or avoided to make an RTP
      session with multiple media types function without issues.</t>
      <section title="Allowing numbered="true" toc="default">
        <name>Allowing Multiple Media Types in an RTP Session">
        <t>Section 5.2 of <xref target="RFC3550">"RTP: Session</name>
        <t><xref target="RFC3550" sectionFormat="of" section="5.2">
"RTP: A Transport Protocol for Real-Time Applications"</xref> states:<list style="empty"> states:</t>
        <blockquote>
          <t>For example, in a teleconference composed of audio and video
          media encoded separately, each medium SHOULD <bcp14>SHOULD</bcp14> be carried in a
            separate RTP session with its own destination transport
            address.</t>
<t>Separate audio and video streams SHOULD NOT <bcp14>SHOULD NOT</bcp14> be carried in a
            single RTP session and demultiplexed based on the payload type or
            SSRC fields.</t>
          </list></t>
        </blockquote>

        <t>This specification changes both of these sentences. The first
        sentence is changed to:<list style="empty">
            <t>For to:</t>
        <blockquote>
          For example, in a teleconference composed of audio and video
            media encoded separately, each medium SHOULD <bcp14>SHOULD</bcp14> be carried in a
            separate RTP session with its own destination transport address,
            unless specification [RFCXXXX] is the guidelines specified in [RFC8860] are followed and the application
            meets the applicability constraints.</t>
          </list></t> constraints.
	</blockquote>
        <t>The second sentence is changed to:<list style="empty">
            <t>Separate to:</t>
        <blockquote>
          Separate audio and video media sources SHOULD NOT <bcp14>SHOULD NOT</bcp14> be carried in
            a single RTP session, unless the guidelines specified in [RFCXXXX] [RFC8860]
            are followed.</t>
          </list></t>

        <t>Second followed.
	</blockquote>
        <t>The second paragraph of Section 6 in <xref target="RFC3551">RTP target="RFC3551" sectionFormat="of" section="6">"RTP Profile for Audio and Video Conferences with Minimal Control</xref> Control"</xref> says:</t>

        <t><list style="empty">
            <t>The
        <blockquote>
          The payload types currently defined in this profile are
            assigned to exactly one of three categories or media types: audio
            only, video only and those combining audio and video. The media
            types are marked in Tables 4 and 5 as "A", "V" and "AV",
            respectively. Payload types of different media types SHALL NOT <bcp14>SHALL NOT</bcp14> be
            interleaved or multiplexed within a single RTP session, but
            multiple RTP sessions MAY <bcp14>MAY</bcp14> be used in parallel to send multiple
            media types. An RTP source MAY <bcp14>MAY</bcp14> change payload types within the
            same media type during a session. See the section "Multiplexing
            RTP Sessions" of RFC 3550 for additional explanation.</t>
          </list>This explanation.
	</blockquote>
        <t>This specification's purpose is to override that existing
        SHALL NOT the above-listed
        "<bcp14>SHALL&nbsp;NOT</bcp14>" under certain conditions. Thus Thus, this sentence also has to be
        changed to allow for multiple media type's types' payload types in the same
        session. The sentence containing "SHALL NOT" "<bcp14>SHALL NOT</bcp14>" in the above paragraph is
        changed to:<list style="empty">
            <t>Payload to:</t>
        <blockquote>
          Payload types of different media types SHALL NOT <bcp14>SHALL NOT</bcp14> be interleaved
            or multiplexed within a single RTP session unless [RFCXXXX] [RFC8860] is
            used,
            used and the application conforms to the applicability
            constraints. Multiple RTP sessions MAY <bcp14>MAY</bcp14> be used in parallel to send
            multiple media types.</t>
          </list></t>

        <t>RFC-Editor Note: Please replace RFCXXXX with the RFC number of this
        specification when assigned.</t> types.
	</blockquote>
      </section>
      <section anchor="sec:demuxing"
               title="Demultiplexing media types anchor="sec_demuxing" numbered="true" toc="default">
        <name>Demultiplexing Media Types within an RTP session"> Session</name>
        <t>When receiving packets from a transport layer transport-layer flow, an endpoint
        will first separate the RTP and RTCP packets from the non-RTP packets, packets
        and pass them to the RTP/RTCP protocol handler. The RTP and RTCP
        packets are then demultiplexed based on their SSRC into the different
        RTP streams. streams based on their SSRC. For each RTP stream, incoming RTCP packets are processed,
        and the RTP payload type is used to select the appropriate media
        decoder. This process remains the same irrespective of whether
        multiple media types are sent in a single RTP session or not.</t>
        <t>As explained below, it is important to note that the RTP payload
        type is never used to distinguish RTP streams. The RTP packets are
        demultiplexed into RTP streams based on their SSRC, then SSRC; the RTP
        payload type is then used to select the correct media decoding media-decoding pathway for
        each RTP stream.</t>
      </section>
      <section anchor="sec-source-restrcitctions"
               title="Per-SSRC numbered="true" toc="default">
        <name>Per-SSRC Media Type Restrictions"> Restrictions</name>
        <t>An SSRC in an RTP session can change between media formats of the
        same type, subject to certain restrictions <xref target="RFC7160"/>, target="RFC7160" format="default"/>,
        but MUST NOT <bcp14>MUST NOT</bcp14> change its media type during its lifetime. For example, an
        SSRC can change between different audio formats, but it cannot start
        sending audio and then change to sending video. The lifetime of an SSRC
        ends when an RTCP BYE packet for that SSRC is sent, sent or when it ceases
        transmission for long enough that it times out for the other
        participants in the session.</t>
        <t>The main motivation is that a given SSRC has its own RTP timestamp
        and sequence number spaces. The same way that you can't send two
        encoded streams of audio with the same SSRC, you can't send one
        encoded audio and one encoded video stream with the same SSRC. Each
        encoded stream stream, when made into an RTP stream stream, needs to have the sole
        control over the sequence number and timestamp space. If not, one
        would not be able to detect packet loss for that particular encoded
        stream. Nor can
        stream, nor could one easily determine which clock rate a particular
        SSRCs
        SSRC's timestamp will increase with. For additional arguments regarding
	why RTP
        payload type based multiplexing of multiple media sources that is based on RTP payload type doesn't
        work, see <xref target="I-D.ietf-avtcore-multiplex-guidelines"/>.</t> target="RFC8872" format="default"/>.</t>
        <t>Within an RTP session where multiple media types have been
        configured for use, an SSRC can only send one type of media during its
        lifetime (i.e., it can switch between different audio codecs, since
        those are both the same type of media, but it cannot switch between audio
        and video). Different SSRCs MUST <bcp14>MUST</bcp14> be used for the different media
        sources, the same way multiple media sources of the same media type
        already have to do. The payload type will inform a receiver which
        media type the SSRC is being used for. Thus Thus, the payload type MUST <bcp14>MUST</bcp14> be
        unique across all of the payload configurations configurations, independent of the media
        type that is used in the RTP session.</t>
      </section>

      <!-- This adds nothing compared to the later sections (csp)
      <section title="Payload Type Applicability">
        <t>Most Payload Types have a native media type, like an audio codec is
        natural belonging to the audio media type. However, there exist a
        number of RTP payload anchor="sec.rtcp" numbered="true" toc="default">
        <name>RTCP Considerations</name>
        <t>When sending multiple types of media that don't have a native media type. For
        example, transport robustness mechanisms like <xref
        target="RFC4588">RTP Retransmission</xref> and <xref
        target="RFC5109">Generic FEC</xref> inherit their media type from what
        they protect. RTP Retransmission is explicitly bound to the payload
        type it is protecting, and thus will inherit it. However Generic FEC
        is a excellent example of an RTP payload type that has no natural
        media type. The media type for what it protects is not relevant as it
        is the recovered RTP packets that have a particular media type, and
        thus Generic FEC is best categorized as an application media type.</t>

        <t>The above discussion is relevant to what limitations exist for RTP
        payload type usage within an RTP session that has multiple media
        types. In fact <xref target="sec-generic-fec">this document</xref>
        suggest that for usage of Generic FEC (XOR-based) as defined in RFC
        5109 can actually use a single media type when used with independent
        RTP sessions for source and repair data. <list style="hanging">
            <t>Note a particular SSRC carrying Generic FEC will clearly only
            protect a specific SSRC and thus that instance is bound to the
            SSRC's media type. For this specific case, it is possible to have
            one be applicable to both. However, in cases when the signalling
            is setup to enable fall back to using separate RTP sessions, then
            using a different media type, e.g. application, than the media
            being protected can create issues.</t>
          </list></t>
      </section>
      -->

      <section anchor="sec.rtcp" title="RTCP Considerations">
        <t>When sending multiple types of media that have different rates in different rates in a
        single RTP session, endpoints MUST <bcp14>MUST</bcp14> follow the guidelines for handling
        RTCP described as provided in Section 7 of <xref
        target="I-D.ietf-avtcore-rtp-multi-stream"/>.</t> target="RFC8108" sectionFormat="of" section="7"/>.</t>
      </section>
    </section>
    <section anchor="sec:extn" title="Extension Considerations"> anchor="sec_extn" numbered="true" toc="default">
      <name>Extension Considerations</name>
      <t>This section outlines known issues and incompatibilities with RTP and
      RTCP extensions when multiple media types are used in a single RTP
      sessions.
      session. Future extensions to RTP and RTCP need to consider, and
      document, any potential incompatibility.</t> incompatibilities.</t>
      <section anchor="sec-rtx" title="RTP numbered="true" toc="default">
        <name>RTP Retransmission Payload Format"> Format</name>
        <t>The RTP Retransmission Payload Format retransmission payload format <xref target="RFC4588"/> target="RFC4588" format="default"/> can
        operate in either SSRC-multiplexed mode or session-multiplex session-multiplexed mode.</t>
        <t>In SSRC-multiplexed mode, retransmitted RTP packets are sent in the
        same RTP session as the original packets, packets but use a different SSRC
        with the same RTCP SDES Source Description (SDES) CNAME. If each endpoint sends only a single
        original RTP stream and a single retransmission RTP stream in the
        session, this is sufficient. If an endpoint sends multiple original
        and retransmission RTP streams, as would occur when sending multiple
        media types in a single RTP session, then each original RTP stream and
        the retransmission RTP stream have to be associated using heuristics.
        By having retransmission requests outstanding for only one SSRC not
        yet mapped, a receiver can determine the binding between the original and
        retransmission RTP stream. streams. Another alternative is the use of different
        RTP payload types, allowing the signalled "apt" (associated payload
        type) parameter <xref target="RFC4588"/> of the RTP retransmission payload format to be used to
        associate retransmitted and original packets.</t>
        <t>Session-multiplexed mode sends the retransmission RTP stream in a
        separate RTP session to the original RTP stream, but using the same
        SSRC for each, with the association being done by matching SSRCs between
        the two sessions. This is unaffected by the use of multiple media
        types in a single RTP session, since each media type will be sent
        using a different SSRC in the original RTP session, and the same SSRCs
        can be used in the retransmission session, allowing the streams to be
        associated. This can be signalled using SDP with the BUNDLE grouping
	extension <xref
        target="I-D.ietf-mmusic-sdp-bundle-negotiation"/>	target="RFC8843" format="default"/> and FID the Flow Identification (FID)
	grouping extension <xref target="RFC5888"/> extensions. target="RFC5888" format="default"/>. These SDP extensions require each
        "m=" line to only be included in a single FID group, but the RTP
        retransmission payload format uses FID groups to indicate the m= "m=" lines
        that form an original and retransmission pair. Accordingly, when using
        the BUNDLE extension to allow multiple media types to be sent in a
        single RTP session, each original media source (m= ("m=" line) that is
        retransmitted needs a corresponding m= "m=" line in the retransmission RTP
        session. In case If there are multiple media lines for retransmission,
        these media lines will form an independent BUNDLE group from the
        BUNDLE group with the source streams.</t>
        <t>An example SDP fragment showing the grouping structures is provided
        in <xref target="fig-rtx-session"/>. target="fig-rtx-session" format="default"/>. This example is not legal SDP SDP, and
        only the most important attributes have been left in place. Note that
        this SDP is not an initial BUNDLE offer. As can be seen in this example, there are two
        bundle groups, groups -- one for the source RTP session and one for the
        retransmissions. Then Then, each of the media sources are is grouped with its
        retransmission flow using FID, resulting in three more groupings.</t>
        <figure anchor="fig-rtx-session"
                title="SDP example of Session Multiplexed RTP Retransmission">
          <artwork><![CDATA[ anchor="fig-rtx-session">
          <name>SDP Example of Session-Multiplexed RTP Retransmission</name>
          <sourcecode name="sdp-example" type="sdp"><![CDATA[       a=group:BUNDLE foo bar fiz
       a=group:BUNDLE zoo kelp glo
       a=group:FID foo zoo
       a=group:FID bar kelp
       a=group:FID fiz glo
       m=audio 10000 RTP/AVP 0
       a=mid:foo
       a=rtpmap:0 PCMU/8000
       m=video 10000 RTP/AVP 31
       a=mid:bar
       a=rtpmap:31 H261/90000
       m=video 10000 RTP/AVP 31
       a=mid:fiz
       a=rtpmap:31 H261/90000
       m=audio 40000 RTP/AVPF 99
       a=rtpmap:99 rtx/90000
       a=fmtp:99 apt=0;rtx-time=3000
       a=mid:zoo
       m=video 40000 RTP/AVPF 100
       a=rtpmap:100 rtx/90000
       a=fmtp:199 apt=31;rtx-time=3000
       a=mid:kelp
       m=video 40000 RTP/AVPF 100
       a=rtpmap:100 rtx/90000
       a=fmtp:199 apt=31;rtx-time=3000
       a=mid:glo
]]></artwork>
       a=mid:glo]]></sourcecode>
        </figure>
      </section>
      <section anchor="sec-generic-fec"
               title="RTP numbered="true" toc="default">
        <name>RTP Payload Format for Generic FEC"> FEC</name>
        <t>The RTP Payload Format payload format for Generic generic Forward Error Correction (FEC) (FEC),
        as defined in <xref target="RFC5109"/> target="RFC5109" format="default"/> (and its predecessor predecessor, <xref
        target="RFC2733"/>) target="RFC2733" format="default"/>), can either send the FEC stream as a separate RTP
        stream,
        stream or it can send the FEC combined with the original RTP stream
        as a redundant encoding <xref target="RFC2198"/>.</t> target="RFC2198" format="default"/>.</t>
        <t>When sending FEC as a separate stream, the RTP Payload Format payload format for
        generic FEC requires that FEC stream to be sent in a separate RTP
        session to the original stream, using the same SSRC, with the FEC
        stream being associated by matching the SSRC between sessions. The RTP
        session used for the original streams can include multiple RTP
        streams, and those RTP streams can use multiple media types. The
        repair session only needs one RTP Payload payload type to indicate FEC data,
        irrespective of the number of FEC streams sent, since the SSRC is used
        to associate the FEC streams with the original streams. Hence, it is
        RECOMMENDED
        <bcp14>RECOMMENDED</bcp14> that the FEC stream use the "application/ulpfec" media
        type in the case of support for <xref target="RFC5109"/>, target="RFC5109" format="default"/> and the "application/parityfec" "application/&wj;parityfec"
        media type in the case of support for <xref target="RFC2733"/>. target="RFC2733" format="default"/>. It is legal, but NOT
        RECOMMENDED, <bcp14>NOT
        RECOMMENDED</bcp14>, to send FEC streams using media specific media-specific payload format
        names (e.g., using both the "audio/ulpfec" and "video/ulpfec" payload
        formats for a single RTP session containing both audio and video
        flows), since this unnecessarily (1)&nbsp;unnecessarily uses up RTP payload type values, values and
        adds
        (2)&nbsp;adds no value for demultiplexing since because there might be multiple streams
        of the same media type).</t>
        <t>The combination of an original RTP session using multiple media
        types with an associated generic FEC session can be signalled using
        SDP with the BUNDLE extension <xref
        target="I-D.ietf-mmusic-sdp-bundle-negotiation"/>. target="RFC8843" format="default"/>. In this case, the
        RTP session carrying the FEC streams will be its own BUNDLE group. The
        m=
        "m=" line for each original stream and the m= "m=" line for the corresponding
        FEC stream are grouped using the SDP grouping framework Grouping Framework, using either
        the <xref target="RFC5956">FEC-FR</xref> grouping target="RFC5956" format="default">FEC-FR grouping</xref> or, for backwards
        compatibility, the FEC grouping <xref target="RFC4756"/> grouping. target="RFC4756" format="default"/>. This is
        similar to the situation that arises for RTP retransmission with
        session
        session-based multiplexing as discussed in <xref target="sec-rtx"/>.</t> target="sec-rtx" format="default"/>.</t>
        <t>The <xref target="RFC5576">Source-Specific Media Attributes</xref>
        specification target="RFC5576" format="default">source-specific media
	attributes specification</xref>
        defines an SDP extension (the "FEC" semantic of the
        "ssrc-group" attribute) to signal FEC relationships between multiple
        RTP streams within a single RTP session. This cannot be used with
        generic FEC, since the FEC repair packets need to have the same SSRC
        value as the source packets being protected.
There was work on existed a proposal (now abandoned) for an
        Unequal Layer Uneven Level Protection (ULP)
extension to allow it be use enable transmission of the FEC RTP
        streams within the same RTP Session session as the source stream <xref
        target="I-D.lennox-payload-ulp-ssrc-mux"/>.</t>
	target="I-D.lennox-payload-ulp-ssrc-mux" format="default"/>.</t>
        <t>When the FEC is sent as a redundant encoding, the considerations in
        <xref target="sec-red"/> target="sec-red" format="default"/> apply.</t>
      </section>
      <section anchor="sec-red" title="RTP numbered="true" toc="default">
        <name>RTP Payload Format for Redundant Audio"> Audio</name>
        <t>The RTP Payload Format payload format for Redundant Audio redundant audio <xref target="RFC2198"/> target="RFC2198" format="default"/>
        can be used to protect audio streams. It can also be used along with
        the generic FEC payload format to send original and repair data in the
        same RTP packets. Both are compatible with RTP sessions containing
        multiple media types.</t>
        <t>This payload format requires each different redundant encoding to use
        a different RTP payload type number. When used with generic FEC in
        sessions that contain multiple media types, this requires each media
        type to use a different payload type for the FEC stream. For example,
        if audio and text are sent in a single RTP session with generic ULP
        FEC sent as a redundant encoding for each, then payload types need to
        be assigned for FEC using the audio/ulpfec and text/ulpfec text/&wj;ulpfec payload
        formats. If multiple original payload types are used in the session,
        different redundant payload types need to be allocated for each one.
        This has potential to rapidly exhaust the available RTP payload type
        numbers.</t>
      </section>
    </section>
    <section anchor="sec:sig" title="Signalling"> anchor="sec_sig" numbered="true" toc="default">
      <name>Signalling</name>
      <t>Establishing a single RTP session using multiple media types requires
      signalling. This signalling has to:<list style="numbers">
          <t>ensure to:</t>
      <ol spacing="normal" type="1">
        <li>ensure that any participant in the RTP session is aware that this
          is an RTP session with multiple media types;</t>

          <t>ensure types;</li>
        <li>ensure that the payload types in use in the RTP session are using
          unique values, with no overlap between the media types;</t>

          <t>ensure types;</li>
        <li>ensure that RTP session level parameters, session-level parameters -- for example example, the RTCP RR and
          RS bandwidth modifiers, modifiers <xref target="RFC3556"/>, the RTP/AVPF
	  trr-int parameter, parameter <xref target="RFC4585"/>, transport
          protocol, RTCP extensions in use, and any security parameters, parameters -- are
          consistent across the session; and</t>

          <t>ensure and</li>
        <li>ensure that RTP and RTCP functions that can be bound to a
          particular media type are reused where possible, rather than
          configuring multiple code-points code points for the same thing.</t>
        </list></t> thing.</li>
      </ol>
      <t>When using SDP signalling, the BUNDLE extension <xref
      target="I-D.ietf-mmusic-sdp-bundle-negotiation"/> target="RFC8843" format="default"/> is used to signal RTP
      sessions containing multiple media types.</t>
    </section>
    <section anchor="Security" title="Security Considerations"> numbered="true" toc="default">
      <name>Security Considerations</name>
      <t>RTP provides a range of strong security mechanisms that can be used
      to secure sessions <xref target="RFC7201"/>, target="RFC7201" format="default"/> <xref target="RFC7202"/>. target="RFC7202" format="default"/>.
      The majority of these are independent of the type of media sent in the
      RTP session; however however, it is important to check that the security
      mechanism chosen is compatible with all types of media sent within the
      session.</t>
      <t>Sending multiple media types in a single RTP session will generally
      require that all use the same security mechanism, whereas media sent
      using different RTP sessions can be secured in different ways. When
      different media types have different security requirements, it might be
      necessary to send them using separate RTP sessions to meet those
      different requirements. This can have significant costs in terms of
      resource usage, session set-up setup time, etc.</t>
    </section>
    <section anchor="IANA" title="IANA Considerations"> numbered="true" toc="default">
      <name>IANA Considerations</name>
      <t>This memo makes document has no request of IANA.</t> IANA actions.</t>
    </section>
  </middle>
  <back>

<displayreference target="I-D.lennox-payload-ulp-ssrc-mux" to="FEC-Src-Multiplexing"/>

    <references>
      <name>References</name>
      <references>
       <name>Normative References</name>
       <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.2119.xml"/>
       <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.3550.xml"/>
       <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.3551.xml"/>
       <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.8174.xml"/>

<!-- draft-ietf-mmusic-sdp-bundle-negotiation (RFC 8843) -->
    <reference anchor="RFC8843" target="https://www.rfc-editor.org/info/rfc8843">
      <front>
        <title>Negotiating Media Multiplexing Using the Session Description Protocol (SDP)</title>
        <author initials="C" surname="Holmberg" fullname="Christer Holmberg">
          <organization/>
        </author>
        <author initials="H" surname="Alvestrand" fullname="Harald Alvestrand">
          <organization/>
        </author>
        <author initials="C" surname="Jennings" fullname="Cullen Jennings">
          <organization/>
        </author>
        <date month="January" year="2021"/>
      </front>
        <seriesInfo name="RFC" value="8843"/>
        <seriesInfo name="DOI" value="10.17487/RFC8843"/>
    </reference>

        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.8108.xml"/>

      </references>
      <references>
        <name>Informative References</name>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.3556.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.4585.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.7656.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.7657.xml"/>

<!-- draft-ietf-avtcore-multiplex-guidelines (RFC 8872) -->
<reference anchor="RFC8872" target="https://www.rfc-editor.org/info/rfc8872">
   <front>
      <title>Guidelines for Using the Multiplexing Features of RTP to Support
      Multiple Media Streams</title>
      <author initials="M" surname="Westerlund" fullname="Magnus Westerlund">
	 <organization/>
      </author>
      <author initials="B" surname="Burman" fullname="Bo Burman">
	 <organization/>
      </author>
      <author initials="C" surname="Perkins" fullname="Colin Perkins">
	 <organization/>
      </author>
      <author initials="H" surname="Alvestrand" fullname="Harald Alvestrand">
	 <organization/>
      </author>
      <author initials="R" surname="Even" fullname="Roni Even">
      </author>
      <date month="January" year="2021"/>
   </front>
   <seriesInfo name="RFC" value="8872"/>
   <seriesInfo name="DOI" value="10.17487/RFC8872"/>
</reference>

<!-- draft-lennox-payload-ulp-ssrc-mux (Expired) -->
        <xi:include href="https://xml2rfc.tools.ietf.org/public/rfc/bibxml3/reference.I-D.draft-lennox-payload-ulp-ssrc-mux-00.xml"/>

        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.2198.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.2733.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.4566.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.4588.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.4756.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.5109.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.5576.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.5888.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.5956.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.6466.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.7160.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.7201.xml"/>
        <xi:include href="https://xml2rfc.ietf.org/public/rfc/bibxml/reference.RFC.7202.xml"/>
      </references>
    </references>

    <section anchor="Acknowledgements" title="Acknowledgements"> numbered="false" toc="default">
      <name>Acknowledgements</name>
      <t>The authors would like to thank Christer Holmberg, Gunnar
      Hellstr&ouml;m, Charles Eckel, Tolga Asveren, Warren Kumari, and Meral
      Shirazipour <contact fullname="Christer
      Holmberg"/>, <contact fullname="Gunnar Hellström"/>, <contact
      fullname="Charles Eckel"/>, <contact fullname="Tolga Asveren"/>,
      <contact fullname="Warren Kumari"/>, and <contact fullname="Meral
      Shirazipour"/> for their feedback on the this document.</t>
    </section>
  </middle>

  <back>
    <references title="Normative References">
      <?rfc include="reference.RFC.2119"?>

      <?rfc include='reference.RFC.3550'?>

      <?rfc include='reference.RFC.3551'?>

      <?rfc include='reference.I-D.ietf-mmusic-sdp-bundle-negotiation'?>

      <?rfc include='reference.I-D.ietf-avtcore-rtp-multi-stream'?>
    </references>

    <references title="Informative References">
      <?rfc include='reference.I-D.ietf-avtcore-multiplex-guidelines'?>

      <?rfc include='reference.RFC.7656'?>

      <?rfc include='reference.RFC.7657'?>

      <?rfc include='reference.I-D.lennox-payload-ulp-ssrc-mux'?>

      <?rfc include='reference.RFC.2198'?>

      <?rfc include='reference.RFC.2733'?>

      <?rfc include='reference.RFC.4566'?>

      <?rfc include='reference.RFC.4588'?>

      <?rfc include='reference.RFC.4756'?>

      <?rfc include='reference.RFC.5109'?>

      <?rfc include='reference.RFC.5576'?>

      <?rfc include='reference.RFC.5888'?>

      <?rfc include='reference.RFC.5956'?>

      <?rfc include='reference.RFC.6466'?>

      <?rfc include='reference.RFC.7160'?>

      <?rfc include='reference.RFC.7201'?>

      <?rfc include='reference.RFC.7202'?>
    </references>
  </back>
</rfc>