Status of This Memo

This is an Internet Standards Track document.¶

This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 7841.¶

Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at https://www.rfc-editor.org/info/rfc8845.¶

Copyright Notice

Copyright (c) 2021 IETF Trust and the persons identified as the document authors. All rights reserved.¶

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.¶

Table of Contents

1. Introduction

Current telepresence systems, though based on open standards such as RTP [RFC3550] and SIP [RFC3261], cannot easily interoperate with each other. A major factor limiting the interoperability of telepresence systems is the lack of a standardized way to describe and negotiate the use of multiple audio and video streams comprising the media flows. This document provides a framework for protocols to enable interoperability by handling multiple streams in a standardized way. The framework is intended to support the use cases described in "Use Cases for Telepresence Multistreams" [RFC7205] and to meet the requirements in "Requirements for Telepresence Multistreams" [RFC7262]. This includes cases using multiple media streams that are not necessarily telepresence.¶

The basic session setup for the use cases is based on SIP [RFC3261] and SDP offer/answer [RFC3264]. In addition to basic SIP & SDP offer/answer, signaling that is ControLling mUltiple streams for tElepresence (CLUE) specific is required to exchange the information describing the multiple Media Streams. The motivation for this framework, an overview of the signaling, and the information required to be exchanged are described in subsequent sections of this document. Companion documents describe the signaling details [RFC8848], the data model [RFC8846], and the protocol [RFC8847].¶

2. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶

3. Definitions

The terms defined below are used throughout this document and in companion documents. Capitalization is used in order to easily identify a defined term.¶

Advertisement:

A CLUE message a Media Provider sends to a Media Consumer describing specific aspects of the content of the Media and any restrictions it has in terms of being able to provide certain Streams simultaneously.¶

Audio Capture (AC):

Media Capture for audio. Denoted as "ACn" in the examples in this document.¶

Capture:

Same as Media Capture.¶

Capture Device:

A device that converts physical input, such as audio, video, or text, into an electrical signal, in most cases to be fed into a Media encoder.¶

Capture Encoding:

A specific Encoding of a Media Capture, to be sent by a Media Provider to a Media Consumer via RTP.¶

Capture Scene:

A structure representing a spatial region captured by one or more Capture Devices, each capturing Media representing a portion of the region. The spatial region represented by a Capture Scene may correspond to a real region in physical space, such as a room. A Capture Scene includes attributes and one or more Capture Scene Views, with each view including one or more Media Captures.¶

Capture Scene View (CSV):

A list of Media Captures of the same Media type that together form one way to represent the entire Capture Scene.¶

CLUE:

CLUE is an acronym for "ControLling mUltiple streams for tElepresence", which is the name of the IETF working group in which this document and certain companion documents have been developed. Often, CLUE-* refers to something that has been designed by the CLUE working group; for example, this document may be called the CLUE-framework document herein and elsewhere.¶

CLUE-capable device:

A device that supports the CLUE data channel [RFC8850], the CLUE protocol [RFC8847] and the principles of CLUE negotiation; it also seeks CLUE-enabled calls.¶

CLUE-enabled call:

A call in which two CLUE-capable devices have successfully negotiated support for a CLUE data channel in SDP [RFC4566]. A CLUE-enabled call is not necessarily immediately able to send CLUE-controlled Media; negotiation of the data channel and of the CLUE protocol must complete first. Calls between two CLUE-capable devices that have not yet successfully completed negotiation of support for the CLUE data channel in SDP are not considered CLUE-enabled.¶

Conference:

Used as defined in "A Framework for Conferencing within the Session Initiation Protocol (SIP)" [RFC4353].¶

Configure Message:

A CLUE message a Media Consumer sends to a Media Provider specifying which content and Media Streams it wants to receive, based on the information in a corresponding Advertisement message.¶

Consumer:

Short for Media Consumer.¶

Encoding:

Short for Individual Encoding.¶

Encoding Group:

A set of Encoding parameters representing a total Media Encoding capability to be subdivided across potentially multiple Individual Encodings.¶

Endpoint:

A CLUE-capable device that is the logical point of final termination through receiving, decoding and Rendering, and/or initiation through capturing, encoding, and sending of Media Streams. An Endpoint consists of one or more physical devices that source and sink Media Streams, and exactly one [RFC4353] Participant (which, in turn, includes exactly one SIP User Agent). Endpoints can be anything from multiscreen/multicamera rooms to handheld devices.¶

Global View:

A set of references to one or more CSVs of the same Media type that are defined within Scenes of the same Advertisement. A Global View is a suggestion from the Provider to the Consumer for one set of CSVs that provide a useful representation of all the Scenes in the Advertisement.¶

Global View List:

A list of Global Views included in an Advertisement. A Global View List may include Global Views of different Media types.¶

Individual Encoding:

a set of parameters representing a way to encode a Media Capture to become a Capture Encoding.¶

Multipoint Control Unit (MCU):

a CLUE-capable device that connects two or more Endpoints into one single multimedia Conference [RFC7667]. An MCU includes a Mixer like that described in [RFC4353], without the requirement of [RFC4353] to send Media to each participant.¶

Media:

Any data that, after suitable encoding, can be conveyed over RTP, including audio, video, or timed text.¶

Media Capture (MC):

A source of Media, such as from one or more Capture Devices or constructed from other Media Streams.¶

Media Consumer:

A CLUE-capable device that intends to receive Capture Encodings.¶

Media Provider:

A CLUE-capable device that intends to send Capture Encodings.¶

Multiple Content Capture (MCC):

A Capture that mixes and/or switches other Captures of a single type (for example, all audio or all video). Particular Media Captures may or may not be present in the resultant Capture Encoding, depending on time or space. Denoted as "MCCn" in the example cases in this document.¶

Plane of Interest:

The spatial plane within a Scene containing the most-relevant subject matter.¶

Provider:

Same as a Media Provider.¶

Render:

The process of generating a representation from Media, such as displayed motion video or sound emitted from loudspeakers.¶

Scene:

Same as a Capture Scene.¶

Simultaneous Transmission Set:

A set of Media Captures that can be transmitted simultaneously from a Media Provider.¶

Single Media Capture:

A Capture that contains Media from a single source Capture Device, e.g., an Audio Capture from a single microphone or a Video Capture from a single camera.¶

Spatial Relation:

The arrangement of two objects in space, in contrast to relation in time or other relationships.¶

Stream:

A Capture Encoding sent from a Media Provider to a Media Consumer via RTP [RFC3550].¶

Stream Characteristics:

The Media Stream attributes commonly used in non-CLUE SIP/SDP environments (such as Media codec, bitrate, resolution, profile/level, etc.) as well as CLUE-specific attributes, such as the Capture ID or a spatial location.¶

Video Capture (VC):

Media Capture for video. Denoted as VCn in the example cases in this document.¶

Video Composite:

A single image that is formed, normally by an RTP mixer inside an MCU, by combining visual elements from separate sources.¶

4. Overview and Motivation

This section provides an overview of the functional elements defined in this document to represent a telepresence or multistream system. The motivations for the framework described in this document are also provided.¶

Two key concepts introduced in this document are the terms "Media Provider" and "Media Consumer". A Media Provider represents the entity that sends the Media and a Media Consumer represents the entity that receives the Media. A Media Provider provides Media in the form of RTP packets; a Media Consumer consumes those RTP packets. Media Providers and Media Consumers can reside in Endpoints or in Multipoint Control Units (MCUs). A Media Provider in an Endpoint is usually associated with the generation of Media for Media Captures; these Media Captures are typically sourced from cameras, microphones, and the like. Similarly, the Media Consumer in an Endpoint is usually associated with renderers, such as screens and loudspeakers. In MCUs, Media Providers and Consumers can have the form of outputs and inputs, respectively, of RTP mixers, RTP translators, and similar devices. Typically, telepresence devices, such as Endpoints and MCUs, would perform as both Media Providers and Media Consumers, the former being concerned with those devices' transmitted Media and the latter with those devices' received Media. In a few circumstances, a CLUE-capable device includes only Consumer or Provider functionality, such as recorder-type Consumers or webcam-type Providers.¶

The motivations for the framework outlined in this document include the following:¶

(1)

Endpoints in telepresence systems typically have multiple Media Capture and Media Render devices, e.g., multiple cameras and screens. While previous system designs were able to set up calls that would capture Media using all cameras and display Media on all screens, for example, there was no mechanism that could associate these Media Captures with each other in space and time, in a cross-vendor interoperable way.¶

(2)

The mere fact that there are multiple Media Capture and Media Render devices, each of which may be configurable in aspects such as zoom, leads to the difficulty that a variable number of such devices can be used to capture different aspects of a region. The Capture Scene concept allows for the description of multiple setups for those multiple Media Capture devices that could represent sensible operation points of the physical Capture Devices in a room, chosen by the operator. A Consumer can pick and choose from those configurations based on its rendering abilities and then inform the Provider about its choices. Details are provided in Section 7.¶

(3)

In some cases, physical limitations or other reasons disallow the concurrent use of a device in more than one setup. For example, the center camera in a typical three-camera conference room can set its zoom objective to capture either the middle few seats only or all seats of a room, but not both concurrently. The Simultaneous Transmission Set concept allows a Provider to signal such limitations. Simultaneous Transmission Sets are part of the Capture Scene description and are discussed in Section 8.¶

(4)

Often, the devices in a room do not have the computational complexity or connectivity to deal with multiple Encoding options simultaneously, even if each of these options is sensible in certain scenarios, and even if the simultaneous transmission is also sensible (i.e., in case of multicast Media distribution to multiple Endpoints). Such constraints can be expressed by the Provider using the Encoding Group concept, which is described in Section 9.¶

(5)

Due to the potentially large number of RTP Streams required for a Multimedia Conference involving potentially many Endpoints, each of which can have many Media Captures and Media renderers, it has become common to multiplex multiple RTP Streams onto the same transport address, so as to avoid using the port number as a multiplexing point and the associated shortcomings such as NAT/firewall traversal. The large number of possible permutations of sensible options a Media Provider can make available to a Media Consumer makes a mechanism desirable that allows it to narrow down the number of possible options that a SIP offer/answer exchange has to consider. Such information is made available using protocol mechanisms specified in this document and companion documents. The Media Provider and Media Consumer may use information in CLUE messages to reduce the complexity of SIP offer/answer messages. Also, there are aspects of the control of both Endpoints and MCUs that dynamically change during the progress of a call, such as audio-level-based screen switching, layout changes, and so on, which need to be conveyed. Note that these control aspects are complementary to those specified in traditional SIP-based conference management, such as Binary Floor Control Protocol (BFCP). An exemplary call flow can be found in Section 5.¶

Finally, all this information needs to be conveyed, and the notion of support for it needs to be established. This is done by the negotiation of a "CLUE channel", a data channel negotiated early during the initiation of a call. An Endpoint or MCU that rejects the establishment of this data channel, by definition, does not support CLUE-based mechanisms, whereas an Endpoint or MCU that accepts it is indicating support for CLUE as specified in this document and its companion documents.¶

5. Description of the Framework/Model

The CLUE framework specifies how multiple Media Streams are to be handled in a telepresence Conference.¶

A Media Provider (transmitting Endpoint or MCU) describes specific aspects of the content of the Media and the Media Stream Encodings it can send in an Advertisement; and the Media Consumer responds to the Media Provider by specifying which content and Media Streams it wants to receive in a Configure message. The Provider then transmits the asked-for content in the specified Streams.¶

This Advertisement and Configure typically occur during call initiation, after CLUE has been enabled in a call, but they MAY also happen at any time throughout the call, whenever there is a change in what the Consumer wants to receive or (perhaps less common) what the Provider can send.¶

An Endpoint or MCU typically acts as both Provider and Consumer at the same time, sending Advertisements and sending Configurations in response to receiving Advertisements. (It is possible to be just one or the other.)¶

The data model [RFC8846] is based around two main concepts: a Capture and an Encoding. A Media Capture, such as of type audio or video, has attributes to describe the content a Provider can send. Media Captures are described in terms of CLUE-defined attributes, such as Spatial Relationships and purpose of the Capture. Providers tell Consumers which Media Captures they can provide, described in terms of the Media Capture attributes.¶

A Provider organizes its Media Captures into one or more Capture Scenes, each representing a spatial region, such as a room. A Consumer chooses which Media Captures it wants to receive from the Capture Scenes.¶

In addition, the Provider can send the Consumer a description of the Individual Encodings it can send in terms of identifiers that relate to items in SDP [RFC4566].¶

The Provider can also specify constraints on its ability to provide Media, and a sensible design choice for a Consumer is to take these into account when choosing the content and Capture Encodings it requests in the later offer/answer exchange. Some constraints are due to the physical limitations of device; for example, a camera may not be able to provide zoom and non-zoom views simultaneously. Other constraints are system based, such as maximum bandwidth.¶

The following diagram illustrates the information contained in an Advertisement.¶

...................................................................
.  Provider Advertisement             +--------------------+      .
.                                     | Simultaneous Sets  |      .
.        +------------------------+   +--------------------+      .
.        |       Capture Scene N  |   +--------------------+      .
.      +-+----------------------+ |   | Global View List   |      .
.      |       Capture Scene 2  | |   +--------------------+      .
.    +-+----------------------+ | |      +----------------------+ .
.    |  Capture Scene 1       | | |      |  Encoding Group N    | .
.    |    +---------------+   | | |    +-+--------------------+ | .
.    |    | Attributes    |   | | |    |   Encoding Group 2   | | .
.    |    +---------------+   | | |  +-+--------------------+ | | .
.    |                        | | |  |   Encoding Group 1   | | | .
.    |    +----------------+  | | |  |     parameters       | | | .
.    |    |  V i e w s     |  | | |  |      bandwidth       | | | .
.    |    |  +---------+   |  | | |  | +-------------------+| | | .
.    |    |  |Attribute|   |  | | |  | | V i d e o         || | | .
.    |    |  +---------+   |  | | |  | | E n c o d i n g s || | | .
.    |    |                |  | | |  | | Encoding 1        || | | .
.    |    | View 1         |  | | |  | |                   || | | .
.    |    |  (list of MCs) |  | |-+  | +-------------------+| | | .
.    |    +----|-|--|------+  |-+    |                      | | | .
.    +---------|-|--|---------+      | +-------------------+| | | .
.              | |  |                | | A u d i o         || | | .
.              | |  |                | | E n c o d i n g s || | | .
.              v |  |                | | Encoding 1        || | | .
.      +---------|--|--------+       | |                   || | | .
.      | Media Capture N     |------>| +-------------------+| | | .
.    +-+---------v--|------+ |       |                      | | | .
.    | Media Capture 2     | |       |                      | |-+ .
.  +-+--------------v----+ |-------->|                      | |   .
.  | Media Capture  1    | | |       |                      |-+   .
.  |  +----------------+ |---------->|                      |     .
.  |  | Attributes     | | |_+       +----------------------+     .
.  |  +----------------+ |_+                                      .
.  +---------------------+                                        .
.                                                                 .
...................................................................

Figure 2 illustrates the call flow used by a simple system (two Endpoints) in compliance with this document. A very brief outline of the call flow is described in the text that follows.¶

      +-----------+                     +-----------+
      | Endpoint1 |                     | Endpoint2 |
      +----+------+                     +-----+-----+
           | INVITE (BASIC SDP+CLUECHANNEL)   |
           |--------------------------------->|
           |    200 0K (BASIC SDP+CLUECHANNEL)|
           |<---------------------------------|
           | ACK                              |
           |--------------------------------->|
           |                                  |
           |<################################>|
           |       BASIC MEDIA SESSION        |
           |<################################>|
           |                                  |
           |    CONNECT (CLUE CTRL CHANNEL)   |
           |=================================>|
           |            ...                   |
           |<================================>|
           |   CLUE CTRL CHANNEL ESTABLISHED  |
           |<================================>|
           |                                  |
           | ADVERTISEMENT 1                  |
           |*********************************>|
           |                  ADVERTISEMENT 2 |
           |<*********************************|
           |                                  |
           |                      CONFIGURE 1 |
           |<*********************************|
           | CONFIGURE 2                      |
           |*********************************>|
           |                                  |
           | REINVITE (UPDATED SDP)           |
           |--------------------------------->|
           |              200 0K (UPDATED SDP)|
           |<---------------------------------|
           | ACK                              |
           |--------------------------------->|
           |                                  |
           |<################################>|
           |     UPDATED MEDIA SESSION        |
           |<################################>|
           |                                  |
           v                                  v

An initial offer/answer exchange establishes a basic Media session, for example, audio-only, and a CLUE channel between two Endpoints. With the establishment of that channel, the Endpoints have consented to use the CLUE protocol mechanisms and, therefore, MUST adhere to the CLUE protocol suite as outlined herein.¶

Over this CLUE channel, the Provider in each Endpoint conveys its characteristics and capabilities by sending an Advertisement as specified herein. The Advertisement is typically not sufficient to set up all Media. The Consumer in the Endpoint receives the information provided by the Provider and can use it for several purposes. It uses it, along with information from an offer/answer exchange, to construct a CLUE Configure message to tell the Provider what the Consumer wishes to receive. Also, the Consumer may use the information provided to tailor the SDP it is going to send during any following SIP offer/answer exchange, and its reaction to SDP it receives in that step. It is often a sensible implementation choice to do so. Spatial relationships associated with the Media can be included in the Advertisement, and it is often sensible for the Media Consumer to take those spatial relationships into account when tailoring the SDP. The Consumer can also limit the number of Encodings it must set up resources to receive, and not waste resources on unwanted Encodings, because it has the Provider's Advertisement information ahead of time to determine what it really wants to receive. The Consumer can also use the Advertisement information for local rendering decisions.¶

This initial CLUE exchange is followed by an SDP offer/answer exchange that not only establishes those aspects of the Media that have not been "negotiated" over CLUE, but also has the effect of setting up the Media transmission itself, involving potentially security exchanges, Interactive Connectivity Establishment (ICE), and whatnot. This step is considered "plain vanilla SIP".¶

During the lifetime of a call, further exchanges MAY occur over the CLUE channel. In some cases, those further exchanges lead to a modified system behavior of Provider or Consumer (or both) without any other protocol activity such as further offer/answer exchanges. For example, a Configure Message requesting that the Provider place a different Capture source into a Capture Encoding, signaled over the CLUE channel, ought not to lead to heavy-handed mechanisms like SIP re-invites. In other cases, however, after the CLUE negotiation, an additional offer/answer exchange becomes necessary. For example, if both sides decide to upgrade the call from one screen to a multi-screen call, and more bandwidth is required for the additional video channels compared to what was previously negotiated using offer/answer, a new offer/answer exchange is required.¶

One aspect of the protocol outlined herein, and specified in more detail in companion documents, is that it makes available to the Consumer information regarding the Provider's capabilities to deliver Media and attributes related to that Media such as their Spatial Relationship. The operation of the renderer inside the Consumer is unspecified in that it can choose to ignore some information provided by the Provider and/or not Render Media Streams available from the Provider (although the Consumer follows the CLUE protocol and, therefore, gracefully receives and responds to the Provider's information using a Configure operation).¶

A CLUE-capable device interoperates with a device that does not support CLUE. The CLUE-capable device can determine, by the result of the initial offer/answer exchange, if the other device supports and wishes to use CLUE. The specific mechanism for this is described in [RFC8848]. If the other device does not use CLUE, then the CLUE-capable device falls back to behavior that does not require CLUE.¶

As for the Media, Provider and Consumer have an end-to-end communication relationship with respect to (RTP-transported) Media; and the mechanisms described herein and in companion documents do not change the aspects of setting up those RTP flows and sessions. In other words, the RTP Media sessions conform to the negotiated SDP whether or not CLUE is used.¶

6. Spatial Relationships

In order for a Consumer to perform a proper rendering, it is often necessary (or at least helpful) for the Consumer to have received spatial information about the Streams it is receiving. CLUE defines a coordinate system that allows Media Providers to describe the Spatial Relationships of their Media Captures to enable proper scaling and spatially sensible rendering of their Streams. The coordinate system is based on a few principles:¶

• Each Capture Scene has a distinct coordinate system, unrelated to the coordinate systems of other Scenes.¶
• Simple systems that do not have multiple Media Captures to associate spatially need not use the coordinate model, although it can still be useful to provide an Area of Capture.¶

• Coordinates can either be in real, physical units (millimeters), have an unknown scale, or have no physical scale. Systems that know their physical dimensions (for example, professionally installed Telepresence room systems) MUST provide those real-world measurements to enable the best user experience for advanced receiving systems that can utilize this information. Systems that don't know specific physical dimensions but still know relative distances MUST use "Unknown Scale". "No Scale" is intended to be used only where Media Captures from different devices (with potentially different scales) will be forwarded alongside one another (e.g., in the case of an MCU).¶

  • "Millimeters" means the scale is in millimeters.¶
  • "Unknown Scale" means the scale is not necessarily in millimeters, but the scale is the same for every Capture in the Capture Scene.¶
  • "No Scale" means the scale could be different for each Capture -- an MCU Provider that advertises two adjacent Captures and picks sources (which can change quickly) from different Endpoints might use this value; the scale could be different and changing for each Capture. But the areas of capture still represent a Spatial Relation between Captures.¶
• The coordinate system is right-handed Cartesian X, Y, Z with the origin at a spatial location of the Provider's choosing. The Provider MUST use the same coordinate system with the same scale and origin for all coordinates within the same Capture Scene.¶

The direction of increasing coordinate values is as follows: X increases from left to right, from the point of view of an observer at the front of the room looking toward the back; Y increases from the front of the room to the back of the room; Z increases from low to high (i.e., floor to ceiling).¶

Cameras in a Scene typically point in the direction of increasing Y, from front to back. But there could be multiple cameras pointing in different directions. If the physical space does not have a well-defined front and back, the Provider chooses any direction for X, Y, and Z consistent with right-handed coordinates.¶

7. Media Captures and Capture Scenes

This section describes how Providers can describe the content of Media to Consumers.¶

   ........................................................
   . Advertisement                                        .
   .                                                      .
   . +--------------+         +-------------------------+ .
   . |Scene 1       |         |Global View List         | .
   . |              |         |                         | .
   . | CSV1 (v)<----------------- Global View (CSV 1)   | .
   . |         <-------.      |                         | .
   . |              |  *--------- Global View (CSV 1,5) | .
   . | CSV2 (v)     |  |      |                         | .
   . |              |  |      |                         | .
   . | CSV3 (v)<---------*------- Global View (CSV 3,5) | .
   . |              |  | |    |                         | .
   . | CSV4 (a)<----------------- Global View (CSV 4)   | .
   . |         <-----------.  |                         | .
   . +--------------+  | | *----- Global View (CSV 4,6) | .
   .                   | | |  |                         | .
   . +--------------+  | | |  +-------------------------+ .
   . |Scene 2       |  | | |                              .
   . |              |  | | |                              .
   . | CSV5 (v)<-------' | |                              .
   . |         <---------' |                              .
   . |              |      |        (v) = video           .
   . | CSV6 (a)<-----------'        (a) = audio           .
   . |              |                                     .
   . +--------------+                                     .
   `......................................................'

8. Simultaneous Transmission Set Constraints

In many practical cases, a Provider has constraints or limitations on its ability to send Captures simultaneously. One type of limitation is caused by the physical limitations of capture mechanisms; these constraints are represented by a Simultaneous Transmission Set. The second type of limitation reflects the encoding resources available, such as bandwidth or video encoding throughput (macroblocks/second). This type of constraint is captured by Individual Encodings and Encoding Groups, discussed below.¶

Some Endpoints or MCUs can send multiple Captures simultaneously; however, sometimes there are constraints that limit which Captures can be sent simultaneously with other Captures. A device may not be able to be used in different ways at the same time. Provider Advertisements are made so that the Consumer can choose one of several possible mutually exclusive usages of the device. This type of constraint is expressed in a Simultaneous Transmission Set, which lists all the Captures of a particular Media type (e.g., audio, video, or text) that can be sent at the same time. There are different Simultaneous Transmission Sets for each Media type in the Advertisement. This is easier to show in an example.¶

Consider the example of a room system where there are three cameras, each of which can send a separate Capture covering two people each: VC0, VC1, and VC2. The middle camera can also zoom out (using an optical zoom lens) and show all six people, VC3. But the middle camera cannot be used in both modes at the same time; it has to either show the space where two participants sit or the whole six seats, but not both at the same time. As a result, VC1 and VC3 cannot be sent simultaneously.¶

Simultaneous Transmission Sets are expressed as sets of the Media Captures that the Provider could transmit at the same time (though, in some cases, it is not intuitive to do so). If a Multiple Content Capture is included in a Simultaneous Transmission Set, it indicates that the Capture Encoding associated with it could be transmitted as the same time as the other Captures within the Simultaneous Transmission Set. It does not imply that the Single Media Captures contained in the Multiple Content Capture could all be transmitted at the same time.¶

In this example, the two Simultaneous Transmission Sets are shown in Table 5. If a Provider advertises one or more mutually exclusive Simultaneous Transmission Sets, then, for each Media type, the Consumer MUST ensure that it chooses Media Captures that lie wholly within one of those Simultaneous Transmission Sets.¶

A Provider OPTIONALLY can include the Simultaneous Transmission Sets in its Advertisement. These constraints apply across all the Capture Scenes in the Advertisement. It is a syntax-conformance requirement that the Simultaneous Transmission Sets MUST allow all the Media Captures in any particular CSV to be used simultaneously. Similarly, the Simultaneous Transmission Sets MUST reflect the simultaneity expressed by any Global View.¶

For shorthand convenience, a Provider MAY describe a Simultaneous Transmission Set in terms of CSVs and Capture Scenes. If a CSV is included in a Simultaneous Transmission Set, then all Media Captures in the CSV are included in the Simultaneous Transmission Set. If a Capture Scene is included in a Simultaneous Transmission Set, then all its CSVs (of the corresponding Media type) are included in the Simultaneous Transmission Set. The end result reduces to a set of Media Captures, of a particular Media type, in either case.¶

If an Advertisement does not include Simultaneous Transmission Sets, then the Provider MUST be able to simultaneously provide all the Captures from any one CSV of each Media type from each Capture Scene. Likewise, if there are no Simultaneous Transmission Sets and there is a Global View list, then the Provider MUST be able to simultaneously provide all the Captures from any particular Global View (of each Media type) from the Global View list.¶

If an Advertisement includes multiple CSVs in a Capture Scene, then the Consumer MAY choose one CSV for each Media type, or it MAY choose individual Captures based on the Simultaneous Transmission Sets.¶

9. Encodings

Individual Encodings and Encoding Groups are CLUE's mechanisms allowing a Provider to signal its limitations for sending Captures, or combinations of Captures, to a Consumer. Consumers can map the Captures they want to receive onto the Encodings, with the Encoding parameters they want. As for the relationship between the CLUE-specified mechanisms based on Encodings and the SIP offer/answer exchange, please refer to Section 5.¶

,-------------------------------------------------.
|             Media Provider                      |
|                                                 |
|  ,--------------------------------------.       |
|  | ,--------------------------------------.     |
|  | | ,--------------------------------------.   |
|  | | |          Encoding Group              |   |
|  | | | ,-----------.                        |   |
|  | | | |           | ,---------.            |   |
|  | | | |           | |         | ,---------.|   |
|  | | | | Encoding1 | |Encoding2| |Encoding3||   |
|  `.| | |           | |         | `---------'|   |
|    `.| `-----------' `---------'            |   |
|      `--------------------------------------'   |
`-------------------------------------------------'

10. Consumer's Choice of Streams to Receive from the Provider

After receiving the Provider's Advertisement message (which includes Media Captures and associated constraints), the Consumer composes its reply to the Provider in the form of a Configure message. The Consumer is free to use the information in the Advertisement as it chooses, but there are a few obviously sensible design choices, which are outlined below.¶

If multiple Providers connect to the same Consumer (i.e., in an MCU-less multiparty call), it is the responsibility of the Consumer to compose Configures for each Provider that both fulfill each Provider's constraints as expressed in the Advertisement, as well as its own capabilities.¶

In an MCU-based multiparty call, the MCU can logically terminate the Advertisement/Configure negotiation in that it can hide the characteristics of the receiving Endpoint and rely on its own capabilities (transcoding/transrating/etc.) to create Media Streams that can be decoded at the Endpoint Consumers. The timing of an MCU's sending of Advertisements (for its outgoing ports) and Configures (for its incoming ports, in response to Advertisements received there) is up to the MCU and is implementation dependent.¶

As a general outline, a Consumer can choose, based on the Advertisement it has received, which Captures it wishes to receive, and which Individual Encodings it wants the Provider to use to encode the Captures.¶

On receipt of an Advertisement with an MCC, the Consumer treats the MCC as per other non-MCC Captures with the following differences:¶

• The Consumer would understand that the MCC is a Capture that includes the referenced individual Captures (or any Captures, if none are referenced) and that these individual Captures are delivered as part of the MCC's Capture Encoding.¶
• The Consumer may utilize any of the attributes associated with the referenced individual Captures and any Capture Scene attributes from where the individual Captures were defined to choose Captures and for Rendering decisions.¶
• If the MCC attribute Allow Subset Choice is true, then the Consumer may or may not choose to receive all the indicated Captures. It can choose to receive a subset of Captures indicated by the MCC.¶

For example, if the Consumer receives:¶

• MCC1(VC1,VC2,VC3){attributes}¶

A Consumer could choose all the Captures within an MCC; however, if the Consumer determines that it doesn't want VC3, it can return MCC1(VC1,VC2). If it wants all the individual Captures, then it returns only the MCC identity (i.e., MCC1). If the MCC in the Advertisement does not reference any individual Captures, or the Allow Subset Choice attribute is false, then the Consumer cannot choose what is included in the MCC: it is up to the Provider to decide.¶

A Configure Message includes a list of Capture Encodings. These are the Capture Encodings the Consumer wishes to receive from the Provider. Each Capture Encoding refers to one Media Capture and one Individual Encoding.¶

For each Capture the Consumer wants to receive, it configures one of the Encodings in that Capture's Encoding Group. The Consumer does this by telling the Provider, in its Configure Message, which Encoding to use for each chosen Capture. Upon receipt of this Configure from the Consumer, common knowledge is established between Provider and Consumer regarding sensible choices for the Media Streams. The setup of the actual Media channels, at least in the simplest case, is left to a following offer/answer exchange. Optimized implementations may speed up the reaction to the offer/answer exchange by reserving the resources at the time of finalization of the CLUE handshake.¶

CLUE Advertisements and Configure Messages don't necessarily require a new SDP offer/answer for every CLUE message exchange. But the resulting Encodings sent via RTP must conform to the most-recent SDP offer/answer result.¶

In order to meaningfully create and send an initial Configure, the Consumer needs to have received at least one Advertisement, and an SDP offer defining the Individual Encodings, from the Provider.¶

In addition, the Consumer can send a Configure at any time during the call. The Configure MUST be valid according to the most recently received Advertisement. The Consumer can send a Configure either in response to a new Advertisement from the Provider or on its own, for example, because of a local change in conditions (people leaving the room, connectivity changes, multipoint related considerations).¶

When choosing which Media Streams to receive from the Provider, and the encoding characteristics of those Media Streams, the Consumer advantageously takes several things into account: its local preference, simultaneity restrictions, and encoding limits.¶

11. Extensibility

One important characteristics of the Framework is its extensibility. The standard for interoperability and handling multiple Streams must be future-proof. The framework itself is inherently extensible through expanding the data model types. For example:¶

• Adding more types of Media, such as telemetry, can done by defining additional types of Captures in addition to audio and video.¶
• Adding new functionalities, such as 3-D Video Captures, may require additional attributes describing the Captures.¶

The infrastructure is designed to be extended rather than requiring new infrastructure elements. Extension comes through adding to defined types.¶

12. Examples - Using the Framework (Informative)

This section gives some examples, first from the point of view of the Provider, then the Consumer, then some multipoint scenarios.¶

   ,-. d
  (   )`--.__        +---+
   `-' /     `--.__  |   |
 ,-.  |            `-.._ |_-+Camera 2 (VC2)
(   ).'     <--(AC1)-+-''`+-+
 `-' |_...---''      |   |
 ,-.c+-..__          +---+
(   )|     ``--..__  |   |
 `-' |             ``+-..|_-+Camera 1 (VC1)
 ,-. |      <--(AC2)..--'|+-+                          ^
(   )|     __..--'   |   |                             |
 `-'b|..--'          +---+                             |X
 ,-. |``---..___     |   |                             |
(   )\          ```--..._|_-+Camera 0 (VC0)            |
 `-'  \     <--(AC0) ..-''`-+                          |
  ,-. \      __.--'' |   |                  <----------+
 (   ) |..-''        +---+                     Y
  `-' a                          (0,0,0) origin is under Camera 1

    bottom left    bottom right  top left         top right
VC0 (-2011,2850,0) (-673,3000,0) (-2011,2850,757) (-673,3000,757)
VC1 ( -673,3000,0) ( 673,3000,0) ( -673,3000,757) ( 673,3000,757)
VC2 (  673,3000,0) (2011,2850,0) (  673,3000,757) (2011,3000,757)
MCC3(-2011,2850,0) (2011,2850,0) (-2011,2850,757) (2011,3000,757)
MCC4(-2011,2850,0) (2011,2850,0) (-2011,2850,757) (2011,3000,757)
VC5 (-2011,2850,0) (2011,2850,0) (-2011,2850,757) (2011,3000,757)
VC6 none

VC0 (-1678,0,800)
VC1 (0,0,800)
VC2 (1678,0,800)
MCC3 none
MCC4 none
VC5 (0,0,800)
VC6 none

Point of Capture:           Point on Line of Capture:
AC0 (-1342,2000,800)       (-1342,2925,379)
AC1 ( 1342,2000,800)       ( 1342,2925,379)
AC2 (    0,2000,800)       (    0,3000,379)
AC3 (    0,2000,800)       (    0,3000,379)
AC4 none

encodeGroupID=EG0, maxGroupBandwidth=6000000
    encodeID=ENC0, maxWidth=1920, maxHeight=1088, maxFrameRate=60,
                   maxPps=124416000, maxBandwidth=4000000
    encodeID=ENC1, maxWidth=1280, maxHeight=720, maxFrameRate=30,
                   maxPps=27648000, maxBandwidth=4000000
    encodeID=ENC2, maxWidth=960, maxHeight=544, maxFrameRate=30,
                   maxPps=15552000, maxBandwidth=4000000
encodeGroupID=EG1  maxGroupBandwidth=6000000
    encodeID=ENC3, maxWidth=1920, maxHeight=1088, maxFrameRate=60,
                   maxPps=124416000, maxBandwidth=4000000
    encodeID=ENC4, maxWidth=1280, maxHeight=720, maxFrameRate=30,
                   maxPps=27648000, maxBandwidth=4000000
    encodeID=ENC5, maxWidth=960, maxHeight=544, maxFrameRate=30,
                   maxPps=15552000, maxBandwidth=4000000
encodeGroupID=EG2  maxGroupBandwidth=6000000
    encodeID=ENC6, maxWidth=1920, maxHeight=1088, maxFrameRate=60,
                   maxPps=124416000, maxBandwidth=4000000
    encodeID=ENC7, maxWidth=1280, maxHeight=720, maxFrameRate=30,
                   maxPps=27648000, maxBandwidth=4000000
    encodeID=ENC8, maxWidth=960, maxHeight=544, maxFrameRate=30,
                   maxPps=15552000, maxBandwidth=4000000

encodeGroupID=EG3, maxGroupBandwidth=320000
    encodeID=ENC9, maxBandwidth=64000
    encodeID=ENC10, maxBandwidth=64000
    encodeID=ENC11, maxBandwidth=64000
    encodeID=ENC12, maxBandwidth=64000
    encodeID=ENC13, maxBandwidth=64000

encodeGroupID=EG0 maxGroupBandwidth=6000000
    encodeID=VIDENC0, maxWidth=1920, maxHeight=1088,
      maxFrameRate=60, maxPps=62208000, maxBandwidth=4000000
    encodeID=VIDENC1, maxWidth=1920, maxHeight=1088,
      maxFrameRate=60, maxPps=62208000, maxBandwidth=4000000
    encodeID=AUDENC0, maxBandwidth=96000
    encodeID=AUDENC1, maxBandwidth=96000
    encodeID=AUDENC2, maxBandwidth=96000

encodeGroupID=EG0 maxGroupBandwidth=6000000
    encodeID=VIDENC0, maxWidth=1920, maxHeight=1088,
      maxFrameRate=60, maxPps=62208000, maxBandwidth=4000000
    encodeID=VIDENC1, maxWidth=1920, maxHeight=1088,
      maxFrameRate=60, maxPps=62208000, maxBandwidth=4000000
encodeGroupID=EG1 maxGroupBandwidth=500000
    encodeID=AUDENC0, maxBandwidth=96000
    encodeID=AUDENC1, maxBandwidth=96000
    encodeID=AUDENC2, maxBandwidth=96000

   +---+---+---+ +-------------+ +-------------+ +-------------+
   |   |   |   | |             | |             | |             |
   +---+---+---+ |             | |             | |             |
   |   |   |   | |             | |             | |             |
   +---+---+---+ |             | |             | |             |
   |   |   |   | |             | |             | |             |
   +---+---+---+ +-------------+ +-------------+ +-------------+

           +-------------+ +-------------+ +-------------+
           |             | |             | |             |
           |             | |             | |             |
           |             | |             | |             |
           | +-+ +-+ +-+ | | +-+ +-+ +-+ | | +-+ +-+ +-+ |
           | +-+ +-+ +-+ | | +-+ +-+ +-+ | | +-+ +-+ +-+ |
           +-------------+ +-------------+ +-------------+

13. IANA Considerations

This document has no IANA actions.¶

14. Security Considerations

There are several potential attacks related to telepresence, specifically the protocols used by CLUE. This is the case due to conferencing sessions, the natural involvement of multiple Endpoints, and the many, often user-invoked, capabilities provided by the systems.¶

An MCU involved in a CLUE session can experience many of the same attacks as a conferencing system such as the one enabled by the Conference Information Data Model for Centralized Conferencing (XCON) framework [RFC5239]. Examples of attacks include the following: an Endpoint attempting to listen to sessions in which it is not authorized to participate, an Endpoint attempting to disconnect or mute other users, and theft of service by an Endpoint in attempting to create telepresence sessions it is not allowed to create. Thus, it is RECOMMENDED that an MCU implementing the protocols necessary to support CLUE follow the security recommendations specified in the conference control protocol documents. In the case of CLUE, SIP is the conferencing protocol, thus the security considerations in [RFC4579] MUST be followed. Other security issues related to MCUs are discussed in the XCON framework [RFC5239]. The use of xCard with potentially sensitive information provides another reason to implement recommendations in Section 11 of [RFC5239].¶

One primary security concern, surrounding the CLUE framework introduced in this document, involves securing the actual protocols and the associated authorization mechanisms. These concerns apply to Endpoint-to-Endpoint sessions as well as sessions involving multiple Endpoints and MCUs. Figure 2 in Section 5 provides a basic flow of information exchange for CLUE and the protocols involved.¶

As described in Section 5, CLUE uses SIP/SDP to establish the session prior to exchanging any CLUE-specific information. Thus, the security mechanisms recommended for SIP [RFC3261], including user authentication and authorization, MUST be supported. In addition, the Media MUST be secured. Datagram Transport Layer Security (DTLS) / Secure Real-time Transport Protocol (SRTP) MUST be supported and SHOULD be used unless the Media, which is based on RTP, is secured by other means (see [RFC7201] [RFC7202]). Media security is also discussed in [RFC8848] and [RFC8849]. Note that SIP call setup is done before any CLUE-specific information is available, so the authentication and authorization are based on the SIP mechanisms. The entity that will be authenticated may use the Endpoint identity or the Endpoint user identity; this is an application issue and not a CLUE-specific issue.¶

A separate data channel is established to transport the CLUE protocol messages. The contents of the CLUE protocol messages are based on information introduced in this document. The CLUE data model [RFC8846] defines, through an XML schema, the syntax to be used. One type of information that could possibly introduce privacy concerns is the xCard information, as described in Section 7.1.1.10. The decision about which xCard information to send in the CLUE channel is an application policy for point-to-point and multipoint calls based on the authenticated identity that can be the Endpoint identity or the user of the Endpoint. For example, the telepresence multipoint application can authenticate a user before starting a CLUE exchange with the telepresence system and have a policy per user.¶

In addition, the (text) description field in the Media Capture attribute (Section 7.1.1.6) could possibly reveal sensitive information or specific identities. The same would be true for the descriptions in the Capture Scene (Section 7.3.1) and CSV (Section 7.3.2) attributes. An implementation SHOULD give users control over what sensitive information is sent in an Advertisement. One other important consideration for the information in the xCard as well as the description field in the Media Capture and CSV attributes is that while the Endpoints involved in the session have been authenticated, there are no assurance that the information in the xCard or description fields is authentic. Thus, this information MUST NOT be used to make any authorization decisions.¶

While other information in the CLUE protocol messages does not reveal specific identities, it can reveal characteristics and capabilities of the Endpoints. That information could possibly uniquely identify specific Endpoints. It might also be possible for an attacker to manipulate the information and disrupt the CLUE sessions. It would also be possible to mount a DoS attack on the CLUE Endpoints if a malicious agent has access to the data channel. Thus, it MUST be possible for the Endpoints to establish a channel that is secure against both message recovery and message modification. Further details on this are provided in the CLUE data channel solution document [RFC8850].¶

There are also security issues associated with the authorization to perform actions at the CLUE Endpoints to invoke specific capabilities (e.g., rearranging screens, sharing content, etc.). However, the policies and security associated with these actions are outside the scope of this document and the overall CLUE solution.¶

Acknowledgements

Allyn Romanow and Brian Baldino were authors of early draft versions. Mark Gorzynski also contributed much to the initial approach. Many others also contributed, including Christian Groves, Jonathan Lennox, Paul Kyzivat, Rob Hanton, Roni Even, Christer Holmberg, Stephen Botzko, Mary Barnes, John Leslie, and Paul Coverdale.¶

Authors' Addresses