NVO3 Working Group
Internet Engineering Task Force (IETF)                             Y. Li
INTERNET-DRAFT
Request for Comments: 8394                               D. Eastlake
Intended Status: 3rd
Category: Informational                              Huawei Technologies
ISSN: 2070-1721                                               L. Kreeger
                                                            Arrcus, Inc Inc.
                                                               T. Narten
                                                                     IBM
                                                                D. Black
                                                                Dell EMC
Expires: September 14, 2018                               March 13,
                                                                May 2018

Split Network Virtualization Edge (Split-NVE) Control Plane Control-Plane Requirements
                   draft-ietf-nvo3-hpvr2nve-cp-req-17

Abstract

   In a the Split Network Virtualization Edge (Split-NVE) architecture,
   the functions of the NVE (Network Virtualization Edge) are split across a server and an a piece of
   external network equipment which that is called an external
   NVE. "External NVE".  The
   server-resident control plane control-plane functionality resides in control
   software, which may be part of hypervisor or container
   management container-management
   software; for simplicity, this document refers to the hypervisor as
   the location "location" of this software.

   Control plane protocol(s)

   One or more control-plane protocols between a hypervisor and its
   associated
   external External NVE(s) are used by the hypervisor to distribute
   its virtual
   machine virtual-machine networking state to the external External NVE(s) for
   further handling.  This document illustrates the functionality
   required by this type of
   control plane control-plane signaling protocol and
   outlines the high level high-level requirements. Virtual machine  Virtual-machine states as well
   as state transitioning are summarized to help clarify the protocol
   requirements.

Status of this This Memo

   This Internet-Draft document is submitted to IETF in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents not an Internet Standards Track specification; it is
   published for informational purposes.

   This document is a product of the Internet Engineering Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as
   Internet-Drafts.

   Internet-Drafts are draft documents valid for a maximum
   (IETF).  It represents the consensus of six months the IETF community.  It has
   received public review and may be updated, replaced, or obsoleted has been approved for publication by other the
   Internet Engineering Steering Group (IESG).  Not all documents at
   approved by the IESG are candidates for any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list level of Internet
   Standard; see Section 2 of RFC 7841.

   Information about the current Internet-Drafts can be accessed at
   http://www.ietf.org/1id-abstracts.html

   The list status of Internet-Draft Shadow Directories can this document, any errata,
   and how to provide feedback on it may be accessed obtained at
   http://www.ietf.org/shadow.html
   https://www.rfc-editor.org/info/rfc8394.

Copyright and License Notice

   Copyright (c) 2018 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info)
   (https://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the Simplified BSD License.

Table of Contents

   1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . . .  4
     1.1   2
     1.1.  Terminology . . . . . . . . . . . . . . . . . . . . . . . .  5
     1.2   4
     1.2.  Target Scenarios  . . . . . . . . . . . . . . . . . . . . .  6   5
   2.  VM Lifecycle  . . . . . . . . . . . . . . . . . . . . . . . . .  8
     2.1   7
     2.1.  VM Creation Event . . . . . . . . . . . . . . . . . . . . .  8
     2.2   7
     2.2.  VM Live Migration Event . . . . . . . . . . . . . . . . . .  9
     2.3   8
     2.3.  VM Termination Event  . . . . . . . . . . . . . . . . . . . . 10
     2.4   9
     2.4.  VM Pause, Suspension Suspension, and Resumption Events . . . . . . . . . 10   9
   3.  Hypervisor-to-NVE Control Plane Control-Plane Protocol Functionality  . . . . 11
     3.1 VN Connect  10
     3.1.  VN_Connect and Disconnect  . . VN_Disconnect  . . . . . . . . . . . . . . . 11
     3.2  10
     3.2.  TSI Associate and Activate  . . . . . . . . . . . . . . . . . 13
     3.3  12
     3.3.  TSI Disassociate De-Associate and Deactivate . . . . . . . . . . . . . . 15  14
   4.  Hypervisor-to-NVE Control Plane Control-Plane Protocol Requirements . . . . . 16  15
   5.  VDP Applicability and Enhancement Needs . . . . . . . . . . . . 17  16
   6.  Security Considerations . . . . . . . . . . . . . . . . . . . . 19  18
   7.  IANA Considerations . . . . . . . . . . . . . . . . . . . . . . 20  19
   8. Acknowledgements  References  . . . . . . . . . . . . . . . . . . . . . . . . .  20
   8.
     8.1.  Normative References  . . . . . . . . . . . . . . . . . .  20
     8.2.  Informative References  . . . . . . . . . . . . . . . . .  20
     8.1  Normative References
   Appendix A.  VDP Illustrations (per IEEE 802.1Q) (for Information
                Only)  . . . . . . . . . . . . . . . . . . . 20
     8.2  Informative References . . . .  21
   Acknowledgements  . . . . . . . . . . . . . . 21
   Appendix A. IEEE 802.1Q VDP Illustration (For information only) . 21 . . . . . . . . .  24
   Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . . .  24

1.  Introduction

   In the Split-NVE Split Network Virtualization Edge (Split-NVE) architecture
   shown in Figure 1, the functionality of the NVE (Network Virtualization Edge) is split across an
   end device supporting virtualization and an external network device which
   that is called an external NVE. "External NVE".  The portion of the NVE
   functionality located on the end device is called the tNVE "tNVE"
   (terminal-side NVE) NVE), and the portion located on the external External NVE is
   called the nNVE "nNVE" (network-side NVE) in this document.  Overlay
   encapsulation/decapsulation functions are normally off-loaded offloaded to the
   nNVE on the external External NVE.

                       +------------ Split-NVE ---------+
                       |                                |
                       |                                |
     +-----------------|-----+                          |
     | +---------------|----+|                          |
     | | +--+         \|/   ||                          |
     | | |V |TSI  +-------+ ||                   +------|-------------+
     | | |M |-----+       | ||                   |     \|/            |
     | | +--+     |       | ||                   |+--------+          |
     | | +--+     | tNVE  | ||-------------------||        |          |
     | | |V |TSI  |       | ||                   || nNVE   |          |
     | | |M |-----|       | ||                   ||        |          |
     | | +--+     +-------+ ||                   |+--------+          |
     | |                    ||                   +--------------------+
     | +-----Hypervisor-----+|
     +-----------------------+
            End Device                               External NVE

                       Figure 1 1: Split-NVE structure Structure

   The tNVE is normally implemented as a part of a hypervisor or
   container and/or a virtual switch in an a virtualized end device.  This
   document uses the term "hypervisor" throughout when describing the
   Split-NVE scenario where part of the NVE functionality is off-loaded offloaded
   to a separate device from the "hypervisor" that contains a VM
   (Virtual Machine) connected to a VN (Virutal (Virtual Network).  In this
   context, the term "hypervisor" is meant to cover any device type
   where part of the NVE functionality is off-loaded offloaded in this fashion, e.g.,a
   e.g., a Network Service Appliance or Linux Container.

   The NVO3 Network Virtualization over Layer 3 (NVO3) problem statement [RFC7364],
   [RFC7364] discusses the needs need for a
   control plane control-plane protocol (or
   protocols) to populate each NVE with the state needed to perform the
   required functions.  In one scenario, an NVE provides overlay
   encapsulation/decapsulation packet forwarding packet-forwarding services to Tenant
   Systems (TSs) that are co-resident within the NVE on the same End Device (e.g. end device
   (e.g., when the NVE is embedded within a hypervisor or a Network
   Service Appliance).  In such cases, there is no need for a
   standardized protocol between the hypervisor and the NVE, as the
   interaction is implemented via software on a single device.
   While  However,
   in the Split-NVE architecture scenarios, as scenarios shown in figure Figures 2
   to figure 4, control plane protocol(s) through 4
   (see Section 1.2), one or more control-plane protocols between a
   hypervisor and its associated external External NVE(s) are required for the
   hypervisor to distribute the virtual machines VM's networking states to the NVE(s) for
   further handling.  The protocol is an NVE-internal protocol and runs
   between tNVE and nNVE logical entities.  This protocol is mentioned
   in the "third work area" text in Section 4.5 of the NVO3 problem
   statement [RFC7364] and appears as the third work
   item.

   Virtual machine [RFC7364].

   VM states and state transitioning are summarized in this
   document document,
   showing events where the NVE needs to take specific actions.  Such
   events might correspond to actions that the control plane control-plane signaling
   protocol(s)
   protocol or protocols need to take between the tNVE and the nNVE in
   the Split-NVE scenario.  The high level high-level requirements to be fulfilled
   are stated.

   This listed in Section 4.

   To describe the requirements, this document uses VMs as an example of
   Tenant Systems (TSs) in order
   to describe the requirements, Systems, even though a VM is just one type of Tenant System
   that may connect to a VN.  For example, a service instance within a
   Network Service Appliance is another type of TS, Tenant System, as are
   systems running on an OS-level virtualization technologies like
   containers.  The fact that VMs have lifecycles (e.g., can be created
   and destroyed, can be moved, and can be started or stopped) results
   in a general set of protocol requirements, most of which are
   applicable to other forms of TSs Tenant Systems, although not all of the
   requirements are applicable to all forms of TSs. Tenant Systems.

   Section 2 describes VM states and state transitioning in the VM's
   lifecycle.  Section 3 introduces Hypervisor-to-NVE control plane hypervisor-to-NVE control-plane
   protocol functionality derived from VM operations and network events.
   Section 4 outlines the requirements of the control plane control-plane protocol to
   achieve the required functionality.

1.1

1.1.  Terminology

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
   "OPTIONAL" in this document are to be interpreted as described in
   BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all
   capitals, as shown here.

   This document uses the same terminology as the terminology found in
   [RFC7365].  This section defines additional terminology used by this
   document.

   Split-NVE: a  A type of NVE (Network Virtualization Edge) where the
      functionalities are split across an end device supporting
      virtualization and an external network device.

   tNVE: the  Terminal-side NVE.  The portion of Split-NVE functionalities
      located on the end device supporting virtualization. It  The tNVE
      interacts with a tenant system Tenant System through an internal interface in
      the end device.

   nNVE: the  Network-side NVE.  The portion of Split-NVE functionalities
      located on the network device that is directly or indirectly
      connected to the end device
   holding that contains the corresponding tNVE.
      The nNVE normally performs encapsulation to and decapsulation from
      the overlay network.

   External NVE: the  The physical network device holding that contains the nNVE nNVE.

   Hypervisor: the  The logical collection of software, firmware firmware, and/or
      hardware that allows the creation and running of server or service
      appliance virtualization.  The tNVE is located under a Hypervisor.
   Hypervisor hypervisor.
      The term "hypervisor" is loosely used in this document to refer to
      the end device supporting the virtualization.  For simplicity, we
      also use
   Hypervisor to represent both hypervisor and container.

   Container: Please refer to Hypervisor. For simplicity this document
   use the term hypervisor "hypervisor" to represent both the hypervisor
      and the container.

   Container:  Please see "Hypervisor:" above.

   VN Profile:  Meta data  Metadata that is associated with a VN (Virtual Network) that is and applied to any
      attachment point to the VN. That is, VN (i.e., VAP (Virtual Access Point)
      properties that are applied to all VAPs associated with a given VN
      and used by an NVE when ingressing/egressing packets to/from a
      specific VN.  Meta data VN).  Metadata could include such information as
   ACLs, Access
      Control Lists (ACLs) and QoS settings, etc. settings.  The VN Profile contains
      parameters that apply to the VN as a whole.  Control protocols
      between the NVE and the NVA (Network Virtualization Authority)
      could use the VN ID or VN Name to obtain the VN Profile.

   VSI:  Virtual Station Interface. [IEEE 802.1Q]  See [IEEE802.1Q].

   VDP:  VSI Discovery and Configuration Protocol [IEEE 802.1Q]

1.2 Protocol.  See [IEEE802.1Q].

1.2.  Target Scenarios

   In the Split-NVE architecture, an external External NVE can provide an offload offloading
   of the encapsulation / decapsulation encapsulation/decapsulation functions and network policy
   enforcement as well as offloading of overhead from the VN Overlay protocol overhead. overlay
   protocol.  This offloading may improve performance and/or save
   resources in the End
   Device (e.g. end device (e.g., hypervisor) using the external
   External NVE.

   The following figures

   Figures 2 through 4 give example scenarios of a for the Split-NVE
   architecture.

              Hypervisor             Access Switch
         +------------------+       +-----+-------+
         | +--+   +-------+ |       |     |       |
         | |VM|---|       | | VLAN  |     |       |
         | +--+   | tNVE  |---------+ nNVE|       +--- Underlying
         | +--+   |       | | Trunk |     |       |    Network
         | |VM|---|       | |       |     |       |
         | +--+   +-------+ |       |     |       |
         +------------------+       +-----+-------+

                 Figure 2 2: Hypervisor with an External NVE

             Hypervisor       L2 Switch
         +---------------+     +-----+     +----+---+
         | +--+   +----+ |     |     |     |    |   |
         | |VM|---|    | |VLAN |     |VLAN |    |   |
         | +--+   |tNVE|-------+     +-----+nNVE|   +--- Underlying
         | +--+   |    | |Trunk|     |Trunk|    |   |    Network
         | |VM|---|    | |     |     |     |    |   |
         | +--+   +----+ |     |     |     |    |   |
         +---------------+     +-----+     +----+---+

                 Figure 3 3: Hypervisor with an External NVE
                   connected
                Connected through an Ethernet Access Switch

        Network Service Appliance          Access Switch
         +--------------------------+
     +-----------------------------+      +-----+-------+
     | +------------+ +---------------+    | \    |      |     |       |
     | |Net Service |----| |Network Service|----|  \   |      |     |       |
     | |Instance       |    |   \  | VLAN |     |       |
     | +------------+ +---------------+    |tNVE| |------+nNVE |       +--- Underlying
     | +------------+ +---------------+    |    | | Trunk|     |       |    Network
     | |Net Service |----| |Network Service|----|   /  |      |     |       |
     | |Instance       |    |  /   |      |     |       |
     | +------------+ +---------------+    | /    |      |     |       |
         +--------------------------+
     +-----------------------------+      +-----+-------+

     Figure 4 4: Physical Network Service Appliance with an External NVE

   Tenant Systems connect to external External NVEs via a Tenant System Interface
   (TSI).  The TSI logically connects to the external External NVE via a Virtual
   Access Point (VAP) VAP
   [RFC8014].  The external External NVE may provide Layer 2 or Layer 3
   forwarding.  In the Split-NVE architecture, the external External NVE may be
   able to reach multiple MAC Media Access Control (MAC) addresses and IP
   addresses via a TSI.  An IP address can be in either IPv4 or IPv6
   format.  For example, Tenant Systems that are providing network
   services (such as a transparent firewall, load balancer, or VPN
   gateway) are likely to have a complex address hierarchy.  This
   implies that if a given TSI disassociates de-associates from one VN, all the MAC
   and/or IP addresses are also disassociated. de-associated.  There is no need to
   signal the deletion of every MAC or IP address when the TSI is
   brought down or deleted.  In the majority of cases, a VM will be
   acting as a simple host that will have a single TSI and as well as a
   single MAC and IP address visible to the external External NVE.

   Figures 2 through 4 show the use of VLANs to separate traffic for
   multiple VNs between the tNVE and the nNVE; VLANs are not strictly
   necessary if only one VN is involved, but multiple VNs are expected
   in most cases. Hence  Hence, this draft document assumes the presence of VLANs.

2.  VM Lifecycle

   Figure 2 of [RFC7666] shows the state transition states and transitions of a VM.  Some
   of the VM states are of interest to the external External NVE.  This section
   illustrates the relevant phases and events in the VM lifecycle.  Note
   that the following subsections do not give an exhaustive traversal descriptions of
   VM lifecycle state. They states.  Rather, they are intended as the illustrative
   examples
   which that are relevant to the Split-NVE architecture, architecture and not as
   prescriptive text; the goal is to capture sufficient detail to set a
   context for the signaling protocol signaling-protocol functionality and requirements
   described in the following sections.

2.1

2.1.  VM Creation Event

   The VM creation event causes the VM state to transition from Preparing the
   "preparing" state to Shutdown the "shutdown" state and then to Running the "running"
   state [RFC7666].  The end device allocates and initializes local
   virtual resources like storage in the VM
   Preparing VM's preparing state.  In the Shutdown
   shutdown state, the VM has everything ready ready, except that CPU
   execution is not scheduled by the hypervisor and the VM's memory is
   not resident in the hypervisor.  The transition from the
   Shutdown shutdown
   state to the Running running state normally requires human action or a system triggered
   system-triggered event. Running  The running state indicates that the VM is
   in the normal execution state.  As part of transitioning the VM to
   the
   Running running state, the hypervisor must also provision network
   connectivity for the VM's TSI(s) so that Ethernet frames can be sent
   and received correctly.  Initially, when Running, in the running state, no
   ongoing migration, suspension suspension, or shutdown is in process.

   In the VM creation phase, the VM's TSI has to be associated with the
   external
   External NVE. Association  "Association" here indicates that the hypervisor and
   the
   external External NVE have signaled each other and reached some form of
   agreement.  Relevant networking parameters or information have been
   provisioned properly.  The External NVE should be informed of the
   VM's TSI MAC address and/or IP address.  In addition to external
   network connectivity, the hypervisor may provide local network
   connectivity between the VM's TSI and TSIs for other VM's TSI VMs that are
   co-resident on the same hypervisor.  When the intra- or
   inter-hypervisor connectivity is extended to the external External NVE, a
   locally significant tag, e.g. e.g., VLAN ID, should be used between the
   hypervisor and the external External NVE to differentiate each VN's traffic.
   Both the hypervisor and external External NVE sides must agree on that tag
   value for traffic identification, isolation, and forwarding.

   The external External NVE may need to do some preparation before it signals
   successful association with the TSI.  Such preparation may include
   locally saving the states and binding information of the tenant
   system interface TSI and its VN,
   VN or communicating with the NVA for network
   provisioning, etc.

   Tenant System interface provisioning.

   A TSI association should be performed before the VM enters the Running
   running state, preferably in the Shutdown shutdown state.  If the association
   with an external External NVE fails, the VM should not go into the
   Running running
   state.

2.2

2.2.  VM Live Migration Event

   Live migration is sometimes referred to as "hot" migration in that,
   from an external viewpoint, the VM appears to continue to run while
   being migrated to another server (e.g., TCP connections generally
   survive this class of migration).  In contrast, "cold" migration
   consists of shutting down VM execution on one server and restarting
   it on another.  For simplicity, the following abstract summary of
   live migration assumes shared storage, so that the VM's storage is
   accessible to the source and destination servers.  Assume that the VM live
   migrates
   "live migrates" from hypervisor 1 to hypervisor 2.  Such a migration
   event involves state transitions on both source hypervisor 1 and
   destination hypervisor 2.  The VM state on source hypervisor 1
   transits
   transitions from Running the running state to Migrating the "migrating" state and then
   to Shutdown the shutdown state [RFC7666].  The VM state on destination
   hypervisor 2 transits transitions from Shutdown the shutdown state to
   Migrating the migrating
   state and then Running. to the running state.

   The external External NVE connected to destination hypervisor 2 has to
   associate the migrating VM's TSI with it itself (i.e., the External NVE)
   by discovering the TSI's MAC and/or IP addresses, discovering its VN,
   discovering its locally significant VLAN ID if any, (if any), and
   provisioning other network related network-related parameters of the TSI.  The
   external
   External NVE may be informed about the VM's peer VMs, storage devices
   devices, and other network appliances with which the VM needs to
   communicate or is communicating.  The migrated VM on destination
   hypervisor 2 should not go to Running the running state until all the network
   provisioning and binding has have been done.

   The states of VM state on both the source hypervisor and the destination hypervisors both are
   Migrating
   hypervisor will be the migrating state during the transfer of migration VM
   execution.  The migrating VM should not be in Running the running state at
   the same time on the source hypervisor and destination hypervisor
   during migration.  The VM on the source hypervisor does not
   transition into Shutdown to the shutdown state until the VM successfully enters the Running
   running state on the destination hypervisor.  It is possible that the
   VM on the source hypervisor stays in Migrating the migrating state for a while
   after the VM on the destination hypervisor enters Running the running state.

2.3

2.3.  VM Termination Event

   A VM termination event is also referred to as "powering off" a VM.  A
   VM termination event leads to its state becoming Shutdown. There the VM's transition to the shutdown
   state.  Per [RFC7666], there are two possible causes of VM termination [RFC7666]. One is the normal
   "power off" of a
   termination:

   1.  A running VM; the other is that the VM has undergone a normal "power-off".

   2.  The VM has been migrated to another hypervisor hypervisor, and the VM image
       on the source hypervisor has to stop executing and be shutdown. shut down.

   In VM termination, the external External NVE connecting to that VM needs to
   deprovision the VM, i.e. i.e., delete the network parameters associated
   with that VM.  In other words, the external External NVE has to de-associate
   the VM's TSI.

2.4

2.4.  VM Pause, Suspension Suspension, and Resumption Events

   A VM pause event leads to the VM transiting transitioning from Running the running state
   to
   Paused the "paused" state.  The Paused paused state indicates that the VM is
   resident in memory but no longer that CPU execution is not scheduled to execute by the
   hypervisor [RFC7666].  The VM can be easily re-activated reactivated from Paused the
   paused state to
   Running the running state.

   A VM suspension event leads to the VM transiting transitioning from Running the running
   state to Suspended the "suspended" state.  A VM resumption event leads to the
   VM transiting
   state transitioning from Suspended the suspended state to Running the running state. Suspended state means  In
   the suspended state, the memory and CPU execution state of the virtual machine VM are
   saved to persistent store. storage.  During this state, CPU execution for
   the virtual machine VM is not scheduled to execute by the hypervisor [RFC7666].

   In the Split-NVE architecture, the external External NVE should not
   disassociate
   de-associate the paused or suspended VM VM, as the VM can return to
   Running the
   running state at any time.

3.  Hypervisor-to-NVE Control Plane Control-Plane Protocol Functionality

   The following subsections show illustrative examples of the state
   transitions of an external External NVE which that are relevant to Hypervisor-to-
   NVE Signaling protocol hypervisor-to-NVE
   signaling-protocol functionality. It should be noted this  Note: This is not prescriptive
   text for the full state machine.

3.1 VN Connect

3.1.  VN_Connect and Disconnect VN_Disconnect

   In the Split-NVE scenario, a protocol is needed between the End
   Device (e.g. Hypervisor) end
   device (e.g., hypervisor) and the external External NVE it is using using, in order
   to make the external External NVE aware of the changing VN membership
   requirements of the Tenant Systems within the End Device. end device.

   A key driver for using a protocol rather than using static
   configuration of the external External NVE is because that the VN connectivity
   requirements can change frequently as VMs are brought up, moved, and
   brought down on various hypervisors throughout the data center or
   external cloud.

   Figure 5 shows the state transition for a VAP on the external External NVE.
   An NVE that supports the hypervisor to NVE control plane hypervisor-to-NVE control-plane protocol
   should support one instance of the state machine for each active VN.
   The state transition on the external External NVE is normally triggered by the
   hypervisor-facing side
   events and behaviors. behaviors on the hypervisor-facing side.  Some of the
   interleaved
   interaction interactions between the NVE and the NVA will be
   illustrated to better explain the whole procedure; procedure, while others of them may other
   interactions will not be shown.

       +---------------+

   +----------------+   Receive VN_connect;     +-------------------+
       |VN_Disconnected| VN_Connect;     +----------------------+
   |VN_Disconnected |   return Local_Tag value  |VN_Connected          |
       +---------------+
   +----------------+   for VN if successful;   +-------------------+   +----------------------+
   |VN_ID;          |-------------------------->|VN_ID;                |
   |VN_State=       |                           |VN_State=connected;|
       |disconnected;                           |VN_State=VN_Connected;|
   |VN_Disconnected;|                           |Num_TSI_Associated;   |                           |Num_TSI_Associated;|
   |                |<--Receive VN_disconnect---|Local_Tag; VN_Disconnect---|Local_Tag;            |
       +---------------+
   +----------------+                           |VN_Context;           |
                                                   +-------------------+
                                                +----------------------+

           Figure 5. 5: State Transition Example of a VAP Instance
                            on an External NVE

   The external External NVE must be notified when an End Device end device requires a
   connection to a particular VN and when it no longer requires a
   connection.  Connection clean up cleanup for the failed devices should be
   employed which
   employed.  Note that this topic is out of the scope of for the protocol
   specified in this document.

   In addition, the external External NVE should provide a local tag value for
   each connected VN to the End Device end device to use for exchanging packets
   between the End Device end device and the external External NVE (e.g. (e.g., a locally
   significant [IEEE 802.1Q] tag value). value per [IEEE802.1Q]).  How "local" the
   significance is depends on whether

   1.  the Hypervisor hypervisor has a direct physical connection to the external
       External NVE (in which case the significance is local to the
       physical link), link) or whether

   2.  there is an Ethernet switch (e.g. (e.g., a blade switch) connecting the Hypervisor
       hypervisor to the NVE (in which case the significance is local to
       the intervening switch and all the links connected to it).

   These VLAN tags are used to differentiate between different VNs as
   packets cross the shared access shared-access network to the external External NVE.  When
   the
   external External NVE receives packets, it uses the VLAN tag to identify
   their VN coming from a given TSI, strips the tag, adds the
   appropriate overlay encapsulation for that VN, and sends it towards
   the corresponding remote NVE across the underlying IP network.

   The Identification of the VN in this protocol could either be through either
   a VN Name or a VN ID.  A globally unique VN Name facilitates
   portability of a Tenant's Virtual Data Center. tenant's virtual data center.  Once an external External NVE
   receives a VN connect indication, VN_Connect message, the NVE needs a way to get a VN
   Context
   VN_Context allocated (or to receive the already allocated VN Context) already-allocated VN_Context)
   for a given VN Name or VN ID (as well as any other information needed
   to transmit encapsulated packets).  How this is done is the subject
   of the NVE-to-NVA protocol which are part protocol; see the "first two areas of work items 1 and 2 work" text in
   Section 4.5 of [RFC7364].  The external External NVE needs to synchronize the
   mapping information of the local tag and VN Name or VN ID with the
   NVA.

   The VN_connect VN_Connect message can be explicit or implicit. Explicit  "Explicit" means
   that the hypervisor sends a request message explicitly for the
   connection to a VN. Implicit  "Implicit" means that the external External NVE receives
   other messages,
   e.g. e.g., the very first TSI associate Associate message (see the
   next subsection) for a given VN, that implicitly indicate its
   interest in connecting to a VN.

   A VN_disconnect VN_Disconnect message indicates that the NVE can release all the
   resources for that disconnected VN and transit transition to VN_disconnected the
   VN_Disconnected state.  The local tag assigned for that VN can
   possibly be reclaimed for use by another VN.

3.2

3.2.  TSI Associate and Activate

   Typically, a TSI is assigned a single MAC address address, and all frames
   transmitted and received on that TSI use that single MAC address.  As
   mentioned earlier, it is also possible for a Tenant System to
   exchange frames using multiple MAC addresses or packets with multiple
   IP addresses.

   Particularly in the case of a TS Tenant System that is forwarding frames
   or packets from other TSs, Tenant Systems, the external External NVE will need to
   communicate the mapping between the NVE's IP address on the
   underlying network and ALL the addresses the TS Tenant System is
   forwarding on behalf of the corresponding VN to the NVA.

   The NVE has two ways it can discover the tenant addresses for which
   frames are to be forwarded to a given End Device end device (and ultimately to
   the TS Tenant System within that End Device). end device).

   1.  It can glean the addresses by inspecting the source addresses in
       packets it receives from the End Device. end device.

   2.  The hypervisor can explicitly signal the address associations of
       a TSI to the external External NVE.  An address association includes all
       the MAC and/or IP addresses possibly used as source addresses in
       a packet sent from the hypervisor to external the External NVE.  The external
       External NVE may further use this information to filter the
       future traffic from the hypervisor.

   To use the second approach above, the "hypervisor-to-NVE" control-plane protocol running
   between the hypervisor and the NVE must support End Devices end devices
   communicating new tenant addresses tenant-address associations for a given TSI within
   a given VN.

   Figure 6 shows the an example of a state transition for a TSI connecting
   to a VAP on the external External NVE.  An NVE that supports the hypervisor to
   NVE control plane hypervisor-
   to-NVE control-plane protocol may support one instance of the state
   machine for each TSI connecting to a given VN.

                 disassociate

                De-Associate   +--------+     disassociate     De-Associate
              +--------------->|  Init  |<--------------------+
              |                +--------+                     |
              |                |        |                     |
              |                |        |                     |
              |                +--------+                     |
              |                  |    |                       |
              |       associate       Associate  |    |  activate  Activate             |
              |      +-----------+    +-----------+           |
              |      |                            |           |
              |      |                            |           |
              |     \|/                          \|/          |
      +--------------------+                  +---------------------+
      |     Associated     |                  |       Activated     |
      +--------------------+                  +---------------------+
      |TSI_ID;             |                  |TSI_ID;              |
      |Port;               |-----activate---->|Port;               |-----Activate---->|Port;                |
      |VN_ID;              |                  |VN_ID;               |
       |State=associated;
      |State=Associated;   |                  |State=activated ;                  |State=Activated;     |-+
    +-|Num_Of_Addr;        |<---deactivate        |<---Deactivate ---|Num_Of_Addr;         | |
    | |List_Of_Addr;       |                  |List_Of_Addr;        | |
    | +--------------------+                  +---------------------+ |
    |                    /|\                     /|\                  |
    |                     |                       |                   |
    +---------------------+                       +-------------------+
     add/remove/updt addr;                        add/remove/updt addr;
     or update port;                              or update port;

           Figure 6 6: State Transition Example of a TSI Instance
                            on an External NVE

   The Associated state of a TSI instance on an external External NVE indicates
   that all the addresses for that TSI have already associated with the
   VAP of the external External NVE on a given port e.g. port, e.g., on port p for a given VN
   VN, but no real traffic to and from the TSI is expected and allowed
   to pass through.  An NVE has reserved all the necessary resources for
   that TSI.  An external External NVE may report the mappings of its underlay IP
   address and the associated TSI addresses to NVA the NVA, and relevant
   network nodes may save such information to their mapping tables but
   not their forwarding tables.  An NVE may create ACL ACLs or filter rules
   based on the associated TSI addresses on that attached port p but not
   enable them yet.  The local tag for the VN corresponding to the TSI
   instance should be provisioned on port p to receive packets.

   The VM migration event (discussed section in Section 2) may cause the
   hypervisor to send an associate Associate message to the NVE connected to the
   destination hypervisor of the migration.  A VM creation event may
   also cause to trigger the same practice. scenario.

   The Activated state of a TSI instance on an external External NVE indicates
   that all the addresses for that TSI are functioning correctly on a
   given port, e.g., port e.g. port p p, and traffic can be received from and sent
   to that TSI via the NVE.  The mappings of the NVE's underlay IP
   address and the associated TSI addresses should be put into added to the
   forwarding table rather than the mapping table on relevant network
   nodes. ACL  ACLs or filter rules based on the associated TSI addresses on
   the attached port p in on the NVE are enabled.  The local tag for the VN
   corresponding to the TSI instance must be provisioned on port p to
   receive packets.

   The Activate message makes the state transit transition from Init or
   Associated to Activated.  VM creation, VM migration, and VM
   resumption events
   discussed (discussed in Section 4 2) may trigger sending the
   Activate message from the hypervisor to the external External NVE.

   TSI information may get updated in either the Associated state or the
   Activated state.  The following are considered updates to the TSI
   information: add or remove the associated addresses, update the
   current associated addresses (for example updating example, update the IP address for
   a given MAC), MAC address), and update the NVE port information based on
   where the NVE receives messages.  Such updates do not change the
   state of the TSI.  When any address associated with a given TSI
   changes, the NVE should inform the NVA to update the mapping
   information for the NVE's underlying address and the associated TSI
   addresses.  The NVE should also change its local ACL ACLs or filter
   settings accordingly for the relevant addresses.  Port information
   updates will cause the provisioning of the local tag for the VN
   corresponding to the TSI instance on the new port and removal from
   the old port.

3.3

3.3.  TSI Disassociate De-Associate and Deactivate

   Disassociate

   De-Associate and deactivate Deactivate behaviors are conceptually the reverse of
   associate
   Associate and activate. Activate.

   From the Activated state to the Associated state, the external External NVE
   needs to make sure the resources are still reserved but the addresses
   associated to with the TSI are not functioning.  No traffic to or from
   the TSI is expected or allowed to pass through.  For example, the NVE
   needs to tell the NVA to remove the relevant addresses mapping information regarding
   address mapping from the forwarding and routing tables. ACL  ACLs and filtering
   filter rules regarding the relevant addresses should be disabled.

   From the Associated or Activated state to the Init state, the NVE
   releases all the resources relevant to TSI instances.  The NVE should
   also inform the NVA to remove the relevant entries from the mapping
   table. ACL  ACLs or filtering filter rules regarding the relevant addresses should
   be removed.  Local tag provisioning on the connecting port on the NVE
   should be cleared.

   A VM suspension event (discussed in section Section 2) may cause the relevant
   TSI instance(s) on the NVE to transit transition from the Activated state to
   the Associated state.

   A VM pause event normally does not affect the state of the relevant
   TSI instance(s) on the NVE NVE, as the VM is expected to run again soon.

   A VM shutdown event will normally cause the relevant TSI instance(s)
   on the NVE to transition to the Init state from the Activated state.
   All resources should be released.

   A VM migration will cause the TSI instance on the source NVE to leave
   the Activated state.  When a VM migrates to another hypervisor
   connecting to the same NVE, i.e. i.e., the source and destination NVE are
   the same, the NVE should use the TSI_ID and the incoming port to
   differentiate two TSI instances.

   Although the triggering messages for the state transition shown in
   Figure 6 does do not indicate the difference between a VM
   creation/shutdown event and a VM migration arrival/departure event,
   the external External NVE can make optimizations if it is given such
   information.  For example, if the NVE knows that the incoming activate
   Activate message is caused by migration rather than VM creation, some
   mechanisms may be employed or triggered to make sure the dynamic
   configurations or provisionings on the destination NVE are the same
   as those on the source NVE for the migrated VM.  For example example, an IGMP
   query [RFC2236] can be triggered by the destination external External NVE to
   the migrated VM so that the VM is forced to send an IGMP report to
   the multicast router. Then a  The multicast router can then correctly route
   the multicast traffic to the new external External NVE for those multicast
   groups the VM joined before the migration.

4.  Hypervisor-to-NVE Control Plane Control-Plane Protocol Requirements

   Req-1:   The protocol MUST support a bridged network connecting End
   Devices end
            devices to the External NVE.

   Req-2:   The protocol MUST support multiple End Devices end devices sharing the
            same External NVE via the same physical port across a
            bridged network.

   Req-3:   The protocol MAY support an End Device end device using multiple external
            External NVEs simultaneously, but only one external External NVE for
            each VN. VN (active-standby External NVE case for a VN).

   Req-4:   The protocol MAY support an End Device end device using multiple external
            External NVEs simultaneously for the same VN. VN (active-active
            External NVE case for a VN).

   Req-5:   The protocol MUST allow the End Device end device to initiate a request
            to its associated External NVE to be connected/disconnected
            to a given VN.

   Req-6:   The protocol MUST allow an External NVE initiating a request
            to its connected End Devices end devices to be disconnected from a
            given VN.

   Req-7:   When a TS Tenant System attaches to a VN, the protocol MUST
            allow for an End
   Device end device and its external External NVE to negotiate
            one or more locally- locally significant tag(s) tags for carrying traffic
            associated with a specific VN (e.g., [IEEE 802.1Q] tags). tags per [IEEE802.1Q]).

   Req-8:   The protocol MUST allow an End Device end device initiating a request
            to
   associate/disassociate associate/de-associate and/or activate/deactive activate/deactivate some or
            all
   address(es) addresses of a TSI instance to a VN on an NVE port.

   Req-9:   The protocol MUST allow the External NVE initiating a
            request to disassociate de-associate and/or deactivate some or all address(es)
            addresses of a TSI instance to a VN on an NVE port.

   Req-10:  The protocol MUST allow an End Device end device initiating a request
            to add, remove remove, or update address(es) associated with a TSI
            instance on the external External NVE.  Addresses can be expressed in
            different formats, formats -- for example, MAC, IP IP, or pair of IP and MAC. IP-MAC pair.

   Req-11:  The protocol MUST allow the External NVE and the connected
   End Device
            end device to authenticate each other.

   Req-12:  The protocol MUST be able to run over L2 Layer 2 links between
            the
   End Device end device and its External NVE.

   Req-13:  The protocol SHOULD support the End Device indicating if an
   associate end device that indicates
            that an Associate or activate Activate request from it the end device is
            the result of a VM hot migration event.

5.  VDP Applicability and Enhancement Needs

   The Virtual Station Interface (VSI) Discovery and Configuration
   Protocol (VDP) [IEEE 802.1Q] [IEEE802.1Q] can be the control plane control-plane protocol running
   between the hypervisor and the external External NVE.  Appendix A illustrates provides
   informative VDP illustrations for the reader's information. reader.

   VDP facilitates the automatic discovery and configuration of Edge
   Virtual Bridging (EVB) stations and Edge Virtual Bridging (EVB) EVB bridges.  An EVB station is
   normally an end station running multiple VMs. It  In this document, it
   is considered conceptually equivalent to a hypervisor in this document. hypervisor.  An EVB bridge
   is conceptually equivalent to the external External NVE.

   VDP is able to pre-associate/associate/de-associate a VSI on an EVB
   station with a port on the EVB bridge. A VSI is approximately  In the
   concept context of this
   document, a VSI is conceptually approximate to a virtual port by
   which a VM connects to the hypervisor in
   this document's context. hypervisor.  The EVB station and the EVB
   bridge can reach agreement on VLAN ID(s) assigned to a VSI via a VDP
   message exchange.  Other configuration parameters can be exchanged
   via VDP as well.  VDP is carried over the Edge Control Protocol(ECP) [IEEE 802.1Q] Protocol (ECP)
   [IEEE802.1Q], which provides a reliable transportation over a layer Layer 2
   network.

   VDP protocol needs some extensions to fulfill the requirements listed in
   Section 4 of this document.  Table 1 shows the needed extensions
   and/or clarifications in the NVO3 context.

   +------+-----------+-----------------------------------------------+
   | Req  | Supported |   remarks                    Remarks                    |
   |      | by VDP?   |                                               |
   +------+-----------+-----------------------------------------------+
   | Req-1|           |                                               |
   +------+           |Needs extension.  Must be able to send to a    |
   | Req-2|           |specific unicast MAC MAC, and should be able to send|    |
   +------+ Partially |to |send to a non-reserved well known well-known multicast address    |
   | Req-3|           |other           |address other than the nearest customer bridge address.|
   +------+ |
   +------+           |address.                                       |
   | Req-4|           |                                               |
   +------+-----------+-----------------------------------------------+
   | Req-5| Yes       |VN       |The VN is indicated by GroupID GroupID.                |
   +------+-----------+-----------------------------------------------+
   | Req-6| Yes       |Bridge       |The bridge sends De-Associate a De-Associate.               |
   +------+-----------+------------------------+----------------------+
   |      |           |VID==NULL in request and the request.  The bridge returns the  |
   | Req-7| Yes       |assigned      |           |the assigned VLAN ID (VID) value in response or specify GroupID the        |
   | Req-7| Yes       |response.  GroupID, which is optionally present|
   |      |           |in request and get VID assigned the request, is equivalent to the VN ID in returning  |
   |      |           |response.           |the context of NVO3.  Multiple VLANs per group are allowed.|
   +------+-----------+------------------------+----------------------+ |
   |      |  requirements           |are allowed.                                   |  VDP equivalence
   +------+-----------+------------------------+----------------------+
   |      |           |           +------------------------+----------------------+     Requirements       |    VDP Equivalent    |
   |  associate/disassociate|pre-asso/de-associate      |           +------------------------+----------------------+
   | Req-8| Partially |  activate/deactivate   |associate/de-associate| Associate/De-Associate |Pre-Assoc/De-Associate|
   |      |           | Activate/Deactivate    |Associate/De-Associate|
   |      |           +------------------------+----------------------|
   |      |           |Needs extension to allow associate->pre-assoc Associate->Pre-Assoc. |
   +------+-----------+------------------------+----------------------+
   | Req-9| Yes       |       |The VDP bridge initiates de-associate a De-Associate.       |
   +------+-----------+-----------------------------------------------+
   |Req-10| Partially |Needs extension for an IPv4/IPv6 address. Add a      |
   |      |           |new           |Add a new "filter info information format" type.    |
   +------+-----------+-----------------------------------------------+
   |Req-11| No        |Out-of-band
   |      |           |An out-of-band mechanism is preferred, e.g. MACSec| e.g.,   |
   |           |or      |           |MACsec or 802.1X.  Implicit authentication based on    |     |      |           |control
   |Req-11| No        |based on control of physical connectivity exists in VDP      |
   |      |           |when           |exists in VDP when the External NVE connects to the End to|
   |      |      |           |Device           |the end device directly and is reachable with the  |
   |      |           |nearest           |the nearest customer bridge address.           |
   +------+-----------+-----------------------------------------------+
   |Req-12| Yes       |L2 protocol       |VDP naturally runs on the Layer 2 protocol.    |
   +------+-----------+-----------------------------------------------+
   |      |           |M bit for migrated VM on destination hypervisor|           |A migration event may cause the M-bit to be set|
   |      |           |and S bit for that on source hypervisor.           |to 1 in the VDP request to the migration       |
   |      |           |destination hypervisor and the S-bit to be set |
   |      |           |to 1 in the VDP request to the migration source|
   |      |           |hypervisor.  However, a setting of M-bit = 0 or|
   |Req-13| Partially |It is indistinguishable when M/S is |S-bit = 0 between can indicate that no information is  |
   |      |           |no guidance and events           |available regarding migration or that the      |
   |      |           |events in question are not caused by migration | migration.|
   |      |           |where NVE may act differently. Needs new           |To fully meet the requirement, this ambiguity  |
   |      |           |New bits for           |would need to be fixed so that migration indication in new or no |
   |      |           |"filter info format" type.           |migration could be safely inferred from the    |
   |      |           |M-bit or S-bit settings.                       |
   +------+-----------+-----------------------------------------------+

    Table 1 Compare 1: Comparison of Split-NVE Requirements and VDP with the requirements

   Simply Capabilities

   By simply adding the ability to carry layer Layer 3 addresses, addresses as per
   Req-10, VDP can serve
   the Hypervisor-to-NVE control plane functions pretty well. Other
   extensions are the improvement provide most of the protocol capabilities for
   better fit in an NVO3 network. hypervisor-to-NVE control-plane
   functionality required.

6.  Security Considerations

   External NVEs must ensure that only properly authorized Tenant
   Systems are allowed to join and become a part of any particular
   Virtual Network. VN.
   In some cases, the tNVE may want to connect to the nNVE for
   provisioning purposes.  This may require that the tNVE authenticate
   the nNVE in addition to the nNVE authenticating the tNVE.  If a
   secure channel is required between the tNVE and the nNVE to carry the
   encrypted split-NVE control plane Split-NVE control-plane protocol, then existing mechanisms
   such as MACsec [IEEE 802.1AE] [IEEE802.1AE] can be used.  In some deployments,
   authentication may be implicit implicit, based on control of physical
   connectivity, e.g., if the nNVE is located in the bridge that is
   directly connected to the server that contains the tNVE. Use  The use of
   the "nearest customer bridge address" in VDP [IEEE 802.1Q] [IEEE802.1Q] is an
   example of where this sort of implicit authentication is possible,
   although explicit authentication also applies in that case.

   As the control plane control-plane protocol results in configuration changes for
   both the tNVE and the nNVE, tNVE and nNVE implementations should log
   all state changes, including those described in Section 3.
   Implementations should also log significant protocol events, such as
   the establishment or loss of control plane control-plane protocol connectivity
   between the tNVE and nNVE and the nNVE, as well as authentication results.

   In addition, external External NVEs will need appropriate mechanisms to ensure
   that any hypervisor wishing to use the services of an NVE is properly
   authorized to do so.  One design point is whether the hypervisor
   should

   1.  supply the external External NVE with necessary information (e.g., VM
       addresses, VN information, or other parameters) that the external
       External NVE uses directly, directly or whether the hypervisor should

   2.  only supply a VN ID and an identifier for the associated VM
       (e.g., its MAC address), with the external External NVE using that
       information to obtain the information needed to validate the
       hypervisor-provided parameters or obtain related parameters in a
       secure manner.

   The former approach can be used in a trusted environment so that the external
   External NVE can directly use all the information retrieved from the
   hypervisor for local configuration.  It saves relieves the effort on the external External NVE
   side from of effort related to information retrieval and/or validation.
   The latter approach gives more reliable information information, as the external
   External NVE needs to retrieve them it from some management system a management-system database. Especially
   In particular, some network related
   parameters like network-related parameters, such as VLAN IDs IDs, can
   be passed back to the hypervisor to be used as a form of provisioning
   that is more authoritative provisioning. However authoritative.  However, in certain cases, cases it is
   difficult or inefficient for an external External NVE to have be granted rights to
   access or query
   on some information to on those management systems. Then the external  The
   External NVE then has to obtain those the information from the hypervisor.

7.  IANA Considerations

   No

   This document has no IANA action is required. actions.

8.  References

8.1

8.1.  Normative References

   [IEEE802.1Q]
              IEEE, "IEEE Standard for Local and metropolitan area
              networks--Bridges and Bridged Networks", IEEE Standard
              802.1Q-2014, DOI 10.1109/IEEESTD.2014.6991462.

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997. 1997,
              <https://www.rfc-editor.org/info/rfc2119>.

   [RFC7365]  Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y.
              Rekhter, "Framework for DC Data Center (DC) Network
              Virtualization", RFC 7365, DOI 10.17487/RFC7365, October 2014.
              2014, <https://www.rfc-editor.org/info/rfc7365>.

   [RFC7666] Asai  Asai, H., MacFaden MacFaden, M., Schoenwaelder Schoenwaelder, J., Shima Shima, K., Tsou T., and
              T. Tsou, "Management Information Base for Virtual Machines
              Controlled by a Hypervisor", RFC 7666,
              DOI 10.17487/RFC7666, October 2015. 2015,
              <https://www.rfc-editor.org/info/rfc7666>.

   [RFC8014]  Black, D., Hudson, J., Kreeger, L., Lasserre, M., and T.
              Narten,
              T., "An Architecture for Data-Center Network
              Virtualization over Layer 3 (NVO3)", RFC 8014,
              DOI 10.17487/RFC8014, December 2016. 2016,
              <https://www.rfc-editor.org/info/rfc8014>.

   [RFC8174]  Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
              2119 Key Words ", Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
              May 2017.

   [IEEE 802.1Q] 2017, <https://www.rfc-editor.org/info/rfc8174>.

8.2.  Informative References

   [IEEE802.1AE]
              IEEE, "Media "IEEE Standard for Local and Metropolitan Area
              Networks: Media Access Control (MAC) Bridges Security",
              IEEE Standard 802.1AE-2006,
              DOI 10.1109/IEEESTD.2006.245590.

   [NVO3-HYPERVISOR-NVE-CP]
              Kreeger, L., Narten, T., and D. Black, "Network
              Virtualization Hypervisor-to-NVE Overlay Control Protocol
              Requirements", Work in Progress, draft-kreeger-nvo3-
              hypervisor-nve-cp-01, February 2013.

   [NVO3-TES-NVE]
              Yingjie, G. and L. Yizhou, "The mechanism and signalling
              between TES and NVE", Work in Progress, draft-gu-nvo3-tes-
              nve-mechanism-01, October 2012.

   [NVO3-VM-NVE]
              Kompella, K., Rekhter, Y., Morin, T., and D. Black,
              "Signaling Virtual
              Bridged Local Area Networks", IEEE Std 802.1Q-2014,
              November 2014.

8.2  Informative References Machine Activity to the Network
              Virtualization Edge", Work in Progress, draft-kompella-
              nvo3-server2nve-02, April 2013.

   [RFC2236]  Fenner, W., "Internet Group Management Protocol, Version
              2", RFC 2236, DOI 10.17487/RFC2236, November 1997. 1997,
              <https://www.rfc-editor.org/info/rfc2236>.

   [RFC4122]  Leach, P., Mealling, M., and R. Salz, "A Universally
              Unique IDentifier (UUID) URN Namespace", RFC 4122,
              DOI 10.17487/RFC4122, July
              2005. 2005,
              <https://www.rfc-editor.org/info/rfc4122>.

   [RFC7364]  Narten, T., Ed., Gray, E., Ed., Black, D., Fang, L.,
              Kreeger, L., and M. Napierala, "Problem Statement:
              Overlays for Network Virtualization", RFC 7364,
              DOI 10.17487/RFC7364, October 2014.

   [IEEE 802.1AE] IEEE, "MAC Security (MACsec)", IEEE Std 802.1AE-2006,
              August 2006. 2014,
              <https://www.rfc-editor.org/info/rfc7364>.

Appendix A. IEEE 802.1Q  VDP Illustration (For information only)

   The Illustrations (per IEEE 802.1Q) (for Information Only)

   VDP (VSI (the VSI Discovery and Discovery and Configuration Protocol,
   clause Protocol; see
   Clause 41 of [IEEE 802.1Q]) [IEEE802.1Q]) can be considered as a controlling
   protocol running between the hypervisor and the external bridge.  The
   VDP association TLV structure are is formatted as shown in Figure A.1. 7.

   +--------+--------+------+-----+--------+------+------+------+------+
   |TLV type|TLV info|Status|VSI Type|TLV Info|Status|VSI  |VSI Type|VSI ID|VSI ID|Filter|Filter|
   |        |string        |String  |      |Type |Version |Format|      |Info  |Info  |
   |        |length        |Length  |      |ID   |        |      |      |format|      |Format|      |
   +--------+--------+------+-----+--------+------+------+------+------+
   |                 |      |<----VSI type&instance----->|<--Filter--->|      |<--VSI Type and instance--->|<--Filter--->|
   |                 |      |<-------------VSI attributes------------->|
   |<--TLV header--->|<-----------TLV information string ------------->|

                       Figure A.1: 7: VDP association Association TLV

   There are basically four TLV types.

   1. Pre-associate: Pre-associate  Pre-Associate: The Pre-Associate is used to pre-associate Pre-Associate a VSI
       instance with a bridge port.  The bridge validates the request
       and returns a failure Status status in the case of errors.  Successful pre-associate  A successful
       Pre-Associate does not imply that the indicated VSI Type or
       provisioning will be applied to any traffic flowing through the
       VSI. The pre-associate
   enables faster response to an associate, by  By allowing the bridge to obtain the VSI Type prior to an association.
       association, the Pre-Associate enables faster response to an
       Associate.

   2. Pre-associate  Pre-Associate with resource reservation: Pre-associate Resource Reservation: The Pre-Associate with
       Resource Reservation involves the same steps as Pre-associate, those for the
       Pre-Associate, but on success it also reserves resources in the
       bridge to prepare for a subsequent Associate request.

   3.  Associate: The Associate request creates and activates an
       association between a VSI instance and a bridge port. An  A bridge
       allocates any required bridge resources for the referenced VSI.
       The bridge activates the configuration for the VSI Type ID.  This
       association is then applied to the traffic flow to/from the VSI
       instance.

   4. De-associate:  De-Associate: The de-associate De-Associate is used to remove an association
       between a VSI instance and a bridge port.  Pre-associated and
       associated VSIs can be de-associated. De-associate  The De-Associate releases
       any resources that were reserved as a result of prior associate or pre- Associate
       or Pre-Associate operations for that VSI instance.

   De-associate

   The De-Associate can be initiated by either side side, and the other types
   can only be initiated by the server side.

   Some important flag values in the VDP Status field: field are as follows:

   1.  M-bit (Bit 5): Indicates M-bit = 1: indicates that the user of the VSI
       (e.g., the VM) is migrating (M-bit migrating.  M-bit = 1) or provides 0: no guidance on the migration of
   the user indication of whether
       the VSI (M-bit = 0). user is migrating.  The M-bit is used as an indicator
       relative to the VSI that to which the user is migrating to. migrating.

   2.  S-bit (Bit 6): Indicates S-bit = 1: indicates that the VSI user (e.g., the
       VM) is
   suspended (S-bit suspended.  S-bit = 1) or provides 0: no guidance as to whether the user indication of whether the VSI
       user is suspended (S-bit = 0). suspended.  A keep-alive Associate request with S-bit = 1
       can be sent when the VSI user is suspended.  The S-bit is used as
       an indicator relative to the VSI that from which the user is
   migrating from.
       migrating.

   The filter information format currently defines 4 four types. Each
   Information for each of the
   filter information these types is shown in details detail in Figures 8
   through 11.  "PCP" stands for Priority Code Point [IEEE802.1Q].  The
   PCP value, if specified, is used by the EVB station as follows.

   1. VID Filter Info format
      +---------+------+-------+--------+ the default
   PCP value associated with the VSI and VID.  The filter information
   contains a PCP Significant (PS) bit associated with each PCP field,
   indicating whether the PCP field carries a PCP value (binary 1) or
   does not carry a PCP value (binary 0).

                 +----------+-------+--------+--0------+
                 | #of # of     | PS    | PCP    | VID     |
                 |entries  |(1bit)|(3bits)|(12bits)|
      |(2octets)|   |(1 bit)|(3 bits)|(12 bits)|
                 |(2 octets)|       |        |         |
      +---------+------+-------+--------+
                |<--Repeated
                 +----------+-------+--------+---------+
                            |<---Repeated per entry->| entry--->|

                  Figure A.2 8: VID Filter Info format

   2. MAC/VID Filter Info format
      +---------+--------------+------+-------+--------+ Information Format

          +----------+--------------+-------+--------+---------+
          | #of # of     |  MAC address | PS    | PCP    | VID     |
          |entries   |  (6 octets)  |(1bit)|(3bits)|(12bits)|
      |(2octets)|  |(1 bit)|(3 bits)|(12 bits)|
          |(2 octets)|              |       |        |         |
      +---------+--------------+------+-------+--------+
                |<--------Repeated
          +----------+--------------+-------+--------+---------+
                     |<----------Repeated per entry---------->| entry----------->|

                Figure A.3 9: MAC/VID filter format

   3. GroupID/VID Filter Info format
      +---------+--------------+------+-------+--------+ Information Format

         +----------+--------------+-------+--------+---------+
         | #of # of     |  GroupID     | PS    | PCP    | VID     |
         |entries   |  (4 octets)  |(1bit)|(3bits)|(12bits)|
      |(2octets)|  |(1 bit)|(3 bits)|(12 bits)|
         |(2 octets)|              |       |        |         |
      +---------+--------------+------+-------+--------+
                |<--------Repeated
         +----------+--------------+-------+--------+---------+
                    |<----------Repeated per entry---------->| entry----------->|

             Figure A.4 10: GroupID/VID filter format
   4. GroupID/MAC/VID Filter Info format
   +---------+----------+-------------+------+-----+--------+ Information Format

     +----------+----------+-------------+-------+--------+---------+
     | #of # of     | GroupID  | MAC address | PS    | PCP    | VID     |
     |entries   |(4 octets)| (6 octets)  |(1bit)|(3b )|(12bits)|
   |(2octets)|  |(1 bit)|(3 bits)|(12 bits)|
     |(2 octets)|          |             |       |        |         |
   +---------+----------+-------------+------+-----+--------+
             |<-------------Repeated
     +----------+----------+-------------+-------+--------+---------+
                |<---------------Repeated per entry------------->| entry---------------->|

           Figure A.5 11: GroupID/MAC/VID filter format Filter Information Format

   The null VID can be used in the VDP Request sent from the station to
   the external bridge. Use of the  The null VID indicates that the set of VID
   values associated with the VSI is expected to be supplied by the
   bridge.  The set of VID values is returned to the station via the VDP
   Response.  The returned VID value values can be a locally significant value. values.
   When GroupID is used, it is equivalent to the VN ID in NVO3.  GroupID
   will be provided by the station to the bridge.  The bridge maps
   GroupID to a locally significant VLAN ID.

   The VSI ID in the VDP association TLV that identify identifies a VM can be in
   one of the following format: IPV4 formats: IPv4 address, IPV6 IPv6 address, MAC
   address, UUID Universally Unique Identifier (UUID) [RFC4122], or locally
   defined.

8.

Acknowledgements

   This document was initiated based on the merger of the drafts draft-
   kreeger-nvo3-hypervisor-nve-cp, draft-gu-nvo3-tes-nve-mechanism, following
   documents: [NVO3-HYPERVISOR-NVE-CP], [NVO3-TES-NVE], and
   draft-kompella-nvo3-server2nve.
   [NVO3-VM-NVE].  Thanks to all the co-authors coauthors and contributing members
   of those drafts. documents.

   The authors would like to specially thank Lucy Yong and Jon Hudson
   for their generous help in improving this document.

Authors' Addresses

   Yizhou Li
   Huawei Technologies
   101 Software Avenue, Avenue
   Nanjing  210012
   China

   Phone: +86-25-56625409
   EMail:
   Email: liyizhou@huawei.com

   Donald Eastlake 3rd
   Huawei R&D USA
   155 Beaver Street
   Milford, MA  01757 USA
   United States of America

   Phone: +1-508-333-2270
   EMail:
   Email: d3e3e3@gmail.com

   Lawrence Kreeger
   Arrcus, Inc Inc.

   Email: lkreeger@gmail.com
   Thomas Narten
   IBM

   Email: narten@us.ibm.com

   David Black
   Dell EMC
   176 South Street, Street
   Hopkinton, MA  01748 USA
   United States of America

   Email: david.black@dell.com