wdiff rfc9135.alt-original rfc9135.txt

BESS WorkGroup

Internet Engineering Task Force (IETF) A. Sajassi
Internet-Draft
Request for Comments: 9135 S. Salam
Intended status:
Category: Standards Track S. Thoria
Expires: January 27, 2022
ISSN: 2070-1721 Cisco Systems
J. Drake
Juniper
J. Rabadan
Nokia
July 26,
October 2021

Integrated Routing and Bridging in EVPN
draft-ietf-bess-evpn-inter-subnet-forwarding-15 Ethernet VPN (EVPN)

Abstract

Ethernet VPN (EVPN) provides an extensible and flexible multi-homing multihoming
VPN solution over an MPLS/IP network for intra-subnet connectivity
among Tenant Systems and End Devices end devices that can be physical or virtual.
However, there are scenarios for which there is a need for a dynamic
and efficient inter-subnet connectivity among these Tenant Systems
and End Devices end devices while maintaining the multi-homing multihoming capabilities of
EVPN. This document describes an Integrated Routing and Bridging
(IRB) solution based on EVPN to address such requirements.

Status of This Memo

This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents an Internet Standards Track document.

This document is a product of the Internet Engineering Task Force
(IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list It represents the consensus of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid the IETF community. It has
received public review and has been approved for a maximum publication by the
Internet Engineering Steering Group (IESG). Further information on
Internet Standards is available in Section 2 of RFC 7841.

Information about the current status of six months this document, any errata,
and how to provide feedback on it may be updated, replaced, or obsoleted by other documents obtained at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on January 27, 2022.
https://www.rfc-editor.org/info/rfc9135.

This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.

Table of Contents

1. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 3
2. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 5
2. Terminology
2.1. Requirements Language
3. EVPN PE Model for IRB Operation . . . . . . . . . . . . . . . 6
4. Symmetric and Asymmetric IRB . . . . . . . . . . . . . . . . 7
4.1. IRB Interface and its Its MAC and IP addresses . . . . . . . 10 Addresses
4.2. Operational Considerations . . . . . . . . . . . . . . . 12
5. Symmetric IRB Procedures . . . . . . . . . . . . . . . . . . 13
5.1. Control Plane - Advertising PE . . . . . . . . . . . . . 13
5.2. Control Plane - Receiving PE . . . . . . . . . . . . . . 14
5.3. Subnet route advertisement . . . . . . . . . . . . . . . 15 Route Advertisement
5.4. Data Plane - Ingress PE . . . . . . . . . . . . . . . . . 16
5.5. Data Plane - Egress PE . . . . . . . . . . . . . . . . . 17
6. Asymmetric IRB Procedures . . . . . . . . . . . . . . . . . . 17
6.1. Control Plane - Advertising PE . . . . . . . . . . . . . 17
6.2. Control Plane - Receiving PE . . . . . . . . . . . . . . 18
6.3. Data Plane - Ingress PE . . . . . . . . . . . . . . . . . 19
6.4. Data Plane - Egress PE . . . . . . . . . . . . . . . . . 19
7. Mobility Procedure . . . . . . . . . . . . . . . . . . . . . 20
7.1. Initiating a gratutious Gratuitous ARP upon a Move . . . . . . . . . 21
7.2. Sending Data Traffic without an ARP Request . . . . . . . 22
7.3. Silent Host . . . . . . . . . . . . . . . . . . . . . . . 24
8. BGP Encoding . . . . . . . . . . . . . . . . . . . . . . . . 24
8.1. EVPN Router's MAC Extended Community . . . . . . . . . . . . . 25
9. Operational Models for Symmetric Inter-Subnet Forwarding . . 25
9.1. IRB forwarding Forwarding on NVEs for Tenant Systems . . . . . . . . 25
9.1.1. Control Plane Operation . . . . . . . . . . . . . . . 27
9.1.2. Data Plane Operation . . . . . . . . . . . . . . . . 28
9.2. IRB forwarding Forwarding on NVEs for Subnets behind Tenant Systems 30
9.2.1. Control Plane Operation . . . . . . . . . . . . . . . 31
9.2.2. Data Plane Operation . . . . . . . . . . . . . . . . 32
10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 33
11. Security Considerations . . . . . . . . . . . . . . . . . . . 33
12.
11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 34
13.
12. References . . . . . . . . . . . . . . . . . . . . . . . . . 34
13.1.
12.1. Normative References . . . . . . . . . . . . . . . . . . 34
13.2.
12.2. Informative References . . . . . . . . . . . . . . . . . 35
Acknowledgements
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 36

1. Introduction

EVPN [RFC7432] provides an extensible and flexible multi-homing multihoming VPN
solution over an MPLS/IP network for intra-subnet connectivity among
Tenant Systems (TSes) (TSs) and End Devices end devices that can be physical or
virtual; virtual,
where an IP subnet is represented by an EVPN Instance instance (EVI) for a
VLAN-based service or by an (EVI, VLAN) association for a VLAN-aware
bundle service. However, there are scenarios for which there is a
need for a dynamic and efficient inter-subnet connectivity among
these Tenant Systems and End Devices end devices while maintaining the multi-homing
multihoming capabilities of EVPN. This document describes an
Integrated Routing and Bridging (IRB) solution based on EVPN to
address such requirements.

The inter-subnet

Inter-subnet communication is traditionally achieved at typically performed by centralized L3 Gateway (L3GW) devices where
Layer 3 (L3) gateway (GW) devices, which enforce all the inter-subnet
forwarding is performed
communication policies and perform all the inter-subnet communication
policies are enforced. forwarding. When
two TSes TSs belonging to two different subnets connected to the same PE
Provider Edge (PE) wanted to communicate with each other, their
traffic needed to be backhauled from the PE all the way to the
centralized gateway where inter-subnet switching is performed and
then sent back to the PE. For today's large multi-tenant data center, Data Center
(DC), this scheme is very inefficient and sometimes impractical.

In order to overcome the drawback of the centralized layer-3 L3 GW approach,
IRB functionality is needed on the PEs (also referred to as EVPN NVEs)
Network Virtualization Edges (NVEs)) attached to TSes TSs in order to
avoid inefficient forwarding of tenant traffic (i.e., avoid back-hauling
backhauling and hair-pinning). hair pinning). When a PE with IRB capability
receives tenant traffic over an Attachment Circuit (AC), it can not cannot
only locally bridge the tenant intra-subnet traffic but also can locally
route the tenant inter-subnet traffic on a packet by packet basis packet-by-packet basis,
thus meeting the requirements for both intra intra- and inter-subnet
forwarding and avoiding non-optimal traffic forwarding associated
with a centralized layer-3 L3 GW approach.

Some TSes TSs run non-IP protocols in conjunction with their IP traffic.
Therefore, it is important to handle both kinds of traffic optimally
-
-- e.g., to bridge non-IP and intra-subnet traffic and to route inter-
subnet
inter-subnet IP traffic. Therefore, the solution needs to meet the
following requirements:

R1: The solution must provide each tenant with IP routing of its
inter-subnet traffic and Ethernet bridging of its intra-subnet
traffic and non-routable traffic, where non-routable traffic
refers
both to both non-IP traffic and IP traffic whose version differs
from the IP version configured in the IP-VRF. IP Virtual Routing and
Forwarding (IP-VRF). For example, if an IP-VRF in a an NVE is
configured for IPv6 and that NVE receives IPv4 traffic on the
corresponding VLAN, then the IPv4 traffic is treated as non-routable non-
routable traffic.

R2: The solution must allow IP routing of inter-subnet traffic to be
disabled on a per-VLAN basis on those PEs that are backhauling
that traffic to another PE for routing.

2. Terminology

AC: Attachment Circuit

ARP: Address Resolution Protocol

ARP table: Table: A logical view of a forwarding table on a PE that
maintains an IP to a MAC binding entry on an IP interface
for both IPv4 and IPv6. These entries are learned through
ARP/ND or through EVPN.

BD: Broadcast Domain: Domain. As per [RFC7432], an EVI consists of a
single BD or multiple broadcast domains. BDs. In the case of VLAN-bundle and VLAN-
based
VLAN-based service models (see [RFC7432]), a broadcast domain BD is
equivalent to an EVI. In the case of a VLAN-aware bundle
service model, an EVI contains multiple broadcast domains. BDs. Also, in this
document, broadcast domain "BD" and subnet "subnet" are equivalent terms terms, and
wherever "subnet" is used, it means "IP subnet"

Broadcast Domain subnet".

BD Route Target: refers Refers to the Broadcast Domain
assigned broadcast-domain-assigned Route
Target [RFC4364]. In the case of a VLAN-aware bundle
service model, all the broadcast domain BD instances in the MAC-VRF share
the same Route Target Target.

BT: Bridge Table: Table. The instantiation of a broadcast domain BD in a MAC-VRF, as
per [RFC7432].

CE: Customer Edge

DA: Destination Address

Ethernet NVO tunnel: refers Tunnel: Refers to Network Virtualization Overlay
tunnels with an Ethernet payload payload, as specified for VxLAN VXLAN in
[RFC7348] and for NVGRE in [RFC7637].

EVI: EVPN Instance spanning the NVE/PE devices that are
participating on that EVPN, as per [RFC7432].

EVPN: Ethernet Virtual Private Networks, VPN, as per [RFC7432].

IP NVO tunnel: it refers Tunnel: Refers to Network Virtualization Overlay tunnels with
IP payload (no MAC header in the payload) as specified for GPE
Generic Protocol Extension (GPE) in [I-D.ietf-nvo3-vxlan-gpe]. [VXLAN-GPE].

IP-VRF: A Virtual Routing and Forwarding table for IP routes on an
NVE/PE. The IP routes could be populated by EVPN and IP-VPN IP-
VPN address families. An IP-VRF is also an instantiation
of a layer Layer 3 VPN in an NVE/PE.

IRB: Integrated Routing and Bridging interface. It connects an IP-
VRF
IP-VRF to a broadcast domain BD (or subnet).

MAC: Media Access Control

MAC-VRF: A Virtual Routing and Forwarding table for Media Access
Control (MAC) MAC addresses on
an NVE/PE, as per [RFC7432]. A MAC-VRF is also an
instantiation of an EVI in an NVE/PE.

ND: Neighbor Discovery Protocol

NVE: Network Virtualization Edge

NVGRE: Network Virtualization Using Generic Routing Encapsulation,
[RFC7637]
as per [RFC7637].

NVO: Network Virtualization Overlays Overlay

PE: Provider Edge

RT-2: EVPN route type Route Type 2, i.e., MAC/IP Advertisement route, as
defined in [RFC7432] [RFC7432].

RT-5: EVPN route type Route Type 5, i.e., IP Prefix route. As route, as defined in
Section 3 of [I-D.ietf-bess-evpn-prefix-advertisement] [RFC9136].

SA: Source Address

TS: Tenant System

VA: Virtual Appliance

VNI: Virtual Network Identifier. As in [RFC8365], the term is
used as a representation of a 24-bit NVO instance
identifier, with the understanding that VNI "VNI" will refer to
a VXLAN Network Identifier in VXLAN, or a Virtual Subnet
Identifier in NVGRE, etc. etc., unless it is stated otherwise.

VTEP: VXLAN Termination End Point, as in per [RFC7348].

VXLAN: Virtual Extensible LAN, eXtensible Local Area Network, as in per [RFC7348].

This document also assumes familiarity with the terminology of
[RFC7365], [RFC7432], [RFC8365] and [RFC7365].

2.1 [RFC8365].

2.1. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in RFC
2119
BCP 14 [RFC2119] and RFC 8174 [RFC8174] when, and only when, they appear in all
capitals, as shown here.

3. EVPN PE Model for IRB Operation

Since this document discusses IRB operation in relationship to EVPN
MAC-VRF, IP-VRF, EVI, Broadcast Domain, Bridge Table, BD, bridge table, and IRB interfaces, it is
important to understand the relationship between these components.
Therefore, the following PE model is illustrated below to a) describe these
components and b) illustrate the relationship among them.

+-------------------------------------------------------------+
| |
| +------------------+ IRB PE |
| Attachment | +------------------+ |
| Circuit(AC1) | | +----------+ | MPLS/NVO tnl
----------------------*Bridge | | +-----
| | | |Table(BT1)| | +-----------+ / \ \
| | | | *---------* |<--> |Eth|
| | | | VLAN x | |IRB1| | \ / /
| | | +----------+ | | | +-----
| | | ... | | IP-VRF1 | |
| | | +----------+ | | RD2/RT2 |MPLS/NVO tnl
| | | |Bridge | | | | +-----
| | | |Table(BT2)| |IRB2| | / \ \
| | | | *---------* |<--> |IP |
----------------------* VLAN y | | +-----------+ \ / /
| AC2 | | +----------+ | +-----
| | | MAC-VRF1 | |
| +-+ RD1/RT1 | |
| +------------------+ |
| |
| |
+-------------------------------------------------------------+

Figure 1: EVPN IRB PE Model

A tenant needing IRB services on a PE, PE requires an IP Virtual Routing
and Forwarding IP-VRF table (IP-VRF) along
with one or more MAC Virtual
Routing and Forwarding tables (MAC-VRFs). MAC-VRF tables. An IP-VRF, as defined in [RFC4364],
is the instantiation of an IPVPN IP-VPN instance in a PE. A MAC-
VRF, MAC-VRF, as
defined in [RFC7432], is the instantiation of an EVI (EVPN
Instance) in a PE. A MAC-VRF MAC-
VRF consists of one or more bridge tables, where each bridge table
corresponds to a VLAN (broadcast domain). If service interfaces for
an EVPN PE are configured in VLAN- Based VLAN-based mode (i.e., section Section 6.1 of RFC7432),
[RFC7432]), then there is only a single bridge table per MAC-VRF (per
EVI) - -- i.e., there is only one tenant VLAN per EVI. However, if
service interfaces for an EVPN PE are configured in
VLAN-Aware Bundle VLAN-aware bundle
mode (i.e., section Section 6.3 of RFC7432), [RFC7432]), then there are several bridge
tables per MAC-VRF (per EVI) - -- i.e., there are several tenant VLANs
per EVI.

Each bridge table is connected to an IP-VRF via an L3 interface
called IRB interface. an "IRB interface". Since a single tenant subnet is typically
(and in this document) represented by a VLAN (and thus supported by a
single bridge table), for a given tenant tenant, there are as many bridge
tables as there are subnets and thus subnets. Thus, there are also as many IRB
interfaces between the tenant IP-VRF and the associated bridge tables
as shown in the PE model above.

IP-VRF is identified by its corresponding route target Route Target and route
distinguisher Route
Distinguisher, and MAC-VRF is also identified by its corresponding
route target
Route Target and route distinguisher. Route Distinguisher. If operating in EVPN VLAN-
Based
based mode, then a receiving PE that receives an EVPN route with MAC-
VRF route target a
MAC-VRF Route Target can identify the corresponding bridge table;
however, if operating in EVPN VLAN-Aware Bundle VLAN-aware bundle mode, then the
receiving PE needs both the MAC-VRF route target Route Target and VLAN ID in order
to identify the corresponding bridge table.

4. Symmetric and Asymmetric IRB

This document defines and describes two types of IRB solutions -
namely --
namely, symmetric and asymmetric IRB. The description of symmetric
and asymmetric IRB procedures relating to data path operations and
tables in this document is a logical view of data path lookups and
related tables. Actual implementations, while following this logical
view, may not strictly adhere to it for performance tradeoffs. trade-offs.
Specifically,

* References to an ARP table in the context of asymmetric IRB is a
logical view of a forwarding table that maintains an IP to MAC IP-to-MAC
binding entry on a layer Layer 3 interface for both IPv4 and IPv6.
These entries are not subject to ARP or ND protocol. protocols. For IP to IP-to-
MAC bindings learnt learned via EVPN, an implementation may choose to
import these bindings directly to the respective forwarding table
(such as an adjacency/next-hop table) as opposed to importing them
to ARP or ND protocol tables.

* References to a host IP lookup followed by a host MAC lookup in
the context of asymmetric IRB MAY be collapsed into a single IP
lookup in a hardware implementation.

In symmetric IRB IRB, as its name implies, the lookup operation is
symmetric at both the ingress and egress PEs - -- i.e., both ingress
and egress PEs perform lookups on both MAC and IP addresses. The
ingress PE performs a MAC lookup followed by an IP lookup lookup, and the
egress PE performs an IP lookup followed by a MAC lookup lookup, as depicted
in the following figure.

Ingress PE Egress PE
+-------------------+ +------------------+
| | | |
| +-> IP-VRF ----|---->---|-----> IP-VRF -+ |
| | | | | |
| BT1 BT2 | | BT3 BT2 |
| | | | | |
| ^ | | v |
| | | | | |
+-------------------+ +------------------+
^ |
| |
TS1->-+ +->-TS2

Figure 2: Symmetric IRB

In symmetric IRB IRB, as shown in figure-2, Figure 2, the inter-subnet forwarding
between two PEs is done between their associated IP-VRFs. Therefore,
the tunnel connecting these IP-VRFs can be either an IP-only tunnel
(e.g., in the case of MPLS or GPE encapsulation) or an Ethernet NVO
tunnel (e.g., in the case of VxLAN VXLAN encapsulation). If it is an
Ethernet NVO tunnel, the TS1's IP packet is encapsulated in an
Ethernet header consisting of ingress and egress PEs PE MAC addresses - --
i.e., there is no need for the ingress PE to use the destination
TS2's MAC address. Therefore, in symmetric IRB, there is no need for
the ingress PE to maintain ARP entries for the association of the
destination TS2's IP and MAC addresses
association in its ARP table. Each PE
participating in symmetric IRB only maintains ARP entries for locally
connected hosts and maintains
MAC-VRFs/bridge tables MAC-VRFs/BTs for only locally configured subnets.

In asymmetric IRB, the lookup operation is asymmetric and the ingress
PE performs three lookups; lookups, whereas the egress PE performs a single
lookup - -- i.e., the ingress PE performs a MAC lookup, followed by an
IP lookup, followed by a MAC lookup again; whereas, the again. The egress PE performs
just a single MAC lookup as depicted in figure Figure 3 below.

Ingress PE Egress PE
+-------------------+ +------------------+
| | | |
| +-> IP-VRF -> | | IP-VRF |
| | | | | |
| BT1 BT2 | | BT3 BT2 |
| | | | | | | |
| | +--|--->----|--------------+ | |
| | | | v |
+-------------------+ +----------------|-+
^ |
| |
TS1->-+ +->-TS2

Figure 3: Asymmetric IRB

In asymmetric IRB IRB, as shown in figure-3, Figure 3, the inter-subnet forwarding
between two PEs is done between their associated MAC-VRFs/bridge
tables. MAC-VRFs/BTs.
Therefore, the MPLS or NVO tunnel used for inter-subnet forwarding
MUST be of type Ethernet. Since only MAC lookup is performed at the
egress PE (e.g., no IP lookup), the TS1's IP packets need to be
encapsulated with the destination TS2's MAC address. In order for
the ingress PE to perform such encapsulation, it needs to maintain
TS2's IP and MAC address association in its ARP table. Furthermore,
it needs to maintain destination TS2's MAC address in the
corresponding bridge table even though it may not have any TSes TSs of the
corresponding subnet locally attached. In other words, each PE
participating in asymmetric IRB MUST maintain ARP entries for remote
hosts (hosts connected to other PEs) as well as maintain MAC-
VRFs/bridge tables MAC-VRFs/BTs
and IRB interfaces for ALL subnets in an IP VRF IP-VRF, including subnets
that may not be locally attached. Therefore, careful consideration
of the PE scale aspects for its ARP table size, its IRB interfaces,
and the number and size of its bridge tables should be given for the
application of asymmetric IRB.

It should be noted that whenever a PE performs a host IP lookup for a
packet that is routed, the IPv4 TTL Time To Live (TTL) or IPv6 hop limit
for that packet is decremented by one one, and if it reaches zero, the
packet is discarded. In the case of symmetric IRB, the TTL/hop TTL / hop
limit is decremented by both ingress and egress PEs (once by each); whereas, each),
whereas in the case of asymmetric IRB, the TTL/hop TTL / hop limit is
decremented only once by the ingress PE.

The following sections define the control and data plane procedures
for symmetric and asymmetric IRB on ingress and egress PEs. The
following figure is used to describe these procedures, showing a
single IP-VRF and a number of broadcast domains BDs on each PE for a given tenant. I.e.,
That is, an IP-VRF connects one or more EVIs, and each EVI contains
one MAC-VRF, MAC-VRF; each MAC VRF consists of one or more bridge tables, one
per broadcast domain, BD; and a PE has an associated IRB interface for each broadcast domain. BD.

PE 1 +---------+
+-------------+ | |
TS1-----| MACx| | | PE2
(IP1/M1)
(M1/IP1) |(BT1) | | | +-------------+
TS5-----| \ | | MPLS/ | |MACy (BT3) |-----TS3
(IP5/M5)
(M5/IP5) |IPx/Mx \ | | VxLAN/ VXLAN/ | | / | (IP3/M3) (M3/IP3)
| (IP-VRF1)|----| NVGRE |---|(IP-VRF1) |
| / | | | | \ |
TS2-----|(BT2) / | | | | (BT1) |-----TS4
(IP2/M2)
(M2/IP2) | | | | | | (IP4/M4) (M4/IP4)
+-------------+ | | +-------------+
| |
+---------+

Figure 4: IRB forwarding Forwarding

4.1. IRB Interface and its Its MAC and IP addresses Addresses

To support inter-subnet forwarding on a PE, the PE acts as an IP
Default Gateway
default gateway from the perspective of the attached Tenant Systems
where default gateway MAC and IP addresses are configured on each IRB
interface associated with its subnet and falls fall into one of the
following two options:

1. All the PEs for a given tenant subnet use the same anycast
default gateway IP and MAC addresses. On each PE, this these default
gateway IP and MAC addresses correspond to the IRB interface
connecting the bridge table associated with the tenant's VLAN to
the corresponding tenant's IP-VRF.

2. Each PE for a given tenant subnet uses the same anycast default
gateway IP address but its own MAC address. These MAC addresses
are aliased to the same anycast default gateway IP address
through the use of the Default Gateway extended community as
specified in [RFC7432], which is carried in the EVPN MAC/IP
Advertisement routes. On each PE, this default gateway IP
address
address, along with its associated MAC addresses addresses, correspond to
the IRB interface connecting the bridge table associated with the
tenant's VLAN to the corresponding tenant's IP-VRF.

It is worth noting that if the applications that are running on the
TSes
TSs are employing or relying on any form of MAC security, then the
first option (i.e. (i.e., using an anycast MAC address) should be used to
ensure that the applications receive traffic from the same IRB
interface MAC address that to which they are sending to. sending. If the second
option is used, then the IRB interface MAC address MUST be the one
used in the initial ARP reply or ND Neighbor Advertisement (NA)for (NA) for
that TS.

Although both of these options are applicable to both symmetric and
asymmetric IRB, the option-1 option 1 is recommended because of the ease of
anycast MAC address provisioning on not only the IRB interface
associated with a given subnet across all the PEs corresponding to
that VLAN but also on all IRB interfaces associated with all the
tenant's subnets across all the PEs corresponding to all the VLANs
for that tenant. Furthermore, it simplifies the operation as there
is no need for Default Gateway extended community advertisement and
its associated MAC aliasing procedure. Yet another advantage is that
following host mobility, the host does not need to refresh the
default GW ARP/ND entry.

If option-1 option 1 is used, an implementation MAY choose to auto-derive the
anycast MAC address. If auto-derivation is used, the anycast MAC
MUST be auto-derived out of the following ranges (which are defined
in [RFC5798]):

* Anycast IPv4 IRB case: 00-00-5E-00-01-{VRID}

* Anycast IPv6 IRB case: 00-00-5E-00-02-{VRID}

Where the last octet is generated based on a configurable Virtual
Router ID (VRID, range 1-255)). (VRID) (range 1-255). If not explicitly configured, the
default value for the VRID octet is '1'. Auto-derivation of the
anycast MAC can only be used if there is certainty that the auto-
derived MAC does not collide with any customer MAC address.

In addition to IP anycast addresses, IRB interfaces can be configured
with non-anycast IP addresses for the purpose of OAM (such as sending
a traceroute/ping to these interfaces) for both symmetric and
asymmetric IRB. These IP addresses need to be distributed as VPN
routes when PEs operate in symmetric IRB mode. However, they don't
need to be distributed if the PEs are operating in asymmetric IRB
mode as the non-anycast IP addresses are configured along with their
individual MACs MACs, and they get distributed via the EVPN route type-2 type 2
advertisement.

For option-1, option 1 -- irrespective of using whether only the anycast MAC address
or both anycast and non-anycast MAC addresses (where the latter one
is used for the purpose of OAM) are used on the same IRB, IRB -- when a TS
sends an ARP request or ND Neighbor Solicitation (NS) to the PE that to
which it is attached
to, attached, the request is sent for the anycast IP address
of the IRB interface associated with the TS's subnet and then the subnet. The reply will
use an anycast MAC address (in both Source the source MAC in the Ethernet
header and
Sender sender hardware address in the payload). For example, in figure
Figure 4, TS1 is configured with the anycast IPx address as its
default gateway IP address and thus address; thus, when it sends an ARP request for
IPx (anycast IP address of the IRB interface for BT1), the PE1 sends
an ARP reply with the MACx MACx, which is the anycast MAC address of that
IRB interface. Traffic routed from IP-VRF1 to TS1 uses the anycast
MAC address as the source MAC address.

4.2. Operational Considerations

Symmetric and Asymmetric asymmetric IRB modes may coexist in the same network,
and an ingress PE that supports both forwarding modes for a given
tenant can interwork with egress PEs that support either IRB mode.
The egress PE will indicate the desired forwarding mode for a given
host based on the presence of the Label2 field and the IP-VRF route-
target Route
Target in the EVPN MAC/IP Advertisement route. If the Label2 field
of the received MAC/IP Advertisement route for host H1 is non-zero,
and one of its route-targets Route Targets identifies the IP-VRF, the ingress PE
will use Symmetric symmetric IRB mode when forwarding packets destined to H1.
If the Label2 field is zero and the MAC/IP Advertisement route for H1
does not carry any route-target Route Target that identifies the IP-VRF, the
ingress PE will use Asymmetric asymmetric mode when forwarding traffic to H1.

As an example that illustrates the previous statement, suppose PE1
and PE2 need to forward packets from TS2 to TS4 in the example of Figure 4. Since
both PEs are attached to the bridge table of the destination host, Symmetric
symmetric and Asymmetric asymmetric IRB modes are both possible as long as the
ingress PE, PE1, supports both modes. The forwarding mode will
depend on the mode configured in the egress PE, PE2. That is:

1. If PE2 is configured for Symmetric symmetric IRB mode, PE2 will advertise
TS4 MAC/IP addresses in a MAC/IP Advertisement route with a non-
zero Label2 field, e.g., Label2=Lx, Label2 = Lx, and a route-target Route Target that
identifies IP-VRF1 in PE1. IP4 will be installed in PE1's IP-
VRF1,
VRF1; TS4's ARP and MAC information will also be installed in
PE1's IRB interface ARP table and BT1 BT1, respectively. When a
packet from TS2 destined to TS4 is looked up in PE1's IP-VRF
route-table,
route table, a longest prefix match lookup will find IP4 in the
IP-VRF, and PE1 will forward using the Symmetric symmetric IRB mode and
Label Lx.

2. However, if PE2 is configured for Asymmetric asymmetric IRB mode, PE2 will
advertise TS4 MAC/IP information in a MAC/IP Advertisement route
with a zero Label2 field and no route-target Route Target identifying IP-VRF1.
In this case, PE2 will install TS4 information in its ARP table
and BT1. When a packet from TS2 to TS4 arrives at PE1, a longest
prefix match on IP-VRF1's route-table route table will yield the local IRB
interface to BT1, where a subsequent ARP and bridge table lookup
will provide the information for an Asymmetric asymmetric forwarding mode to
PE2.

Refer to [I-D.ietf-bess-evpn-modes-interop] [EVPN] for more information about interoperability between Symmetric
symmetric and Asymmetric asymmetric forwarding modes.

The choice between Symmetric symmetric or Asymmetric asymmetric mode is based on the
operator's preference preference, and it is a trade-off between scale (better (which is
better in the Symmetric symmetric IRB mode) and control plane simplicity (Asymmetric
(asymmetric IRB mode simplifies the control plane). In cases where a
tenant has hosts for every subnet attached to all (or most) most of) the
PEs, the ARP and MAC entries need to be learned by all PEs anyway and therefore anyway;
therefore, the
Asymmetric asymmetric IRB mode simplifies the forwarding model
and saves space in the IP-VRF route-table, route table, since host routes are not
installed in the
route-table. route table. However, if the tenant does not need
to stretch subnets (broadcast domains) to multiple PEs and inter-subnet-forwarding inter-
subnet forwarding is needed, the Symmetric symmetric IRB model will save ARP
and bridge table space in all the PEs (in comparison with the Asymmetric
asymmetric IRB model).

5. Symmetric IRB Procedures

5.1. Control Plane - Advertising PE

When a PE (e.g., PE1 in figure Figure 4 above) learns the MAC and IP address
of a TS (e.g., via an ARP request or Neighbor Solicitation), it adds
the MAC address to the corresponding MAC-VRF/bridge table MAC-VRF/BT of that tenant's
subnet and adds the IP address to the IP-VRF for that tenant.
Furthermore, it adds this TS's MAC and IP address association to its
ARP table or NDP Neighbor Discovery Protocol (NDP) cache. It then builds
an EVPN MAC/IP Advertisement route (type 2) as follows and advertises
it to other PEs participating in that tenant's VPN.

* The Length field of the BGP EVPN NLRI Network Layer Reachability
Information (NLRI) for an EVPN MAC/IP Advertisement route MUST be
either 40 (if the IPv4 address is carried) or 52 (if the IPv6
address is carried).

* The Route Distinguisher (RD), Ethernet Segment Identifier,
Ethernet Tag ID, MAC Address Length, MAC Address, IP Address
Length, IP Address, and MPLS Label1 fields MUST be set per
[RFC7432] and [RFC8365].

* The MPLS Label2 field is set to either an MPLS label or a VNI
corresponding to the tenant's IP-VRF. In the case of an MPLS
label, this field is encoded as 3 octets, where the high-order 20
bits contain the label value.

Just as in [RFC7432], the RD, Ethernet Tag ID, MAC Address Length,
MAC Address, IP Address Length, and IP Address fields are part of the
route key used by BGP to compare routes. The rest of the fields are
not part of the route key.

This route is advertised along with the following two extended
communities:

1. Encapsulation Extended Community

2. EVPN Router's MAC Extended Community

This route is advertised with one or more Encapsulation extended
communities Extended
Communities [RFC9012], one for each encapsulation type supported by
the advertising PE. If one or more encapsulation types require an
Ethernet frame, a single EVPN Router's MAC extended community, section
8.1, Extended Community
(Section 8.1) is also advertised. This extended community specifies
the MAC address to be used as the inner destination MAC address in an
Ethernet frame sent to the advertising PE.

This route MUST be advertised with two route targets, Route Targets, one
corresponding to the MAC-VRF of the tenant's subnet and another
corresponding to the tenant's IP-VRF.

5.2. Control Plane - Receiving PE

When a PE (e.g., PE2 in figure Figure 4 above) receives this EVPN MAC/IP
Advertisement route, it performs the following:

* The MAC-VRF route target Route Target and Ethernet Tag, if the latter is non-
zero, are used to identify the correct MAC-VRF and bridge table table,
and if they are found found, the MAC address is imported. The IP-VRF
route target
Route Target is used to identify the correct IP-VRF IP-VRF, and if it is
found
found, the IP address is imported.

If the MPLS label2 Label2 field is non-zero, it means that this route is to
be used for symmetric IRB IRB, and the MPLS label2 value is to be used
when sending a packet for this IP address to the advertising PE.

If the receiving PE supports asymmetric IRB mode and receives this
route with both the MAC-VRF and IP-
VRF route targets IP-VRF Route Targets but the MAC/IP
Advertisement route does not include
MPLS label2 field and if the receiving PE supports asymmetric IRB
mode, MPLS Label2 field, then the
receiving PE installs the MAC address in the corresponding MAC-VRF
and the (IP, MAC) association in the ARP table for that tenant
(identified by the corresponding IP-VRF route target). Route Target).

If the receiving PE receives this route with both the MAC-VRF and IP-
VRF route targets Route Targets, and if the receiving PE does not support either
asymmetric or symmetric IRB modes, then if it modes but has the corresponding MAC-VRF,
then it only imports the MAC address.

If the receiving PE receives this route with both the MAC-VRF and IP-
VRF route targets Route Targets and the MAC/IP Advertisement route includes the
MPLS
label2 Label2 field but the receiving PE only supports asymmetric IRB
mode, then the receiving PE MUST ignore the MPLS label2 Label2 field and
install the MAC address in the corresponding MAC-VRF and (IP, MAC)
association in the ARP table for that tenant (identified by the
corresponding IP-VRF
route target). Route Target).

5.3. Subnet route advertisement Route Advertisement

In the case of symmetric IRB, a layer-3 Layer 3 subnet and IRB interface
corresponding to a MAC-VRF/bridge table is MAC-VRF/BT are required to be provisioned at a PE
only if that PE has locally attached hosts in that subnet. In order
to enable inter-subnet routing across PEs in a deployment where not
all subnets are provisioned at all PEs participating in an EVPN IRB
instance, PEs MUST advertise local subnet routes as EVPN RT-
5. RT-5. These
subnet routes are required for bootstrapping host (MAC,IP) (IP, MAC) learning
using gleaning procedures initiated by an inter-subnet data packet.

I.e.,

That is, if a given host's (MAC, IP) (IP, MAC) association is unknown, and an
ingress PE needs to send a packet to that host, then that ingress PE
needs to know which egress PEs are attached to the subnet in which
the host resides in order to send the packet to one of those PEs,
causing the PE receiving the packet to probe for that host. For
example, Consider consider a subnet A that is locally attached to PE1 and
subnet B that is locally attached to PE2 and to PE3. Host A in subnet
A, that which is attached to PE1 PE1, initiates a data packet destined to host
B in subnet B that B, which is attached to PE3. If host B's (MAC, IP) (IP, MAC) has
not yet been learnt either learned via either a gratuitous ARP OR via a prior gleaning
procedure, a new gleaning procedure MUST be triggered for host B's (MAC, IP)
(IP, MAC) to be learnt learned and advertised across the EVPN network.
Since host B's subnet is not local to PE1, an IP lookup for host B at
PE1 will not trigger this gleaning procedure for host B's
(MAC, IP). (IP, MAC).
Therefore, PE1 MUST learn subnet B's prefix route via EVPN RT-5
advertised from PE2 and PE3, so it can route the packet to one of the
PEs that have subnet B locally attached. Once the packet is received
at PE2 OR PE3, and the route lookup yields a glean result, an ARP
request is triggered and flooded across the layer-2 Layer 2 overlay. This
ARP request would be received and replied to by host B, resulting in
host B (MAC, IP) (IP, MAC) learning at PE3, PE3 and its advertisement across the
EVPN network. Packets from host A to host B can now be routed
directly from PE1 to PE3. Advertisement of local subnet EVPN RT-5
for an IP VRF IP-VRF MAY typically be achieved via
provisioning connected provisioning-connected
route redistribution to BGP.

5.4. Data Plane - Ingress PE

When an Ethernet frame is received by an ingress PE (e.g., PE1 in
figure
Figure 4 above), the PE uses the AC ID (e.g., VLAN ID) to identify
the associated MAC-VRF/bridge table MAC-VRF/BT, and it performs a lookup on the
destination MAC address. If the MAC address corresponds to its IRB
Interface
interface MAC address, the ingress PE deduces that the packet must be
inter-subnet routed. Hence, the ingress PE performs an IP lookup in
the associated IP-VRF table. The lookup identifies the BGP next hop
of the egress PE along with the tunnel/encapsulation type and the
associated MPLS/VNI values. The ingress PE also decrements the TTL/hop TTL /
hop limit for that packet by one one, and if it reaches zero, the ingress
PE discards the packet.

If the tunnel type is that of an MPLS or IP-only NVO tunnel, then the
TS's IP packet is sent over the tunnel without any Ethernet header.
However, if the tunnel type is that of an Ethernet NVO tunnel, then
an Ethernet header needs to be added to the TS's IP packet. The
source MAC address of this inner Ethernet header is set to the
ingress PE's router MAC address address, and the destination MAC address of
this inner Ethernet header is set to the egress PE's router MAC
address learnt learned via the EVPN Router's MAC extended community Extended Community attached
to the route. The MPLS VPN label is set to the received label2 in
the route. In the case of the Ethernet NVO tunnel type, the VNI may
be set one of two ways:

downstream mode: The VNI is set to the received label2 in the route route,
which is downstream assigned.

global mode: The VNI is set to the received label2 in the route route,
which is domain-wide assigned. assigned domain-wide. This VNI value from the received
label2 MUST be the same as the locally configured VNI for the IP IP-
VRF as all PEs in the NVO MUST be configured with the same IP VRF IP-VRF
VNI for this mode of operation. If the received label2 value does
not match the locally configured VNI value value, the route MUST NOT be used
used, and an error message SHOULD be logged.

PEs may be configured to operate in one of these two modes depending
on the administrative domain boundaries across PEs participating in
the NVO, NVO and the PE's capability to support downstream VNI mode.

In the case of NVO tunnel encapsulation, the outer source and
destination IP addresses are set to the ingress and egress PE BGP
next-hop IP addresses addresses, respectively.

5.5. Data Plane - Egress PE

When the tenant's MPLS or NVO encapsulated packet is received over an
MPLS or NVO tunnel by the egress PE, the egress PE removes the NVO
tunnel encapsulation and uses the VPN MPLS label (for MPLS
encapsulation) or VNI (for NVO encapsulation) to identify the IP-VRF
in which IP lookup needs to be performed. If the VPN MPLS label or
VNI identifies a
MAC- VRF MAC-VRF instead of an IP-VRF, then the procedures in section
Section 6.4 for asymmetric IRB are executed.

The lookup in the IP-VRF identifies a local adjacency to the IRB
interface associated with the egress subnet's MAC-VRF/bridge table. MAC-VRF/BT. The egress
PE also decrements the TTL/hop TTL / hop limit for that packet by
one one, and if
it reaches zero, the egress PE discards the packet.

The egress PE gets the destination TS's MAC address for that TS's IP
address from its ARP table or NDP cache, it cache. It encapsulates the packet
with that destination MAC address and a source MAC address
corresponding to that IRB interface and sends the packet to its
destination subnet MAC-VRF/bridge table. MAC-VRF/BT.

The destination MAC address lookup in the MAC-VRF/bridge table MAC-VRF/BT results in the
local adjacency (e.g., local interface) over which the Ethernet frame
is sent on. sent.

6. Asymmetric IRB Procedures

6.1. Control Plane - Advertising PE

When a PE (e.g., PE1 in figure Figure 4 above) learns the MAC and IP address
of an attached TS (e.g., via an ARP request or ND Neighbor
Solicitation), it populates its MAC-VRF/bridge table, MAC-VRF/BT, IP-VRF, and ARP table or
NDP cache just as in the case for symmetric IRB. It then builds an
EVPN MAC/IP Advertisement route (type 2) as follows and advertises it
to other PEs participating in that tenant's VPN.

* The Length field of the BGP EVPN NLRI for an EVPN MAC/IP
Advertisement route MUST be either 37 (if an IPv4 address is
carried) or 49 (if an IPv6 address is carried).

o Route Distinguisher (RD),

* The RD, Ethernet Segment Identifier, Ethernet Tag ID, MAC Address
Length, MAC Address, IP Address Length, IP Address, and MPLS
Label1 fields MUST be set per [RFC7432] and [RFC8365].

* The MPLS Label2 field MUST NOT be included in this route.

This route is advertised along with the following extended community:

* Tunnel Type Extended Community

For asymmetric IRB mode, the EVPN Router's MAC extended community Extended Community is
not needed because forwarding is performed using destination TS's MAC
address
address, which is carried in this EVPN route type-2 type 2 advertisement.

This route MUST always be advertised with the MAC-VRF route target. Route Target.
It MAY also be advertised with a second route target Route Target corresponding to
the IP-VRF.

6.2. Control Plane - Receiving PE

When a PE (e.g., PE2 in figure Figure 4 above) receives this EVPN MAC/IP
Advertisement route, it performs the following:

* Using the MAC-VRF route target, Route Target, it identifies the corresponding MAC-
VRF
MAC-VRF and imports the MAC address into it. For asymmetric IRB
mode, it is assumed that all PEs participating in a tenant's VPN
are configured with all subnets (i.e., all VLANs) and
corresponding
MAC-VRFs/bridge tables MAC-VRFs/BTs even if there are no locally attached TSes
TSs for some of these subnets. The reason for this This is because the ingress PE
needs to do forwarding based on the destination TS's MAC address
and perform NVO tunnel encapsulation as a the property of a lookup
in
MAC-VRF/bridge table.

o the MAC-VRF/BT.

* If only the MAC-VRF route target Route Target is used, then the receiving PE
uses the MAC-VRF route target Route Target to identify the corresponding IP-VRF
-- i.e., many MAC-VRF route targets Route Targets map to the same IP-VRF for a
given tenant. In this case, MAC-VRF may be used by the receiving
PE to identify the corresponding IP VRF IP-VRF via the IRB interface
associated with the subnet MAC-VRF/bridge table. MAC-VRF/BT. In this case, the MAC-VRF route target
Route Target may be used by the receiving PE to identify the
corresponding IP VRF.

o IP-VRF.

* Using the MAC-VRF route target, Route Target, the receiving PE identifies the
corresponding ARP table or NDP cache for the tenant tenant, and it adds
an entry to the ARP table or NDP cache for the TS's MAC and IP
address association. It should be noted that the tenant's ARP
table or NDP cache at the receiving PE is identified by all the
MAC- VRF route targets
MAC-VRF Route Targets for that tenant.

* If the IP-VRF route target Route Target is included, it may be used to import
the route to IP-VRF. If the IP-VRF route-target Route Target is not included,
MAC-VRF is used to derive the corresponding IP-VRF for import, as
explained in the prior section. In both cases, an IP-VRF route is
installed with the TS MAC binding included in the received route.

If the receiving PE receives the MAC/IP Advertisement route with the
MPLS
label2 Label2 field but the receiving PE only supports asymmetric IRB
mode, then the receiving PE MUST ignore the MPLS label2 Label2 field and
install the MAC address in the corresponding MAC-VRF and (IP, MAC)
association in the ARP table or NDP cache for that tenant (with the
IRB interface identified by the MAC-VRF).

6.3. Data Plane - Ingress PE

When an Ethernet frame is received by an ingress PE (e.g., PE1 in
figure
Figure 4 above), the PE uses the AC ID (e.g., VLAN ID) to identify
the associated MAC-VRF/bridge table MAC-VRF/BT, and it performs a lookup on the
destination MAC address. If the MAC address corresponds to its IRB
Interface
interface MAC address, the ingress PE deduces that the packet must be
inter-subnet routed. Hence, the ingress PE performs an IP lookup in
the associated IP-VRF table. The lookup identifies a local adjacency
to the IRB interface associated with the egress subnet's MAC-VRF/
bridge table. The ingress PE also decrements the TTL/hop TTL / hop limit for
that packet by one one, and if it reaches zero, the ingress PE discards
the packet.

The ingress PE gets the destination TS's MAC address for that TS's IP
address from its ARP table or NDP cache, it cache. It encapsulates the packet
with that destination MAC address and a source MAC address
corresponding to that IRB interface and sends the packet to its
destination subnet MAC-VRF/bridge table. MAC-VRF/BT.

The destination MAC address lookup in the MAC-VRF/bridge table MAC-VRF/BT results in a BGP next hop
next-hop address of the egress PE along with label1 (L2 VPN MPLS
label or VNI). The ingress PE encapsulates the packet using the
Ethernet NVO tunnel of the choice (e.g., VxLAN VXLAN or NVGRE) and sends
the packet to the egress PE. Because the packet forwarding is
between the ingress PE's MAC-VRF/bridge table MAC-VRF/BT and the egress PE's MAC-VRF/
bridge table, the packet encapsulation procedures follow that of
[RFC7432] for MPLS and [RFC8365] for VxLAN VXLAN encapsulations.

6.4. Data Plane - Egress PE

When a tenant's Ethernet frame is received over an NVO tunnel by the
egress PE, the egress PE removes the NVO tunnel encapsulation and
uses the VPN MPLS label (for MPLS encapsulation) or VNI (for NVO
encapsulation) to identify the MAC-VRF/bridge table MAC-VRF/BT in which the MAC lookup
needs to be performed.

The MAC lookup results in a local adjacency (e.g., local interface)
over which the packet needs to get sent.

Note that the forwarding behavior on the egress PE is the same as the
EVPN intra-subnet forwarding described in [RFC7432] for MPLS and
[RFC8365] for NVO networks. In other words, all the packet
processing associated with the inter-subnet forwarding semantics is
confined to the ingress PE for asymmetric IRB mode.

It should also be noted that [RFC7432] provides a different level of
granularity for the EVPN label. Besides identifying the bridge
domain table, it can be used to identify the egress interface or a
destination MAC address on that interface. If an EVPN label is used
for an egress interface or individual MAC address identification,
then no MAC lookup is needed in the egress PE for MPLS encapsulation encapsulation,
and the packet can be directly forwarded to the egress interface just
based on the EVPN label lookup.

7. Mobility Procedure

When a TS moves from one NVE (aka source NVE) to another NVE (aka
target NVE), it is important that the MAC mobility Mobility procedures are be
properly executed and the corresponding MAC-VRF and IP-VRF tables on
all participating NVEs are be updated. [RFC7432] describes the MAC
mobility
Mobility procedures for L2-only services for both single-homed TS and
multi-homed
multihomed TS. This section describes the incremental procedures and
BGP Extended Communities needed to handle the MAC mobility Mobility for IRB.
In order to place the emphasis on the differences between L2-only and
IRB use cases, the incremental procedure is described for
single-homed a single-
homed TS with the expectation that the additional steps needed for multi-homed TS, a
multihomed TS can be extended per section Section 15 of [RFC7432]. This
section describes mobility procedures for both symmetric and
asymmetric IRB. Although the language used in this section is for
IPv4 ARP, it equally applies to IPv6 ND.

When a TS moves from a source NVE to a target NVE, it can behave in
one of the following three ways:

1. TS initiates an ARP request upon a move to the target NVE NVE.

2. TS sends a data packet without first initiating an ARP request to
the target NVE NVE.

3. TS is a silent host and neither initiates an ARP request nor
sends any packets packets.

Depending on the expexted expected TS's behavior, an NVE needs to handle at
least the first bullet option and should be able to handle the 2nd second and the
3rd bullet.
third options. The following subsections describe the procedures for
each of them scenario where it is assumed that the MAC and IP addresses of a
TS have a one-to-one relationship (i.e., there is one IP address per
MAC address and vice versa). The procedures for host mobility
detection in the presence of a many-to-one relationship is outside
the scope of this document document, and it is covered in
[I-D.ietf-bess-evpn-irb-extended-mobility]. [EXTENDED-MOBILITY].
The many-to-one
relationship means "many-to-one relationship" refers to many host IP addresses
corresponding to a single host MAC address or many host MAC addresses
corresponding to a single IP address. It should be noted that in the
case of IPv6, a Link Local link-local IP address does not count in a many-to-one
relationship because that address is confined to a single Ethernet Segment
segment, and it is not used for host moblity mobility (i.e., by definition definition,
host mobility is between two different Ethernet Segments). segments).
Therefore, when an IPv6 host is configured with both a Global Unicast
address (or a Unique Local address) and a Link Local link-local address, for the
purpose of host mobility, it is considered with a single IP address.

7.1. Initiating a gratutious Gratuitous ARP upon a Move

In this scenario scenario, when a TS moves from a source NVE to a target NVE,
the TS initiates a gratuitous ARP upon the move to the target NVE.

The target NVE NVE, upon receiving this ARP message, updates its MAC-VRF,
IP-VRF, and ARP table with the host MAC, IP, and local adjacency
information (e.g., local interface).

Since this NVE has previously learned the same MAC and IP addresses
from the source NVE, it recognizes that there has been a MAC move move,
and it initiates MAC mobility Mobility procedures per [RFC7432] by advertising
an EVPN MAC/IP Advertisement route with both the MAC and IP addresses
filled in (per sections Sections 5.1 and 6.1) along with the MAC Mobility Extended
Community
extended community, with the sequence number incremented by one. The
target NVE also exercises the MAC duplication detection procedure in section
Section 15.1 of [RFC7432].

The source NVE NVE, upon receiving this MAC/IP Advertisement route,
realizes that the MAC has moved to the target NVE. It updates its
MAC-VRF and IP-VRF table accordingly with the adjacency information
of the target NVE. In the case of the asymmetric IRB, the source NVE
also updates its ARP table with the received adjacency information information,
and in the case of the symmetric IRB, the source NVE removes the
entry associated with the received (MAC, IP) (IP, MAC) from its local ARP
table. It then withdraws its EVPN MAC/IP Advertisement route.
Furthermore, it sends an ARP probe locally to ensure that the MAC is
gone. If an ARP response is received, the source NVE updates its ARP
entry for that (IP, MAC) and re-advertises an EVPN MAC/IP
Advertisement route for that (IP, MAC) along with the MAC Mobility
Extended Community
extended community, with the sequence number incremented by one. The
source NVE also exercises the MAC duplication detection procedure in
section
Section 15.1 of [RFC7432].

All other remote NVE devices devices, upon receiving the MAC/IP Advertisement
route with the MAC Mobility extended community community, compare the sequence
number in this advertisement with the one previously received. If
the new sequence number is greater than the old one, then they update
the MAC/IP addresses of the TS in their corresponding MAC-VRF and IP-
VRF tables to point to the target NVE. Furthermore, upon receiving
the MAC/IP withdraw for the TS from the source NVE, these remote PEs
perform the cleanups for their BGP tables.

7.2. Sending Data Traffic without an ARP Request

In this scenario scenario, when a TS moves from a source NVE to a target NVE,
the TS starts sending data traffic without first initiating an ARP
request.

The target NVE NVE, upon receiving the first data packet, learns the MAC
address of the TS in the data plane and updates its MAC-VRF table
with the MAC address and the local adjacency information (e.g., local
interface) accordingly. The target NVE realizes that there has been
a MAC move because the same MAC address has been learned remotely
from the source NVE.

If EVPN-IRB NVEs are configured to advertise MAC-only routes in
addition to MAC-and-IP EVPN routes, then the following steps are
taken:

* The target NVE NVE, upon learning this MAC address in the data plane,
updates this MAC address entry in the corresponding MAC-VRF with
the local adjacency information (e.g., local interface). It also
recognizes that this MAC has moved and initiates MAC mobility Mobility
procedures per [RFC7432] by advertising an EVPN MAC/IP
Advertisement route with only the MAC address filled in along with
the MAC Mobility Extended Community extended community, with the sequence number
incremented by one.

* The source NVE NVE, upon receiving this MAC/IP Advertisement route,
realizes that the MAC has moved to the new NVE. It updates its
MAC-VRF table with the adjacency information for that MAC address
to point to the target NVE and withdraws its EVPN MAC/IP
Advertisement route that has only the MAC address (if it has
advertised such a route previously). Furthermore, it searches for
the corresponding MAC-IP entry and sends an ARP probe for this
(MAC,IP)
(IP, MAC) pair. The ARP request message is sent both locally to
all attached TSes TSs in that subnet as well as it is sent to other NVEs
participating in that subnet subnet, including the target NVE. Note that
the PE needs to maintain a correlation between MAC and MAC-IP
route entries in the MAC-VRF to accomplish this.

* The target NVE passes the ARP request to its locally attached TSes TSs,
and when it receives the ARP response, it updates its IP-VRF and
ARP table with the host (MAC, IP) (IP, MAC) information. It also sends an
EVPN MAC/IP Advertisement route with both the MAC and IP addresses
filled in along with the MAC Mobility Extended Community extended community, with the
sequence number set to the same value as the one for the MAC-only
advertisement
Advertisement route it sent previously.

* When the source NVE receives the EVPN MAC/IP Advertisement route,
it updates its IP-VRF table with the new adjacency information
(pointing to the target NVE). In the case of the asymmetric IRB,
the source NVE also updates its ARP table with the received
adjacency information information, and in the case of the symmetric IRB, the
source NVE removes the entry associated with the received (MAC,
IP) (IP,
MAC) from its local ARP table. Furthermore, it withdraws its
previously advertised EVPN MAC/IP route with both the MAC and IP
address fields filled in.

* All other remote NVE devices devices, upon receiving the MAC/IP
advertisement
Advertisement route with the MAC Mobility extended community community,
compare the sequence number in this advertisement with the one
previously received. If the new sequence number is greater than
the old one, then they update the MAC/IP addresses of the TS in
their corresponding MAC-VRF, IP-VRF, and ARP tables (in the case
of asymmetric IRB) to point to the new NVE. Furthermore, upon
receiving the MAC/IP withdraw for the TS from the old NVE, these
remote PEs perform the cleanups for their BGP tables.

If an EVPN-IRB NVEs are NVE is configured not to advertise MAC-only routes,
then upon receiving the first data packet, it learns the MAC address
of the TS and updates the MAC entry in the corresponding MAC-VRF
table with the local adjacency information (e.g., local interface).
It also realizes that there has been a MAC move because the same MAC
address has been learned remotely from the source NVE. It uses the
local MAC route to find the corresponding local MAC-IP route, route and
sends a unicast ARP request to the host and when host. When receiving an ARP
response, it follows the procedure outlined in section Section 7.1. In the
prior case, where MAC-only routes are also advertised, this procedure
of triggering a unicast ARP probe at the target PE MAY also be used
in addition to the source PE broadcast ARP probing procedure
described earlier for better convergence.

7.3. Silent Host

In this scenario scenario, when a TS moves from a source NVE to a target NVE,
the TS is silent silent, and it neither initiates an ARP request nor it sends
any data traffic. Therefore, neither the target nor the source NVEs
are aware of the MAC move.

On the source NVE, an age-out timer (for the silent host that has
moved) is used to trigger an ARP probe. This age-out timer can be
either an ARP timer or a MAC age-out timer timer, and this is an
implementation choice. The ARP request gets sent both locally to all
the attached
TSes TSs on that subnet as well as it gets sent to all the remote NVEs
(including the target NVE) participating in that subnet. The source
NVE also withdraw withdraws the EVPN MAC/IP Advertisement route with only the
MAC address (if it has previously advertised such a route).

The target NVE passes the ARP request to its locally attached TSes TSs,
and when it receives the ARP response, it updates its MAC-VRF, IP-
VRF, and ARP table with the host (MAC, IP) (IP, MAC) and local adjacency
information (e.g., local interface). It also sends an EVPN MAC/IP
advertisement
Advertisement route with both the MAC and IP address fields filled in
along with the MAC Mobility Extended Community extended community, with the sequence
number incremented by one.

When the source NVE receives the EVPN MAC/IP Advertisement route, it
updates its IP-VRF table with the new adjacency information (pointing
to the target NVE). In the case of the asymmetric IRB, the source
NVE also updates its ARP table with the received adjacency
information
information, and in the case of the symmetric IRB, the source NVE
removes the entry associated with the received (MAC, IP) (IP, MAC) from its
local ARP table. Furthermore, it withdraws its previously advertised
EVPN MAC/IP route with both the MAC and IP address fields filled in.

All other remote NVE devices devices, upon receiving the MAC/IP Advertisement
route with the MAC Mobility extended community community, compare the sequence
number in this advertisement with the one previously received. If
the new sequence number is greater than the old one, then they update
the MAC/IP addresses of the TS in their corresponding MAC-VRF, IP-
VRF, and ARP (in the case of asymmetric IRB) tables to point to the
new NVE. Furthermore, upon receiving the MAC/IP withdraw for the TS
from the old NVE, these remote PEs perform the cleanups for their BGP
tables.

8. BGP Encoding

This document defines one new BGP Extended Community for EVPN.

8.1. EVPN Router's MAC Extended Community

A new EVPN BGP Extended Community called "EVPN Router's MAC MAC" is
introduced here. This new extended community is a transitive
extended community with the a Type field of 0x06 (EVPN) and the a Sub-Type
field of 0x03. It may be advertised along with the Encapsulation
Extended Community defined in
section Section 4.1 of [I-D.ietf-idr-tunnel-encaps]. [RFC9012].

The EVPN Router's MAC Extended Community is encoded as an 8-octet
value as follows:

0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type=0x06 | Sub-Type=0x03 | EVPN Router's MAC |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| EVPN Router's MAC Cont'd |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Figure 5: EVPN Router's MAC Extended Community

This extended community is used to carry the PE's MAC address for
symmetric IRB scenarios scenarios, and it is sent with EVPN RT-2. The
advertising PE SHALL only attach a single EVPN Router's MAC Extended
Community to a route. In case the receiving PE receives more than
one EVPN Router's MAC Extended Community with a route, it SHALL
process the first one in the list and not store and propagate the
others.

9. Operational Models for Symmetric Inter-Subnet Forwarding

The following sections describe two main symmetric IRB forwarding
scenarios (within a DC -- i.e., intra-DC) along with the
corresponding procedures. In the following scenarios, without loss
of generality, it is assumed that a given tenant is represented by a
single IP-VPN instance. Therefore, on a given PE, a tenant is
represented by a single IP-VRF table and one or more MAC-VRF tables.

9.1. IRB forwarding Forwarding on NVEs for Tenant Systems

This section covers the symmetric IRB procedures for the scenario
where each Tenant System (TS) TS is attached to one or more NVEs NVEs, and its host IP and
MAC addresses are learned by the attached NVEs and are distributed to
all other NVEs that are interested in participating in both intra-subnet intra-
subnet and inter-subnet communications with that TS.

In this scenario, without loss of generality, it is assumed that NVEs
operate in VLAN-based service interface mode with one bridge table(s)
per MAC-VRF. Thus, for a given tenant, an NVE has one MAC-VRF for
each tenant subnet (e.g., each VLAN) that is configured for extension
via VxLAN VXLAN or NVGRE encapsulation. In the case of VLAN-aware
bundling, then each MAC-VRF consists of multiple Bridge Tables bridge tables (e.g., one
bridge table per VLAN). The MAC-VRFs on an NVE for a given tenant
are associated with an IP-VRF corresponding to that tenant (or IP-VPN
instance) via their IRB interfaces.

Since VxLAN VXLAN and NVGRE encapsulations require an inner Ethernet header
(inner MAC SA/DA), SA/DA) and since for inter-subnet traffic, a TS MAC address cannot be used, used for
inter-subnet traffic, the ingress NVE's MAC address is used as an
inner MAC SA. The NVE's MAC address is the device MAC address address, and
it is common across all MAC-VRFs and IP-VRFs. This MAC address is
advertised using the new EVPN Router's MAC Extended Community (section
(Section 8.1).

Figure 6 below illustrates this scenario scenario, where a given tenant (e.g.,
an IP-VPN instance) has three subnets represented by MAC-VRF1, MAC-
VRF2, and MAC-VRF3 across two NVEs. There are five TSes TSs that are
associated with these three MAC-VRFs -- i.e., TS1, TS4, and TS5 are
on the same subnet (e.g., the same MAC-VRF/VLAN). TS1 and TS5 are
associated with MAC-VRF1 on NVE1, while TS4 is associated with MAC-
VRF1 on NVE2. TS2 is associated with MAC-VRF2 on NVE1, and TS3 is
associated with MAC-VRF3 on NVE2. MAC-VRF1 and MAC-VRF2 on NVE1 are are,
in turn turn, associated with IP-VRF1 on NVE1 NVE1, and MAC-VRF1 and MAC-VRF3
on NVE2 are associated with IP-VRF1 on NVE2. When TS1, TS5, and TS4
exchange traffic with each other, only the L2 forwarding (bridging)
part of the IRB solution is exercised because all these TSes TSs belong to
the same subnet. However, when TS1 wants to exchange traffic with
TS2 or TS3 TS3, which belong to different subnets, both the bridging and
routing parts of the IRB solution are exercised. The following
subsections describe the control and data planes plane operations for this
IRB scenario in details. detail.

NVE1 +---------+
+-------------+ | |
TS1-----| MACx| | | NVE2
(IP1/M1)
(M1/IP1) |(MAC- | | | +-------------+
TS5-----| VRF1)\ | | MPLS/ | |MACy (MAC- |-----TS3
(IP5/M5)
(M5/IP5) | \ | | VxLAN/ VXLAN/ | | / VRF3) | (IP3/M3) (M3/IP3)
| (IP-VRF1)|----| NVGRE |---|(IP-VRF1) |
| / | | | | \ |
TS2-----|(MAC- / | | | | (MAC- |-----TS4
(IP2/M2)
(M2/IP2) | VRF2) | | | | VRF1) | (IP4/M4) (M4/IP4)
+-------------+ | | +-------------+
| |
+---------+

Figure 6: IRB forwarding Forwarding on NVEs for Tenant Systems

9.1.1. Control Plane Operation

Each NVE advertises a MAC/IP Advertisement route (i.e., Route Type route type 2)
for each of its TSes TSs with the following field set:

* RD and ESI Ethernet Segment Identifier (ESI) per [RFC7432]

* Ethernet Tag = 0; assuming 0 (assuming VLAN-based service

o service)

* MAC Address Length = 48

* MAC Address = Mi ; where (where i = 1,2,3,4, 1, 2, 3, 4, or 5 5) in the Figure 6, above example

* IP Address Length = 32 or 128

* IP Address = IPi ; where (where i = 1,2,3,4, 1, 2, 3, 4, or 5 5) in the Figure 6, above example

* Label1 = MPLS Label label or VNI corresponding to MAC-VRF

* Label2 = MPLS Label label or VNI corresponding to IP-VRF

Each NVE advertises an EVPN RT-2 route with two Route Targets (one
corresponding to its MAC-VRF and the other corresponding to its IP-
VRF.
VRF). Furthermore, the EVPN RT-2 is advertised with two BGP Extended
Communities. The first BGP Extended Community identifies the tunnel
type
type, and it is called Encapsulation "Encapsulation Extended Community Community" as defined
in
[I-D.ietf-idr-tunnel-encaps] [RFC9012], and the second BGP Extended Community includes the MAC
address of the NVE (e.g., MACx for NVE1 or MACy for NVE2) as defined
in section Section 8.1. The EVPN Router's MAC Extended community Community MUST be
added when the Ethernet NVO tunnel is used. If the IP NVO tunnel
type is used, then there is no need to send this second Extended
Community. It should be noted that the IP NVO tunnel type is only
applicable to symmetric IRB procedures.

Upon receiving this advertisement, the receiving NVE performs the
following:

* It uses Route Targets corresponding to its MAC-VRF and IP-VRF for
identifying these tables and subsequently importing the MAC and IP
addresses into them them, respectively.

* It imports the MAC address from the MAC/IP Advertisement route
into the MAC-VRF with the BGP Next Hop next-hop address as the underlay
tunnel destination address (e.g., VTEP DA for VxLAN VXLAN encapsulation)
and
Label1 label1 as the VNI for VxLAN VXLAN encapsulation or an EVPN label for
MPLS encapsulation.

* If the route carries the new EVPN Router's MAC Extended Community, Community
and if the receiving NVE uses an Ethernet NVO tunnel, then the
receiving NVE imports the IP address into IP-VRF with NVE's MAC
address (from the new EVPN Router's MAC Extended Community) as the
inner MAC DA and DA, the BGP Next Hop next-hop address as the underlay tunnel
destination address, the VTEP DA for VxLAN encapsulation VXLAN encapsulation, and Label2
label2 as the IP-VPN VNI for VxLAN VXLAN encapsulation.

* If the receiving NVE uses MPLS encapsulation, then the receiving
NVE imports the IP address into IP-VRF with the BGP Next Hop next-hop
address as the underlay tunnel destination address, address and Label2 label2 as
the IP-VPN label for MPLS encapsulation.

If the receiving NVE receives an EVPN RT-2 with only Label1 label1 and only
a single Route Target corresponding to IP-VRF, or if it receives IP-VRF; an EVPN RT-2 with only
a single Route Target corresponding to MAC-VRF but with both Label1 label1
and Label2, label2; or if it receives an EVPN RT-2 with a MAC Address Length address length of zero, then
it MUST use the treat-as-withdraw approach [RFC7606] and SHOULD log
an error message.

9.1.2. Data Plane Operation

The following description of the data-plane data plane operation describes just
the logical functions functions, and the actual implementation may differ. Lets
Let's consider data-plane the data plane operation when TS1 in subnet-1 (MAC-VRF1) (MAC-
VRF1) on NVE1 wants to send traffic to TS3 in subnet-3 (MAC-VRF3) on
NVE2.

* NVE1 receives a packet with the MAC DA corresponding to the MAC-VRF1 MAC-
VRF1 IRB interface on NVE1 (the interface between MAC-VRF1 and IP-
VRF1),
VRF1) and VLAN-tag the VLAN tag corresponding to MAC-VRF1.

* Upon receiving the packet, the NVE1 uses VLAN-tag the VLAN tag to identify
the MAC-VRF1. It then looks up the MAC DA and forwards the frame
to its IRB interface.

* The Ethernet header of the packet is stripped stripped, and the packet is
fed to the IP-VRF IP-VRF, where an IP lookup is performed on the
destination IP address. NVE1 also decrements the TTL/hop TTL / hop limit
for that packet by one one, and if it reaches zero, NVE1 discards the
packet. This lookup yields the outgoing NVO tunnel and the
required encapsulation. If the encapsulation is for the Ethernet
NVO tunnel, then it includes the egress NVE's MAC address as the
inner MAC DA, the egress NVE's IP address (e.g., BGP Next Hop next-hop
address) as the VTEP DA, and the VPN-ID as the VNI. The inner MAC
SA and VTEP SA are set to NVE's MAC and IP addresses addresses,
respectively. If it is a an MPLS encapsulation, then the
corresponding EVPN and LSP labels are added to the packet. The
packet is then forwarded to the egress NVE.

o On

* If the egress NVE, if the NVE receives a packet arrives on from the Ethernet NVO tunnel
(e.g., it is VxLAN VXLAN encapsulated), then it removes the NVO tunnel header is
removed. Ethernet
header. Since the inner MAC DA is the egress NVE's MAC address,
the egress NVE knows that it needs to perform an IP lookup. It
uses the VNI to identify the IP-VRF table. If the packet is MPLS
encapsulated, then the EVPN label lookup identifies the IP-VRF
table. Next, an IP lookup is performed for the destination TS
(TS3)
(TS3), which results in an access-facing IRB interface over which
the packet is sent. Before sending the packet over this
interface, the ARP table is consulted to get the destination TS's
MAC address. NVE2 also decrements the TTL/hop TTL / hop limit for that
packet by one one, and if it reaches zero, NVE2 discards the packet.

* The IP packet is encapsulated with an Ethernet header header, with the
MAC SA set to that of the IRB interface MAC address (i.e, (i.e., the IRB
interface between MAC-VRF3 and IP-VRF1 on NVE2) and the MAC DA set
to that of the destination TS (TS3) MAC address. The packet is
sent to the corresponding MAC-VRF (i.e., MAC-VRF3) and and, after a
lookup of MAC DA, is forwarded to the destination TS (TS3) over
the corresponding interface.

In this symmetric IRB scenario, inter-subnet traffic between NVEs
will always use the IP-VRF VNI/MPLS label. For instance, traffic
from TS2 to TS4 will be encapsulated by NVE1 using NVE2's IP-VRF VNI/
MPLS label, as long as TS4's host IP is present in NVE1's IP-VRF.

9.2. IRB forwarding Forwarding on NVEs for Subnets behind Tenant Systems

This section covers the symmetric IRB procedures for the scenario
where some Tenant Systems (TSes) TSs support one or more subnets and these TSes TSs are
associated with one or more NVEs. Therefore, besides the
advertisement of MAC/IP addresses for each TS TS, which can be multi-
homed
multihomed with All-Active redundancy mode, the associated NVE needs
to also advertise the subnets statically configured on each TS.

The main difference between this solution and the previous one is the
additional advertisement corresponding to each subnet. These subnet
advertisements are accomplished using the EVPN IP Prefix route
defined in [I-D.ietf-bess-evpn-prefix-advertisement]. [RFC9136]. These subnet prefixes are advertised with the
IP address of their associated TS (which is in an overlay address
space) as their next hop. The receiving NVEs perform recursive route
resolution to resolve the subnet prefix with its advertising NVE so
that they know which NVE to forward the packets to when they are
destined for that subnet prefix.

The advantage of this recursive route resolution is that when a TS
moves from one NVE to another, there is no need to re-advertise any
of the subnet prefixes for that TS. All it that is needed is to
advertise the IP/MAC addresses associated with the TS itself and
exercise the MAC
mobility Mobility procedures for that TS. The recursive
route resolution automatically takes care of the updates for the
subnet prefixes of that TS.

Figure 7 illustrates this scenario where a given tenant (e.g., an IP-
VPN service) has three subnets represented by MAC-VRF1, MAC-VRF2, and
MAC-VRF3 across two NVEs. There are four TSes TSs associated with these
three MAC-VRFs -- i.e., TS1 is connected to MAC-VRF1 on NVE1, TS2 is
connected to MAC-VRF2 on NVE1, TS3 is connected to MAC- VRF3 MAC-VRF3 on NVE2,
and TS4 is connected to MAC-VRF1 on NVE2. TS1 has two subnet
prefixes (SN1 and SN2) SN2), and TS3 has a single subnet prefix, SN3. prefix (SN3).
The MAC-VRFs on each NVE are associated with their corresponding IP-VRF IP-
VRF using their IRB interfaces. When TS4 and TS1 exchange intra-
subnet traffic, only the L2 forwarding (bridging) part of the IRB
solution is used (i.e., the traffic only goes through their MAC-
VRFs); however, when TS3 wants to forward traffic to SN1 or SN2
sitting behind TS1 (inter-subnet traffic), then both the bridging and
routing parts of the IRB solution are exercised (i.e., the traffic
goes through the corresponding MAC-VRFs and IP-VRFs). If TS4, for
example, wants to reach SN1, it uses its default route and sends the
packet to the MAC address associated with the IRB interface on NVE2, NVE2;
NVE2 then makes performs an IP lookup in its IP- VRF, IP-VRF and finds an entry for
SN1. The following subsections describe the control and data planes plane
operations for this IRB scenario in details. detail.

NVE1 +----------+
SN1--+ +-------------+ | |
|--TS1-----|(MAC- \ | | |
SN2--+ IP1/M1 M1/IP1 | VRF1) \ | | |
| (IP-VRF)|---| |
| / | | |
TS2-----|(MAC- / | | MPLS/ |
IP2/M2
M2/IP2 | VRF2) | | VxLAN/ VXLAN/ |
+-------------+ | NVGRE |
+-------------+ | |
SN3--+--TS3-----|(MAC-\ | | |
IP3/M3
M3/IP3 | VRF3)\ | | |
| (IP-VRF)|---| |
| / | | |
TS4-----|(MAC- / | | |
IP4/M4
M4/IP4 | VRF1) | | |
+-------------+ +----------+
NVE2

Figure 7: IRB forwarding Forwarding on NVEs for subnets Subnets behind TSes TSs

Note that in figure Figure 7, above, SN1 and SN2 are configured on NVE1,
which then advertises each in an IP Prefix route. Similarly, SN3 is
configured on NVE2, which then advertises it in an IP Prefix route.

9.2.1. Control Plane Operation

Each NVE advertises a Route Type-5 route type 5 (EVPN RT-5, IP Prefix route
defined in [I-D.ietf-bess-evpn-prefix-advertisement]) [RFC9136]) for each of its subnet prefixes with the IP
address of its TS as the next hop
(gateway address (Gateway Address field) as follows:

* RD associated with the IP-VRF

* ESI = 0

* Ethernet Tag = 0;

o 0

* IP Prefix Length = 0 to 32 or 0 to 128

* IP Prefix = SNi

* Gateway Address = IPi; IP IPi (IP address of TS

o TS)

* MPLS Label = 0

This EVPN RT-5 is advertised with one or more Route Targets
associated with the IP-VRF from which the route is originated.

Each NVE also advertises an EVPN RT-2 (MAC/IP Advertisement Route) route)
along with their its associated Route Targets and Extended Communities for
each of its TSes TSs exactly as described in section Section 9.1.1.

Upon receiving the EVPN RT-5 advertisement, the receiving NVE
performs the following:

* It uses the Route Target to identify the corresponding IP-VRF

o IP-VRF.

* It imports the IP prefix into its corresponding IP-VRF that is configured
with an import RT that is one of the RTs being carried by the EVPN
RT-5 route route, along with the IP address of the associated TS as its
next hop.

When receiving the EVPN RT-2 advertisement, the receiving NVE imports
the MAC/IP addresses of the TS into the corresponding MAC-VRF and IP-VRF IP-
VRF per section Section 9.1.1. When both routes exist, recursive route
resolution is performed to resolve the IP prefix (received in EVPN
RT-5) to its corresponding NVE's IP address (e.g., its BGP next hop).
The BGP next hop will be used as the underlay tunnel destination
address (e.g., VTEP DA for VxLAN encapsulation) VXLAN encapsulation), and the EVPN
Router's MAC will be used as the inner MAC for VxLAN VXLAN encapsulation.

9.2.2. Data Plane Operation

The following description of the data-plane data plane operation describes just
the logical functions functions, and the actual implementation may differ. Lets
Let's consider data-plane the data plane operation when a host on in SN1 sitting behind TS1
wants to send traffic to a host sitting behind in SN3 behind TS3.

* TS1 send sends a packet with MAC DA corresponding to the MAC-VRF1 IRB
interface of NVE1, NVE1 and VLAN-tag a VLAN tag corresponding to MAC-VRF1.

* Upon receiving the packet, the ingress NVE1 uses VLAN-tag the VLAN tag to
identify the MAC-VRF1. It then looks up the MAC DA and forwards
the frame to its IRB interface just like section as in Section 9.1.1.

* The Ethernet header of the packet is stripped stripped, and the packet is
fed to the IP-VRF; where, IP-VRF, where an IP lookup is performed on the
destination address. This lookup yields the fields needed for
VxLAN
VXLAN encapsulation with NVE2's MAC address as the inner MAC DA,
NVE'2
NVE2's IP address as the VTEP DA, and the VNI. The MAC SA is set
to NVE1's MAC address address, and the VTEP SA is set to NVE1's IP
address. NVE1 also decrements the TTL/hop TTL / hop limit for that packet
by one one, and if it reaches zero, NVE1 discards the packet.

* The packet is then encapsulated with the proper header based on
the above info and is forwarded to the egress NVE (NVE2).

* On the egress NVE (NVE2), assuming the packet is VxLAN VXLAN
encapsulated, the VxLAN VXLAN and the inner Ethernet headers are removed
removed, and the resultant IP packet is fed to the IP-VRF
associated with that the VNI.

* Next, a lookup is performed based on the IP DA (which is in SN3)
in the associated IP-VRF of NVE2. The IP lookup yields the access-
facing
access-facing IRB interface over which the packet needs to be
sent. Before sending the packet over this interface, the ARP
table is consulted to get the destination TS (TS3) MAC address.
NVE2 also decrements the TTL/hop TTL / hop limit for that packet by one one,
and if it reaches zero, NVE2 discards the packet.

* The IP packet is encapsulated with an Ethernet header with the MAC
SA set to that of the access-facing IRB interface of the egress
NVE (NVE2) (NVE2), and the MAC DA is set to that of the destination TS
(TS3) MAC address. The packet is sent to the corresponding MAC-VRF3 and MAC-
VRF3 and, after a lookup of MAC DA, is forwarded to the
destination TS (TS3) over the corresponding interface.

11.

10. Security Considerations

The security considerations for layer-2 Layer 2 forwarding in this document
follow that those of [RFC7432] for MPLS encapsulation and it follows that those of
[RFC8365] for VxLAN VXLAN or NVGRE encapsulations. This section describes
additional considerations.

This document describes a set of procedures for Inter-Subnet
Forwarding inter-subnet
forwarding of tenant traffic across PEs (or NVEs). These procedures
include both layer-2 Layer 2 forwarding and layer-3 Layer 3 routing on a packet by packet-by-
packet basis. The security consideration for layer-3 Layer 3 routing in this
document follows that of [RFC4365] [RFC4365], with the exception for of the
application of routing protocols between CEs and PEs. Contrary to
[RFC4364], this document does not describe route distribution
techniques between CEs and PEs, PEs but rather considers the CEs as TSes TSs or
VAs that do not run dynamic routing protocols. This can be
considered a security advantage, since dynamic routing protocols can
be blocked on the NVE/PE ACs, not allowing the tenant to interact
with the infrastructure's dynamic routing protocols.

The VPN scheme described in this document does not provide the
quartet of security properties mentioned in [RFC4365]
(confidentiality protection, source authentication, integrity
protection, and replay protection). If these are desired, they must
be provided by mechanisms that are outside the scope of the VPN
mechanisms.

In this document, the EVPN RT-5 is used for certain scenarios. This
route uses an Overlay Index that requires a recursive resolution to a
different EVPN route (an EVPN RT-2). Because of this, it is worth
noting that any action that ends up filtering or modifying the EVPN
RT-2 route used to convey the Overlay Indexes, Indexes will modify the
resolution of the EVPN RT-5 and therefore the forwarding of packets
to the remote subnet.

12.

11. IANA Considerations

IANA has allocated a new transitive extended community Type of 0x06
and Sub-Type of value 0x03 for in the "EVPN Extended
Community Sub-Types" registry as follows:

+================+======================================+===========+
| Sub-Type Value | Name | Reference |
+================+======================================+===========+
| 0x03 | EVPN Router's MAC | RFC 9135 |
| | Extended Community. Community | |
+----------------+--------------------------------------+-----------+

Table 1

This document has been listed as an additional reference for the MAC/
IP Advertisement route in the EVPN "EVPN Route Type Types" registry.

13.

12. References

13.1.

12.1. Normative References

[I-D.ietf-bess-evpn-prefix-advertisement]
Rabadan, J., Henderickx, W., Drake, J., Lin, W., and A.
Sajassi, "IP Prefix Advertisement in EVPN", draft-ietf-
bess-evpn-prefix-advertisement-11 (work in progress), May
2018.

[I-D.ietf-idr-tunnel-encaps]
Patel, K., Velde, G., Sangli, S., and J. Scudder, "The BGP
Tunnel Encapsulation Attribute", draft-ietf-idr-tunnel-
encaps-22 (work in progress), January 2021.

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.

[RFC4364] Rosen, E. and Y. Rekhter, "BGP/MPLS IP Virtual Private
Networks (VPNs)", RFC 4364, DOI 10.17487/RFC4364, February
2006, <https://www.rfc-editor.org/info/rfc4364>.

[RFC7348] Mahalingam, M., Dutt, D., Duda, K., Agarwal, P., Kreeger,
L., Sridhar, T., Bursell, M., and C. Wright, "Virtual
eXtensible Local Area Network (VXLAN): A Framework for
Overlaying Virtualized Layer 2 Networks over Layer 3
Networks", RFC 7348, DOI 10.17487/RFC7348, August 2014,
<https://www.rfc-editor.org/info/rfc7348>.

[RFC7432] Sajassi, A., Ed., Aggarwal, R., Bitar, N., Isaac, A.,
Uttaro, J., Drake, J., and W. Henderickx, "BGP MPLS-Based
Ethernet VPN", RFC 7432, DOI 10.17487/RFC7432, February
2015, <https://www.rfc-editor.org/info/rfc7432>.

[RFC7606] Chen, E., Ed., Scudder, J., Ed., Mohapatra, P., and K.
Patel, "Revised Error Handling for BGP UPDATE Messages",
RFC 7606, DOI 10.17487/RFC7606, August 2015,
<https://www.rfc-editor.org/info/rfc7606>.

[RFC7637] Garg, P., Ed. and Y. Wang, Ed., "NVGRE: Network
Virtualization Using Generic Routing Encapsulation",
RFC 7637, DOI 10.17487/RFC7637, September 2015,
<https://www.rfc-editor.org/info/rfc7637>.

[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.

[RFC8365] Sajassi, A., Ed., Drake, J., Ed., Bitar, N., Shekhar, R.,
Uttaro, J., and W. Henderickx, "A Network Virtualization
Overlay Solution Using Ethernet VPN (EVPN)", RFC 8365,
DOI 10.17487/RFC8365, March 2018,
<https://www.rfc-editor.org/info/rfc8365>.

13.2.

[RFC9012] Patel, K., Van de Velde, G., Sangli, S., and J. Scudder,
"The BGP Tunnel Encapsulation Attribute", RFC 9012,
DOI 10.17487/RFC9012, April 2021,
<https://www.rfc-editor.org/info/rfc9012>.

[RFC9136] Rabadan, J., Ed., Henderickx, W., Drake, J., Lin, W., and
A. Sajassi, "IP Prefix Advertisement in Ethernet VPN
(EVPN)", RFC 9136, DOI 10.17487/RFC9136, October 2021,
<https://www.rfc-editor.org/info/rfc9136>.

12.2. Informative References

[I-D.ietf-bess-evpn-irb-extended-mobility]

[EVPN] Krattiger, L., Ed., Sajassi, A., Ed., Thoria, S., Rabadan,
J., and J. Drake, "EVPN Interoperability Modes", Work in
Progress, Internet-Draft, draft-ietf-bess-evpn-modes-
interop-00, 26 May 2021,
<https://datatracker.ietf.org/doc/html/draft-ietf-bess-
evpn-modes-interop-00>.

[EXTENDED-MOBILITY]
Malhotra, N., Ed., Sajassi, A., Pattekar, A., Lingala, A., Rabadan, J.,
Lingala, A., and J. Drake, "Extended Mobility Procedures
for EVPN-IRB", draft-ietf-bess-evpn-irb-extended-
mobility-03 (work Work in progress), May 2020.

[I-D.ietf-nvo3-vxlan-gpe]
Maino, F., Kreeger, L., and U. Elzur, "Generic Protocol
Extension for VXLAN (VXLAN-GPE)", draft-ietf-nvo3-vxlan-
gpe-10 (work in progress), July 2020. Progress, Internet-Draft, draft-
ietf-bess-evpn-irb-extended-mobility-07, 2 October 2021,
<https://datatracker.ietf.org/doc/html/draft-ietf-bess-
evpn-irb-extended-mobility-07>.

[RFC4365] Rosen, E., "Applicability Statement for BGP/MPLS IP
Virtual Private Networks (VPNs)", RFC 4365,
DOI 10.17487/RFC4365, February 2006,
<https://www.rfc-editor.org/info/rfc4365>.

[RFC5798] Nadas, S., Ed., "Virtual Router Redundancy Protocol (VRRP)
Version 3 for IPv4 and IPv6", RFC 5798,
DOI 10.17487/RFC5798, March 2010,
<https://www.rfc-editor.org/info/rfc5798>.

[RFC7365] Lasserre, M., Balus, F., Morin, T., Bitar, N., and Y.
Rekhter, "Framework for Data Center (DC) Network
Virtualization", RFC 7365, DOI 10.17487/RFC7365, October
2014, <https://www.rfc-editor.org/info/rfc7365>.

[VXLAN-GPE]
Maino, F., Ed., Kreeger, L., Ed., and U. Elzur, Ed.,
"Generic Protocol Extension for VXLAN (VXLAN-GPE)", Work
in Progress, Internet-Draft, draft-ietf-nvo3-vxlan-gpe-12,
22 September 2021, <https://datatracker.ietf.org/doc/html/
draft-ietf-nvo3-vxlan-gpe-12>.

Acknowledgements

The authors would like to thank Sami Boutros, Jeffrey Zhang,
Krzysztof Szarkowicz, Lukas Krattiger and Neeraj Malhotra for their
valuable comments. The authors would also like to thank Linda
Dunbar, Florin Balus, Yakov Rekhter, Wim Henderickx, Lucy Yong, and
Dennis Cai for their feedback and contributions.

Authors' Addresses

Ali Sajassi
Cisco Systems

Email: sajassi@cisco.com

Samer Salam
Cisco Systems

Email: ssalam@cisco.com

Samir Thoria
Cisco Systems

Email: sthoria@cisco.com

John E Drake
Juniper

Email: jdrake@juniper.net

Jorge Rabadan
Nokia

Email: jorge.rabadan@nokia.com