Routing Area Working GroupInternet Engineering Task Force (IETF) S. LitkowskiInternet-DraftRequest for Comments: 8333 B. DecraeneIntended status:Category: Standards Track OrangeExpires: May 16, 2018ISSN: 2070-1721 C. Filsfils Cisco Systems P. Francois IndividualNovember 12, 2017Contributor March 2018 Micro-looppreventionPrevention byintroducingIntroducing alocal convergence delay draft-ietf-rtgwg-uloop-delay-09Local Convergence Delay Abstract This document describes a mechanism for link-state routing protocolsto preventthat prevents local transient forwarding loops in case of link failure. This mechanism proposes a two-step convergence by introducing a delay between the convergence of the node adjacent to the topology change and thenetwork widenetwork-wide convergence.AsBecause this mechanism delays the IGPconvergenceconvergence, it may only be used for planned maintenance or whenfast rerouteFast Reroute (FRR) protects the traffic during the time between the link failuretimeand the IGP convergence. Theproposedmechanism is limited to thelink downlink-down event in order to keep the mechanism simple. Simulations using real network topologies have been performed and show that local loops are a significant portion (>50%) of the total forwarding loops. Status of This Memo ThisInternet-Draftissubmitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documentsan Internet Standards Track document. This document is a product of the Internet Engineering Task Force (IETF).Note that other groups may also distribute working documents as Internet-Drafts. The listIt represents the consensus ofcurrent Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents validthe IETF community. It has received public review and has been approved fora maximumpublication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 7841. Information about the current status ofsix monthsthis document, any errata, and how to provide feedback on it may beupdated, replaced, or obsoleted by other documentsobtained atany time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on May 16, 2018.https://www.rfc-editor.org/info/rfc8333. Copyright Notice Copyright (c)20172018 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License. Table of Contents 1.Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.Introduction. . . . . . . . . . . . . . . . . . . . . . . . 3....................................................3 2. Terminology .....................................................4 2.1. Acronyms ...................................................4 2.2. Requirements Language ......................................4 3. Side Effects of Transientforwarding loops side effects . . . . . . . . . . . 4Forwarding Loops ......................4 3.1.Fast reroute inefficiency . . . . . . . . . . . . . . . . 4FRR Inefficiency ...........................................5 3.2. Networkcongestion . . . . . . . . . . . . . . . . . . . 7Congestion .........................................7 4. Overview of thesolution . . . . . . . . . . . . . . . . . . 7Solution ........................................8 5. Specification. . . . . . . . . . . . . . . . . . . . . . . . 8...................................................8 5.1. Definitions. . . . . . . . . . . . . . . . . . . . . . . 8................................................8 5.2. Regular IGPreaction . . . . . . . . . . . . . . . . . . 8Reaction .......................................9 5.3. Localevents . . . . . . . . . . . . . . . . . . . . . . 9Events ...............................................9 5.4. LocaldelayDelay forlink down . . . . . . . . . . . . . . . . 10Link-Down Events ..........................10 6. Applicability. . . . . . . . . . . . . . . . . . . . . . . . 10..................................................11 6.1. Applicablecase: local loops . . . . . . . . . . . . . . 10Case: Local Loops ..............................11 6.2.Non applicable case: remote loops . . . . . . . . . . . . 11Non-applicable Case: Remote Loops .........................11 7. Simulations. . . . . . . . . . . . . . . . . . . . . . . . . 11....................................................12 8. Deploymentconsiderations . . . . . . . . . . . . . . . . . . 12Considerations ......................................13 9. Examples. . . . . . . . . . . . . . . . . . . . . . . . . . 13.......................................................14 9.1. Locallink down . . . . . . . . . . . . . . . . . . . . . 14Link-Down Event .....................................14 9.2. Local andremote event . . . . . . . . . . . . . . . . . 18Remote Event ....................................18 9.3. Abortinglocal delay . . . . . . . . . . . . . . . . . . 19Local Delay ......................................20 10. Comparison withother solutions . . . . . . . . . . . . . . . 23Other Solutions ...............................22 10.1. PLSN. . . . . . . . . . . . . . . . . . . . . . . . . . 23.....................................................22 10.2.OFIB . . . . . . . . . . . . . . . . . . . . . . . . . . 23oFIB .....................................................23 11.Implementation Status . . . . . . . . . . . . . . . . . . . . 24IANA Considerations ...........................................23 12. Security Considerations. . . . . . . . . . . . . . . . . . . 25.......................................23 13.Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 25 14. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 26 15.References. . . . . . . . . . . . . . . . . . . . . . . . . 26 15.1.....................................................23 13.1. Normative References. . . . . . . . . . . . . . . . . . 26 15.2......................................23 13.2. Informative References. . . . . . . . . . . . . . . . . 26...................................24 Acknowledgements ..................................................25 Authors' Addresses. . . . . . . . . . . . . . . . . . . . . . . 27 2.................................................25 1. IntroductionMicro-forwarding loopsMicro-loops and some potential solutions arewelldescribed in [RFC5715]. This document describes a simple targeted mechanism that prevents micro-loops that are local to the failure. Based on network analysis, localfailuresmicro-loops make up a significant portion of themicro-forwarding loops.micro-loops. A simple and easily deployable solution for these local micro-loops is critical because these local loops cause some traffic loss aftera fast-reroutean FRR alternate has been used (see Section 3.1). Consider the case in Figure 1 where S does not have an LFA(Loop Free(Loop-Free Alternate) to protect its traffic to D when the S-D link fails. That means that all non-D neighbors of S on the topology will send to S any traffic destined to D; if a neighbor did not, then that neighbor would be loop-free. Regardless of the advancedfast-reroute (FRR)FRR technique used, when S converges to the new topology, it will send its traffic to a neighbor thatwasis not loop-free and will thus cause a localmicro-loop.micro- loop. The deployment of advancedfast-rerouteFRR techniques motivates this simple router-local mechanism to solve this targeted problem. This solution can work with the various techniques described in [RFC5715]. D ------ C | | | | 5 | | S ------ B Figure 1 IntheFigure 1, all links have a metric of 1 except the B-C link, which has a metric of 5. When the S-D link fails, a transient forwarding loop may appear between S and B if S updates its forwarding entry to D before B does.1.2. Terminology 2.1. Acronyms FIB: Forwarding Information Base FRR: FastReRouteReroute IGP: Interior Gateway Protocol LFA:Loop FreeLoop-Free Alternate LSA: Link State Advertisement LSP: Link State Packet MRT:MaximumMaximally RedundantTrees OFIB:Tree oFIB: Ordered FIB PLR: Point of Local Repair PLSN: Path Locking via SafeNeighborNeighbors RIB: Routing Information Base RLFA: RemoteLoop FreeLoop-Free Alternate SPF: Shortest Path First TTL: TimeToto Live 2.2. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in[RFC2119].BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 3. Side Effects of Transientforwarding loops side effectsForwarding Loops Even if they are very limited in duration, transient forwarding loops may cause significant network damage. 3.1.Fast reroute inefficiencyFRR Inefficiency In Figure 2, we consider an IP/LDP routed network. D 1 | | 1 A ------ B | | ^ 10 | | 5 | T | | | E--------C | 1 1 | S Figure 2- RSVP-TE FRR case In the Figure 2, we consider an IP/LDP routed network.An RSVP-TE tunnel T, provisioned on C and terminating on B, is used to protect the traffic against C-B link failure (the IGP shortcut feature, defined in [RFC3906], is activated onC ).C). The primary path of T is C->B and FRR is activated onTT, providing an FRR bypass or detour using path C->E->A->B. On router C, the next hop to D is the tunnelTT, thanks to the IGP shortcut. When the C-B link fails: 1. C detects thefailure,failure and updates the tunnel path using a preprogrammed FRR path. The traffic path from S to Dbecomes:becomes S->E->C->E->A->B->A->D. 2. In parallel, on router C, both the IGP convergence and the TE tunnel convergence (tunnel path recomputation) are occurring: * TheTunneltunnel T path is recomputed and now uses C->E->A->B. * The IGP path to D is recomputed and now uses C->E->A->D. 3. On C, the tail-end of the TE tunnel (router B) is no longer on the shortest-path tree (SPT) to D, so C does not continue to encapsulate the traffic to D using the tunnel T and updates its forwarding entry to D using thenexthopnext-hop E. If C updates its forwarding entry to D before router E, there would be a transient forwarding loop between C and E until E has converged.The tableTable 1belowdescribes a theoretical sequence of events happening when the B-C link fails. This theoretical sequence of events should only be read as an example.+-----------+------------+------------------+-----------------------++------------+--------+---------------------+-----------------------+ | Network | Time | Router CeventsEvents | Router EeventsEvents | |conditionCondition | | | |+-----------+------------+------------------+-----------------------++------------+--------+---------------------+-----------------------+ | S->D | | | | | Traffic| | | | |OK | | | | | | | | | | S->D | t0 | Link B-C fails | Link B-C fails | | Traffic | | | | | lost | | | | | | | | | | |t0+20msect0+20 | C detects the | | | | ms | failure | | | | | | | | S->D |t0+40msect0+40 | C activates FRR | | | Traffic| | | | |OK | ms | | | | | | | | | |t0+50msect0+50 | C updates its local | | | | ms |localLSP/LSA | | | | | | | | |t0+60msect0+60 | Cschedules SPFfloods its local | | | | ms |(100ms)updated LSP/LSA | | | | | | | | |t0+70msect0+62 | Cfloods its | | | | | local updatedschedules SPF | | | | ms |LSP/LSA(100 ms) | | | | | | | | |t0+87msect0+87 | | E receives LSP/LSA | | | ms | | from C andschedules | | | | | SPF (100ms)floods it | | | | | | | |t0+117msect0+92 | | Efloods LSP/LSA fromschedules SPF (100 | | | ms | |Cms) | | | | | | | |t0+160msect0+163 | C computes SPF | | | | ms | | | | |t0+165msec|C starts| | | | t0+165 | C starts updatingits| | | | ms | its RIB/FIB | | | | | | | | |t0+193msect0+193 | | E computes SPF | | | ms | | | | | | |t0+199msec| | | t0+199 | | E starts updating its | | | ms | | RIB/FIB | | | | | | | S->D |t0+255msect0+255 | C updates its | | | Traffic | ms | RIB/FIB for D | | | lost | | | | | | | | | | |t0+340msect0+340 | C convergence ends | | | | ms |ends| | | | | | | | S->D |t0+443msect0+443 | | E updates its RIB/FIB | | Traffic OK | ms | | for D | |OK| | | | | | t0+470 | | E convergence ends | | |t0+470msecms | |E convergence ends|+-----------+------------+------------------+-----------------------++------------+--------+---------------------+-----------------------+ Table 1- Route computation event time scaleThe issue described here is completely independent of thefast- rerouteFRR mechanism involved(TE(e.g., TE FRR,LFA/rLFA, MRT ...)LFA/RLFA, MRT, etc.) when the primary path uses hop-by-hop routing. The protection enabled byfast-reroute is working perfectly,FRR works perfectly butensures a protection, by definition,only ensures protection until the PLR has converged (as soon as the PLR has converged, it replaces its FRR pathbywith a new primary path). When implementing FRR, a service provider wants to guarantee a very limited loss of connectivity time. Thepreviousexample described in this section shows that the benefit of FRR may be completely lost due to a transient forwarding loop appearing when PLR has converged. Delaying FIB updates after the IGP convergence (1) may allowto keepthefast-rerouteFRR path to be kept until the neighbors have converged and (2) preserves the customer traffic. 3.2. Networkcongestion 1 D ------ C | | 1 | | 5 | | A -- S ------ B / | 1 F E Figure 3Congestion Inthe figure above, as presented in Section 2,Figure 3, when thelinkS-D link fails, a transient forwarding loop may appear between S and B for destination D. The traffic on the S-B link will constantly increase due to the looping traffic to D. Depending on the TTL of the packets, the traffic rate destined to D, and the bandwidth of the link, the S-B link may become congested in a few hundreds of milliseconds and will stay congested until the loop is eliminated. 1 D ------ C | | 1 | | 5 | | A -- S ------ B / | 1 F E Figure 3 The congestion introduced by transient forwarding loops is problematic as it can affect traffic that is not directly affected by the failing network component. Inthe example,Figure 3, the congestion of the S-B link will impact some customer traffic that is not directly affected by thefailure: e.g.failure, e.g., traffic from A to B, F to B, and E to B. Class of service may mitigate the congestion for some traffic. However, some traffic not directly affected by the failure will still be dropped as a router is not able to distinguish the looping traffic from the normally forwarded traffic. 4. Overview of thesolutionSolution This document defines a two-step convergence initiated by the router detecting a failure and advertising the topologicalchangeschange in the IGP. This introduces a delay between network-wide convergence and the convergence of the local router. Theproposedsolution described in this document is limited to locallink downlink-down events in order to keep the solution simple. This ordered convergence is similar to the ordered FIBproposed(oFIB) approach defined in [RFC6976], but it is limited to only a"one hop""one-hop" distance. As a consequence, it is more simple and becomes alocal-onlylocal- only feature that does not require interoperability. This benefit comes with the limitation of eliminating transient forwarding loops involving the local router only. Theproposedmechanism also reuses some concepts described in[I-D.ietf-rtgwg-microloop-analysis].[PLSN]. 5. Specification 5.1. Definitions This documentwill referrefers to the following existing IGP timers. These timers may be standardized or implemented as avendor specificvendor-specific local feature. o LSP_GEN_TIMER: The delay between the consecutive generation of twoconsecutiveslocalLSP/LSA generation.LSPs/LSAs. From an operational point of view, this delay is usually tuned to batch multiple local events inonea single local LSP/LSA update. In IS-IS, this timer is defined as minimumLSPGenerationIntervalin[ISO10589]. In OSPF version 2, this timer is defined as MinLSIntervalin[RFC2328]. It is often associated with avendor specificvendor-specific damping mechanism to slow down reactions by incrementing the timer when multiple consecutive events are detected. o SPF_DELAY: The delay between the first IGP event triggering a new routing table computation and the start of that routing table computation. It is often associated with a damping mechanism to slow down reactions by incrementing the timer when the IGP becomes unstable. As an example,[I-D.ietf-rtgwg-backoff-algo][BACKOFF] defines a standard SPF(Shortest Path First)delay algorithm. This document introduces the following new timer: o ULOOP_DELAY_DOWN_TIMER:usedUsed to slow down the local node convergence in case oflink downlink-down events. 5.2. Regular IGPreaction Upon a change ofReaction When the status of anadjacency/link,adjacency or link changes, the regular IGP convergence behavior of the router advertising the event involves the following main steps: 1. IGP is notified of theUp/Downup/down event. 2. The IGP processes the notification and postpones the reaction for LSP_GEN_TIMERmsec.ms. 3. Upon LSP_GEN_TIMER expiration, the IGP updates its LSP/LSA and floods it. 4. The SPF computation is scheduled in SPF_DELAYmsec.ms. 5. Upon SPF_DELAY timer expiration, the SPF is computed, and then the RIB and FIB are updated. 5.3. LocaleventsEvents The mechanism described in this document assumes that there has been a single link failure as seen by the IGP area/level. If this assumption is violated(e.g.(e.g., multiple links or nodes failed), then regular IP convergence must be applied (as described in Section 5.2). To determine if the mechanismcan beis applicable or not, an implementation SHOULD implement logic to correlate the protocol messages (LSP/LSA) received during the SPF scheduling period in order to determine the topology changes thatoccured.occurred. This is necessary as multiple protocol messages may describe the same topologychangechange, and a single protocol message may describe multiple topology changes. As a consequence, determining a particular topology change MUST be independent of the order of reception of those protocol messages. How the logic works is left to the implementation. Using this logic, if an implementation determines that the associated topology change is a single local link failure, then the router MAY use the mechanism described in thisdocument, otherwisedocument; otherwise, the regular IP convergence MUST be used.Example: +--- E ----+--------+ | | | A ---- B -------- C ------ DIn Figure4 Let4, let router B be the computing router when the link B-C fails. B updates its local LSP/LSA describing the linkB->CB-C as down, C does the same, and both start flooding their updatedLSP/LSAs.LSPs/LSAs. During the SPF_DELAY period, B and C learn all the LSPs/LSAs to consider. B sees that C is flooding an advertisement that indicates that a link is down, and B is the other end of that link. B determines that B and C are describing the same single event. Since B receives no other changes, B can determine that this is a local link failure and may decide to activate the mechanism described in this document. +--- E ----+--------+ | | | A ---- B -------- C ------ D Figure 4 5.4. LocaldelayDelay forlink down Upon an adjacency/link down event, thisLink-Down Events This document introduces a change in step 5(Section 5.2)(see list inorder to delaySection 5.2) so that, upon an adjacency or link-down event, the local convergence is delayed compared to thenetwork widenetwork-wide convergence. The new step 5 is described below: 5. Upon SPF_DELAY timer expiration, the SPF is computed. If the condition of a single local link-down event has been met, then an update of the RIB and the FIB MUST be delayed for ULOOP_DELAY_DOWN_TIMERmsecs.ms. Otherwise, the RIB and FIB SHOULD be updated immediately. If a new convergence occurs while ULOOP_DELAY_DOWN_TIMER is running, ULOOP_DELAY_DOWN_TIMER isstoppedstopped, and the RIB/FIB SHOULD be updated as part of the new convergence event. As a result of this addition, routers local to the failure will converge slower than remote routers.HenceHence, it SHOULD only be done for a non-urgent convergence, such asforadministrativede- activationdeactivation (maintenance) or when the traffic is protected byfast- reroute.FRR. 6. Applicability As previously stated, this mechanism only avoids the forwarding loops on the links between the node local to the failure and its neighbors. Forwarding loops may still occur on other links. 6.1. Applicablecase: local loops A ------ B ----- E | / | | / | G---D------------C F All the links have a metric of 1Case: Local Loops In Figure5 Let5, let us consider the traffic from G to F. The primary path is G->D->C->E->F. When the link C-E fails, if C updates its forwarding entry for F before D, a transient loop occurs. This is sub-optimal asC has FRR enabled andit breakstheC's FRR forwardingwhile alleven though upstream routers are still forwarding the traffic toitself.C. A ------ B ----- E | / | | / | G---D------------C F All the links have a metric of 1 Figure 5 By implementing the mechanism defined in this document on C, when the C-E link fails, C delays the update of its forwarding entry to F, in order to allow some time for D to converge. FRR on C keeps protecting the traffic during this period. Whenthe timerULOOP_DELAY_DOWN_TIMER expires on C, its forwarding entry to F is updated. There is no transient forwarding loop on the link C-D. 6.2.Non applicable case: remote loops A ------ B ----- E --- H | | | | G---D--------C ------F --- J ---- K All the links have a metric of 1 except BE=15Non-applicable Case: Remote Loops In Figure6 Let6, let us consider the traffic from G to K. The primary path is G->D->C->F->J->K. When the C-F link fails, if C updates its forwarding entry to K before D, a transient loop occurs between C and D. A ------ B ----- E --- H | | | | G---D--------C ------F --- J ---- K All the links have a metric of 1 except B-E=15 Figure 6 By implementing the mechanism defined in this document on C, when the link C-F fails, C delays the update of its forwarding entry to K, allowing time for D to converge. Whenthe timerULOOP_DELAY_DOWN_TIMER expires on C, its forwarding entry to F is updated. There is no transient forwarding loop between C and D. However, a transient forwarding loop may still occur between D and A. In this scenario, this mechanism is not enough to address all the possible forwarding loops. However, it does not create additional traffic loss. Besides, in some cases-such-- such as when the nodes update their FIB in thefollowingorder C, A,D, for exampleD because the router A is quicker than D toconverge-converge -- the mechanism may still avoid the forwarding loop that would have otherwise occurred. 7. Simulations Simulations have been run on multipleservice providerservice-provider topologies. We evaluated the efficiency of the mechanism on eight different service-provider topologies (different network size and design). Table 2 displays the gain for each topology. +----------+------+ | Topology | Gain | +----------+------+ | T1 | 71% | | T2 | 81% | | T3 | 62% | | T4 | 50% | | T5 | 70% | | T6 | 70% | | T7 | 59% | | T8 | 77% | +----------+------+ Table 2- Number of Repair/Dst that may loopWe evaluated theefficiency of the mechanism on eight different service provider topologies (different network size, design). The benefit is displayed in the table above. The benefit is evaluatedgain as follows: o Weconsiderconsidered a tuple (link A-B, destination D, PLR S, backupnexthopnext-hop N) as a loopifif, upon link A-B failure, the flow from a router S upstream from A (A could be considered as PLR also) to D may loop due to convergence time difference between S and one ofhisits neighbors N. o Weevaluateevaluated the number of potential loop tuples in normal conditions. o Weevaluateevaluated the number of potential loop tuples using the same topological input but taking into account that S converges after N. o The gain ishow manythe relative number of loops (both remote and local) we succeedto suppress. Onin suppressing. For topology 1, implementing the local delay prevented 71% of the transient forwarding loops created by the failure of anylink are prevented by implementing the local delay.link. The analysis shows that all local loops are prevented and only remote loops remain. 8. DeploymentconsiderationsConsiderations Transient forwarding loops have the following drawbacks: o They limit FRRefficiency: evenefficiency. Even if FRR is activated within50msec,50 ms, as soon as the PLR has converged, the traffic may be affected by a transient loop. o They may impact traffic not directly affected by the failure (due to link congestion).ThisThe local delayproposalmechanism is a transient forwarding loop avoidance mechanism (likeOFIB).oFIB). Even if it only addresses local transient loops, the efficiency versus complexity comparison of the mechanism makes it a good solution. It is also incrementally deployable with incremental benefits, which makes it an attractive optionbothfor both vendors to implement and service providers to deploy. Delaying the convergence time is not an issue if we consider that the traffic is protected during the convergence. The ULOOP_DELAY_DOWN_TIMER value should be set according to the maximum IGP convergence time observed in the network (usually observed in the slowest node).The proposedThis mechanism is limited tolink downlink-down events. When a link goes down, it eventually goes back up. As a consequence, withthe proposedthis mechanism deployed, only thelink downlink-down event will be protected against transient forwarding loops while thelink uplink-up event will not. If the operator wants to limit the impact ofthetransient forwarding loops during thelink uplink-up event, it shouldtake care of usingmake sure to use specific procedures to bring the link back online. As examples, the operator can decide to putbackthe link back onlineoutoutside of businesshourshours, or it can use some incremental metric changes to prevent loops (as proposed in [RFC5715]). 9. Examples Wewillconsider the following figure for theassociatedexamples:in this section: D 1 | F----X | 1 | A ------ B | | 10 | | 5 | | E--------C | 1 1 | S Figure 7 The network above is considered to have a convergence time of about 1 second, so ULOOP_DELAY_DOWN_TIMER will be adjusted to this value. We also consider that FRR is running on each node. 9.1. Locallink down The tableLink-Down Event Table 3 describes the events andassociatedtheir timingthat happenonrouterrouters C and E when the link B-C goes down. It is based on a theoretical sequence ofeventevents that should only been read as an example. As C detects a single local event corresponding to alink downlink-down event (its LSP + LSP from B received), it applies the local delay downbehaviorbehavior, and nomicroloopmicro-loop is formed.+-----------+-------------+------------------+----------------------++------------+---------+---------------------+----------------------+ | Network | Time | Router CeventsEvents | Router EeventsEvents | |conditionCondition | | | |+-----------+-------------+------------------+----------------------++------------+---------+---------------------+----------------------+ | S->D | | | | | Traffic| | | | |OK | | | | | | | | | | S->D | t0 | Link B-C fails | Link B-C fails | | Traffic | | | | | lost | | | | | | | | | | |t0+20msect0+20 | C detects the | | | | ms | failure | | | | | | | | S->D |t0+40msect0+40 | C activates FRR | | | Traffic| | | | |OK | ms | | | | | | | | | |t0+50msect0+50 | C updates its local | | | | ms |localLSP/LSA | | | | | | | | |t0+60msect0+53 | Cschedules SPFfloods its local | | | | ms |(100ms)updated LSP/LSA | | | | | | | | |t0+67msect0+60 | Creceivesschedules SPF | | | | ms |LSP/LSA from B(100 ms) | | | | | | | | |t0+70msect0+67 | Cfloods itsreceives LSP/LSA | | | | ms |local updatedfrom B and floods | | | | |LSP/LSAit | | | | | | | | |t0+87msect0+87 | | E receives LSP/LSA | | | ms | | from C andschedulesfloods it | | | | |SPF (100ms)| | | t0+90 | | E schedules SPF (100 | | |t0+117msecms | |E floods LSP/LSAms) | | | | |from C| | | t0+161 | C computes SPF | | | |t0+160msecms |C computes SPF| | | | | | | | |t0+165msect0+165 | C delays its | | | | ms | RIB/FIB update (1 | | | | |(1sec) | | | | | | | | |t0+193msect0+193 | | E computes SPF | | | ms | | | | | |t0+199msec| | | | t0+199 | | E starts updating | | | ms | | its RIB/FIB | | | | | | | |t0+443msect0+443 | | E updates its | | | ms | | RIB/FIB for D | | | | | | | |t0+470msect0+470 | | E convergence ends | | | ms | | | | |t0+1165msec|C starts| | | | t0+1165 | C starts updatingits| | | | ms | its RIB/FIB | | | | | | | | |t0+1255msect0+1255 | C updates its | | | | ms | RIB/FIB for D | | | | | | | | |t0+1340msect0+1340 | C convergence ends | | | | ms |ends| |+-----------+-------------+------------------+----------------------++------------+---------+---------------------+----------------------+ Table 3- Route computation event time scaleSimilarly, upon B-Clink downlink-down event, if LSP/LSA from B is received before C detects the link failure, C will apply the route update delay if the local detection is part of the same SPF run.The tableTable 4 describes the associated theoretical sequence of events. It should only been read as an example.+-----------+-------------+------------------+----------------------++------------+---------+---------------------+----------------------+ | Network | Time | Router CeventsEvents | Router EeventsEvents | |conditionCondition | | | |+-----------+-------------+------------------+----------------------++------------+---------+---------------------+----------------------+ | S->D | | | | | Traffic| | | | |OK | | | | | | | | | | S->D | t0 | Link B-C fails | Link B-C fails | | Traffic | | | | | lost | | | | | | | | | | |t0+32msect0+32 | C receives LSP/LSA | | | | ms |LSP/LSAfrom B and floods | | | | | it | | | | | | | |t0+33msec| t0+33 | C schedules SPF | | | | ms |(100ms)(100 ms) | | | | | | | | |t0+50msect0+50 | C detects the | | | | ms | failure | | | | | | | | S->D |t0+55msect0+55 | C activates FRR | | | Traffic| | | | |OK | ms | | | | | | | | | |t0+55msect0+55 | C updates its local | | | | ms |localLSP/LSA | | | | | | | | |t0+70msect0+70 | C floods its| | | | |localupdated| | | | ms | updated LSP/LSA | | | | | | | | |t0+87msect0+87 | | E receives LSP/LSA | | | ms | | from C andschedulesfloods it | | | | |SPF (100ms)| | | t0+90 | | E schedules SPF (100 | | |t0+117msecms | |E floods LSP/LSAms) | | | | |from C| | | t0+135 | C computes SPF | | | |t0+160msecms |C computes SPF| | | | | | | | |t0+165msect0+140 | C delays its | | | | ms | RIB/FIB update (1 | | | | |(1sec) | | | | | | | | |t0+193msect0+193 | | E computes SPF | | | ms | | | | | | | |t0+199msec| | t0+199 | | E starts updating | | | ms | | its RIB/FIB | | | | | | | |t0+443msect0+443 | | E updates its | | | ms | | RIB/FIB for D | | | | | | | |t0+470msect0+470 | | E convergence ends | | | ms | | | | |t0+1165msec|C starts| | | | t0+1145 | C starts updatingits| | | | ms | its RIB/FIB | | | | | | | | |t0+1255msect0+1255 | C updates its | | | | ms | RIB/FIB for D | | | | | | | | |t0+1340msect0+1340 | C convergence ends | | | | ms |ends| |+-----------+-------------+------------------+----------------------++------------+---------+---------------------+----------------------+ Table 4- Route computation event time scale9.2. Local andremote event The tableRemote Event Table 5 describes the events andassociatedtheir timingthat happenon router C and E when the link B-C goesdown, in addition F-Xdown and when the linkwill failF-X fails in the same time window. C will not apply the local delay because anon localnon-local topology change is also received.The tableTable 5 is based on a theoretical sequence ofeventevents that should only been read as an example.+-----------+------------+-----------------+------------------------++-----------+--------+-------------------+--------------------------+ | Network | Time | Router CeventsEvents | Router EeventsEvents | |conditionCondition | | | |+-----------+------------+-----------------+------------------------++-----------+--------+-------------------+--------------------------+ | S->D | | | | | Traffic | | | | | OK | | | | | | | | | | S->D | t0 | Link B-C fails | Link B-C fails | | Traffic | | | | | lost | | | | | | | | | | |t0+20msect0+20 | C detects the | | | | ms | failure | | | | | | | | |t0+36msect0+36 | Link F-X fails | Link F-X fails | | | ms | | | | | | | | | S->D |t0+40msect0+40 | C activates FRR | | | Traffic | ms | | | | OK | | | | | | | | | | |t0+50msect0+50 | C updates its | | | | ms | local LSP/LSA | | | | | | | | |t0+54msect0+54 | C receives | | | | ms | LSP/LSA from F | | | | | and floods it | | | | | | | | |t0+60msect0+60 | C schedules SPF | | | | ms |(100ms)(100 ms) | | | | | | | | |t0+67msect0+67 | C receives | | | | ms | LSP/LSA from B | | | | | and floods it | | | | | | | | |t0+69msect0+69 | | E receives LSP/LSA from | | | ms | |fromF, floods it and | | | | | schedules SPF(100ms)(100 ms) | | | | | | | |t0+70msect0+70 | C floods its | | | | ms | local updated | | | | | LSP/LSA | | | | | | | | |t0+87msect0+87 | | E receives LSP/LSA from | | | ms | |fromC | | | | | | | |t0+117msect0+117 | | E floods LSP/LSA from C | | | ms | |C| | | | | | | |t0+160msect0+160 | C computes SPF | | | | ms | | | | |t0+165msec|C starts| | | | t0+165 | C starts updatingits| | | | ms | its RIB/FIB (NO | | | | | DELAY) | | | | | | | | |t0+170msect0+170 | | E computes SPF | | | ms | | | | | |t0+173msec| | | | t0+173 | | E starts updating its | | | ms | | RIB/FIB | | | | | | | S->D |t0+365msect0+365 | C updates its | | | Traffic | ms | RIB/FIB for D | | | lost | | | | | | | | | | S->D |t0+443msect0+443 | | E updates its RIB/FIB | | Traffic | ms | | for D | | OK | | | | | | | | | | |t0+450msect0+450 | C convergence | | | | ms | ends | | | | | | | | |t0+470msect0+470 | | E convergence ends | | | ms | | | | | | |+-----------+------------+-----------------+------------------------+| +-----------+--------+-------------------+--------------------------+ Table 5- Route computation event time scale9.3. Abortinglocal delay The tableLocal Delay Table 6 describes the events andassociatedtheir timingthat happenonrouterrouters C and E when the link B-C goes down. In addition, we consider what happens when the F-X link fails during local delay of the FIB update. C will first apply the local delay, but when the new event happens, it will fall back to the standard convergence mechanism without further delaying route insertion. In this example, we consider a ULOOP_DELAY_DOWN_TIMER configured to 2 seconds.The tableTable 6 is based on a theoretical sequence ofeventevents that should only been read as an example.+-----------+------------+-------------------+----------------------++------------+--------+----------------------+----------------------+ | Network | Time | Router CeventsEvents | Router EeventsEvents | |conditionCondition | | | |+-----------+------------+-------------------+----------------------++------------+--------+----------------------+----------------------+ | S->D | | | | | Traffic| | | | |OK | | | | | | | | | | S->D | t0 | Link B-C fails | Link B-C fails | | Traffic | | | | | lost | | | | | | | | | | |t0+20msect0+20 | C detects the | | | | ms | failure | | | | | | | | S->D |t0+40msect0+40 | C activates FRR | | | Traffic| | | | |OK | ms | | | | | | | | | |t0+50msect0+50 | C updates its local | | | | ms |localLSP/LSA | | | | | | | | |t0+60msect0+55 | Cschedules SPFfloods its local | | | | ms |(100ms)updated LSP/LSA | | | | | | | | |t0+67msect0+57 | Creceivesschedules SPF (100 | | | | ms |LSP/LSA from Bms) | | | | | | | | |t0+70msect0+67 | Cfloods its | | | | | local updatedreceives LSP/LSA | | | | ms |LSP/LSAfrom B and floods it | | | | | | | | |t0+87msect0+87 | | E receives LSP/LSA | | | ms | | from C andschedulesfloods it | | | | |SPF (100ms)| | | t0+90 | | E schedules SPF (100 | | |t0+117msecms | |E floods LSP/LSAms) | | | | |from C| | | t0+160 | C computes SPF | | | |t0+160msecms |C computes SPF| | | | | | | | |t0+165msect0+165 | C delays its RIB/FIB | | | | ms |RIB/FIBupdate (2 sec) | | | | |sec)| | | | t0+193 | | E computes SPF | | |t0+193msecms | |E computes SPF| | | | | | | |t0+199msect0+199 | | E starts updating | | | ms | | its RIB/FIB | | | | | | | |t0+254msect0+254 | Link F-X fails | Link F-X fails | | | ms | | | | |t0+300msec|C receives| | | | t0+300 | C receives LSP/LSAfrom F| | | | ms | from F and floods it | | | | | | | | |t0+303msect0+303 | C schedules SPF (200 | | | | ms |(200ms)ms) | | | | | | | | |t0+312msect0+312 | E receives LSP/LSA | | | | ms |LSP/LSAfrom F| | | | |and floods it | | | | | | | | |t0+313msect0+313 | E schedules SPF (200 | | | | ms |(200ms)ms) | | | | | | | | |t0+502msect0+502 | C computes SPF | | | | ms | | | | |t0+505msec| | | | | t0+505 | C starts updating | | | | ms | its RIB/FIB (NO | | | | | DELAY) | | | | | | | | |t0+514msect0+514 | | E computes SPF | | | ms | | | | | |t0+519msec| | | | t0+519 | | E starts updating | | | ms | | its RIB/FIB | | | | | | | S->D |t0+659msect0+659 | C updates its | | | Traffic | ms | RIB/FIB for D | | | lost | | | | | | | | | | S->D |t0+778msect0+778 | | E updates its | | Traffic OK | ms | | RIB/FIB for D | |OK | | || | | | | || | t0+781msect0+781 | C convergence ends | | | | ms |ends| | | | | | | | |t0+810msect0+810 | | E convergence ends |+-----------+------------+-------------------+----------------------+| | ms | | | +------------+--------+----------------------+----------------------+ Table 6- Route computation event time scale10. Comparison withother solutionsOther Solutions As stated in Section 4, theproposedlocal delay solution reuses some concepts already introduced by other IETF proposals but tries to find atradeofftrade- off between efficiency and simplicity. This section tries to compare behaviors of the solutions. 10.1. PLSN PLSN([I-D.ietf-rtgwg-microloop-analysis])[PLSN] describes a mechanism where each node in the network tries to avoid transient forwarding loops upon a topology change by always keeping traffic on a loop-free path for a defined duration (locked path to a safe neighbor). The locked path may be the new primarynexthop,next hop, another neighbor, or the old primarynexthopnext hop depending on how the safety condition is satisfied. PLSN does not solve all transient forwarding loops (see[I-D.ietf-rtgwg-microloop-analysis]Section 4 of [PLSN] for more details).OurThe solution defined in this document reuses someconceptconcepts of PLSN but in a more simple fashion: o PLSN has three different behaviors: (1) keep using the oldnexthop,next hop, (2) use the new primarynexthopnext hop if it is safe, or (3) use another safenexthop, while the proposed solutionnext hop. The local delay solution, however, only has one: keep using the currentnexthop (old primary,next hop (i.e., the old primary next hop oralready activatedan already-activated FRR path). o PLSN may cause some damage while using a safenexthop whichnext hop that is not the new primarynexthop in casenext hop if the new safenexthopnext hop does not provide enough bandwidth (see [RFC7916]).ThisThe solution defined in this document may not experience this issue as the service provider may have control on the FRR path beingusedused, preventing network congestion. o PLSN applies to all nodes in a network (remote or local changes), while theproposedmechanism defined in this document applies onlyonto the nodes connected to the topology change. 10.2.OFIB OFIB ([RFC6976])oFIB oFIB [RFC6976] describes a mechanism where the convergence of the network upon a topology change is ordered in order to prevent transient forwarding loops. Each router in the networkmust deducededuces the failure type from the LSA/LSP received and computes/applies a specific FIB update timer based on the failure type and its rank in thenetworknetwork, considering the failure point as root.ThisThe oFIB mechanismallows to solvesolves all the transient forwardinglooploops in a network at the price of introducing complexity in the convergence process that may requirea strongcareful monitoring by the service provider.OurThe solution defined in this document reuses theOFIBoFIB concept but limits it to the first hop that experiences the topology change. As demonstrated, the mechanismproposeddefined in this document allowsto solveall the local transient forwarding loopsthat represents anto be solved; these represent a high percentage of all the loops.MoreoverMoreover, limitingthe mechanismto one hop allowsto keep thenetwork-wide convergencebehavior. 11. Implementation Status At this time, there are three different implementations of this mechanism. o Implementation 1: * Organization: Cisco * Implementation name: Local Microloop Protection * Operating system: IOS-XE * Level of maturity: production release * Coverage: all the specification is implemented * Protocols supported: ISIS and OSPF * Implementation experience: tested in lab and works as expected * Comment: the feature gives the ability to choose to apply the delay to FRR protected entry only * Report last update: 10-11-2017 o Implementation 2: * Organization: Cisco * Implementation name: Local Microloop Protection * Operating system: IOS-XR * Level of maturity: deployed * Coverage: all the specification is implemented * Protocols supported: ISIS and OSPF * Implementation experience: deployed and works as expected * Comment: the feature gives the ability to choose to apply the delay to FRR protected entry only * Report last update: 10-11-2017 o Implementation 3: * Organization: Juniper Networks * Implementation name: Microloop avoidance when IS-IS link fails * Operating system: JUNOS * Level of maturity: deployed (hidden command) * Coverage: all the specification is implemented * Protocols supported: ISIS only * Implementation experience: deployed and works as expected * Comment: the feature appliesbehavior toall the ISIS routes * Report last update: 10-11-2017be kept. 11. IANA Considerations This document has no IANA actions. 12. Security Considerations This document does not introduce any change intermterms of IGP security. The operation is internal to the router. The local delay does not increase the number of attack vectors as an attacker could only trigger this mechanism ifheit already hasbethe ability to disable or enable an IGP link. The local delay does not increase the negative consequences. If an attacker has the ability to disable or enable an IGP link, it can already harm the network by creating instability and harm the traffic by creating forwarding packet loss and forwarding loss for the traffic crossing that link.14. IANA Considerations This document has no actions for IANA. 15.13. References15.1.13.1. Normative References [ISO10589]"IntermediateInternational Organization for Standardization, "Information technology -- Telecommunications and information exchange between systems -- Intermediate System to Intermediate System intra-domain routeing information exchange protocol for use in conjunction with the protocol for providing the connectionless-mode network service (ISO 8473)",ISO 10589,ISO/IEC 10589:2002, Second Edition, November 2002. [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/info/rfc2119>. [RFC2328] Moy, J., "OSPF Version 2", STD 54, RFC 2328, DOI 10.17487/RFC2328, April 1998, <https://www.rfc-editor.org/info/rfc2328>.15.2.[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/info/rfc8174>. 13.2. Informative References[I-D.ietf-rtgwg-backoff-algo][BACKOFF] Decraene, B., Litkowski, S., Gredler, H., Lindem, A., Francois, P., and C. Bowers, "SPF Back-off Delay algorithm for link state IGPs",draft-ietf-rtgwg-backoff-algo-06 (workWork inprogress), October 2017. [I-D.ietf-rtgwg-microloop-analysis]Progress, draft-ietf-rtgwg- backoff-algo-10, March 2018. [PLSN] Zinin, A., "Analysis and Minimization of Microloops in Link-state Routing Protocols",draft-ietf-rtgwg-microloop- analysis-01 (workWork inprogress),Progress, draft-ietf-rtgwg-microloop-analysis-01, October 2005. [RFC3906] Shen, N. and H. Smit, "Calculating Interior Gateway Protocol (IGP) Routes Over Traffic Engineering Tunnels", RFC 3906, DOI 10.17487/RFC3906, October 2004, <https://www.rfc-editor.org/info/rfc3906>. [RFC5715] Shand, M. and S. Bryant, "A Framework for Loop-Free Convergence", RFC 5715, DOI 10.17487/RFC5715, January 2010, <https://www.rfc-editor.org/info/rfc5715>. [RFC6976] Shand, M., Bryant, S., Previdi, S., Filsfils, C., Francois, P., and O. Bonaventure, "Framework for Loop-Free Convergence Using the Ordered Forwarding Information Base (oFIB) Approach", RFC 6976, DOI 10.17487/RFC6976, July 2013, <https://www.rfc-editor.org/info/rfc6976>. [RFC7916] Litkowski, S., Ed., Decraene, B., Filsfils, C., Raza, K., Horneffer, M., and P. Sarkar, "Operational Management of Loop-Free Alternates", RFC 7916, DOI 10.17487/RFC7916, July 2016, <https://www.rfc-editor.org/info/rfc7916>.13.Acknowledgements We would like tothanksthank the authors of [RFC6976] for introducing the concept of ordered convergence: Mike Shand, Stewart Bryant, Stefano Previdi, and Olivier Bonaventure. Authors' Addresses Stephane Litkowski Orange Email: stephane.litkowski@orange.com Bruno Decraene Orange Email: bruno.decraene@orange.com Clarence Filsfils Cisco Systems Email: cfilsfil@cisco.com Pierre Francois Individual Contributor Email: pfrpfr@gmail.com