6.1. Initialization of a New TCB

TCBs for new connections can be initialized using cached context from past connections as follows:¶

(1)

Note that PMTU is cached at the IP layer [RFC1191] [RFC4821].¶

(2)

Note that some values are not cached when they are computed locally (MMS_R) or indicated in the connection itself (MMS_S in the SYN).¶

Table 2 gives an overview of option-specific information that can be shared. Additional information on some specific TCP options and sharing is provided in Appendix B.¶

6.2. Updates to the TCB Cache

During a connection, the TCB cache can be updated based on events of current connections and their TCBs as they progress over time, as shown in Table 3.¶

(1)

Note that PMTU is cached at the IP layer [RFC1191] [RFC4821].¶

Merge() is the function that combines the current and previous (old) values and may vary for each parameter of the TCB cache. The particular function is not specified in this document; examples include windowed averages (mean of the past N values, for some N) and exponential decay (new = (1-alpha)*old + alpha *new, where alpha is in the range [0..1]).¶

Table 4 gives an overview of option-specific information that can be similarly shared. The TFO cookie is maintained until the client explicitly requests it be updated as a separate event.¶

6.3. Discussion

As noted, there is no particular benefit to caching MMS_S and MMS_R as these are reported by the local IP stack. Caching sendMSS and PMTU is trivial; reported values are cached (PMTU at the IP layer), and the most recent values are used. The cache is updated when the MSS option is received in a SYN or after PMTUD (i.e., when an ICMPv4 Fragmentation Needed [RFC1191] or ICMPv6 Packet Too Big message is received [RFC8201] or the equivalent is inferred, e.g., as from PLPMTUD [RFC4821]), respectively, so the cache always has the most recent values from any connection. For sendMSS, the cache is consulted only at connection establishment and not otherwise updated, which means that MSS options do not affect current connections. The default sendMSS is never saved; only reported MSS values update the cache, so an explicit override is required to reduce the sendMSS. Cached sendMSS affects only data sent in the SYN segment, i.e., during client connection initiation or during simultaneous open; the MSS of all other segments are constrained by the value updated as included in the SYN.¶

RTT values are updated by formulae that merge the old and new values, as noted in Section 6.2. Dynamic RTT estimation requires a sequence of RTT measurements. As a result, the cached RTT (and its variation) is an average of its previous value with the contents of the currently active TCB for that host, when a TCB is closed. RTT values are updated only when a connection is closed. The method for merging old and current values needs to attempt to reduce the transient effects of the new connections.¶

The updates for RTT, RTTVAR, and ssthresh rely on existing information, i.e., old values. Should no such values exist, the current values are cached instead.¶

TCP options are copied or merged depending on the details of each option. For example, TFO state is updated when a connection is established and read before establishing a new connection.¶

Sections 8 and 9 discuss compatibility issues and implications of sharing the specific information listed above. Section 10 gives an overview of known implementations.¶

Most cached TCB values are updated when a connection closes. The exceptions are MMS_R and MMS_S, which are reported by IP [RFC1122]; PMTU, which is updated after Path MTU Discovery and also reported by IP [RFC1191] [RFC4821] [RFC8201]; and sendMSS, which is updated if the MSS option is received in the TCP SYN header.¶

Sharing sendMSS information affects only data in the SYN of the next connection, because sendMSS information is typically included in most TCP SYN segments. Caching PMTU can accelerate the efficiency of PMTUD but can also result in black-holing until corrected if in error. Caching MMS_R and MMS_S may be of little direct value as they are reported by the local IP stack anyway.¶

The way in which state related to other TCP options can be shared depends on the details of that option. For example, TFO state includes the TCP Fast Open cookie [RFC7413] or, in case TFO fails, a negative TCP Fast Open response. RFC 7413 states,¶

The client MUST cache negative responses from the server in order to avoid potential connection failures. Negative responses include the server not acknowledging the data in the SYN, ICMP error messages, and (most importantly) no response (SYN-ACK) from the server at all, i.e., connection timeout.¶

TFOinfo is cached when a connection is established.¶

State related to other TCP options might not be as readily cached. For example, TCP-AO [RFC5925] success or failure between a host-pair for a single SYN destination port might be usefully cached. TCP-AO success or failure to other SYN destination ports on that host-pair is never useful to cache because TCP-AO security parameters can vary per service.¶

7.1. Initialization of a New TCB

TCBs for new connections can be initialized using cached context from concurrent connections as follows:¶

(1)

Note that PMTU is cached at the IP layer [RFC1191] [RFC4821].¶

In Table 5, the cached sum() is a total across all active connections because these parameters act in aggregate; similarly, f() is a function that updates that sum based on the new connection's values, represented as "N".¶

Table 6 gives an overview of option-specific information that can be similarly shared. Again, the TFO_cookie is updated upon explicit client request, which is a separate event.¶

7.2. Updates to the TCB Cache

During a connection, the TCB cache can be updated based on changes to concurrent connections and their TCBs, as shown below:¶

+

Note that the PMTU is cached at the IP layer [RFC1191] [RFC4821].¶

In Table 7, rtt_update() is the function used to combine old and current values, e.g., as a windowed average or exponentially decayed average.¶

Table 8 gives an overview of option-specific information that can be similarly shared.¶

7.3. Discussion

For ensemble sharing, TCB information should be cached as early as possible, sometimes before a connection is closed. Otherwise, opening multiple concurrent connections may not result in TCB data sharing if no connection closes before others open. The amount of work involved in updating the aggregate average should be minimized, but the resulting value should be equivalent to having all values measured within a single connection. The function "rtt_update" in Table 7 indicates this operation, which occurs whenever the RTT would have been updated in the individual TCP connection. As a result, the cache contains the shared RTT variables, which no longer need to reside in the TCB.¶

Congestion window size and ssthresh aggregation are more complicated in the concurrent case. When there is an ensemble of connections, we need to decide how that ensemble would have shared these variables, in order to derive initial values for new TCBs.¶

Sections 8 and 9 discuss compatibility issues and implications of sharing the specific information listed above.¶

There are several ways to initialize the congestion window in a new TCB among an ensemble of current connections to a host. Current TCP implementations initialize it to 4 segments as standard [RFC3390] and 10 segments experimentally [RFC6928]. These approaches assume that new connections should behave as conservatively as possible. The algorithm described in [Ba12] adjusts the initial cwnd depending on the cwnd values of ongoing connections. It is also possible to use sharing mechanisms over long timescales to adapt TCP's initial window automatically, as described further in Appendix C.¶

8.1. Traversing the Same Network Path

TCP is sometimes used in situations where packets of the same host-pair do not always take the same path, such as when connection-specific parameters are used for routing (e.g., for load balancing). Multipath routing that relies on examining transport headers, such as ECMP and Link Aggregation Group (LAG) [RFC7424], may not result in repeatable path selection when TCP segments are encapsulated, encrypted, or altered -- for example, in some Virtual Private Network (VPN) tunnels that rely on proprietary encapsulation. Similarly, such approaches cannot operate deterministically when the TCP header is encrypted, e.g., when using IPsec Encapsulating Security Payload (ESP) (although TCB interdependence among the entire set sharing the same endpoint IP addresses should work without problems when the TCP header is encrypted). Measures to increase the probability that connections use the same path could be applied; for example, the connections could be given the same IPv6 flow label [RFC6437]. TCB interdependence can also be extended to sets of host IP address pairs that share the same network path conditions, such as when a group of addresses is on the same LAN (see Section 9).¶

Traversing the same path is not important for host-specific information (e.g., rwnd), TCP option state (e.g., TFOinfo), or for information that is already cached per-host (e.g., path MTU). When TCB information is shared across different SYN destination ports, path-related information can be incorrect; however, the impact of this error is potentially diminished if (as discussed here) TCB sharing affects only the transient event of a connection start or if TCB information is shared only within connections to the same SYN destination port.¶

In the case of temporal sharing, TCB information could also become invalid over time, i.e., indicating that although the path remains the same, path properties have changed. Because this is similar to the case when a connection becomes idle, mechanisms that address idle TCP connections (e.g., [RFC7661]) could also be applied to TCB cache management, especially when TCP Fast Open is used [RFC7413].¶

8.2. State Dependence

There may be additional considerations to the way in which TCB interdependence rebalances congestion feedback among the current connections. For example, it may be appropriate to consider the impact of a connection being in Fast Recovery [RFC5681] or some other similar unusual feedback state that could inhibit or affect the calculations described herein.¶

8.3. Problems with Sharing Based on IP Address

It can be wrong to share TCB information between TCP connections on the same host as identified by the IP address if an IP address is assigned to a new host (e.g., IP address spinning, as is used by ISPs to inhibit running servers). It can be wrong if Network Address Translation (NAT) [RFC2663], Network Address and Port Translation (NAPT) [RFC2663], or any other IP sharing mechanism is used. Such mechanisms are less likely to be used with IPv6. Other methods to identify a host could also be considered to make correct TCB sharing more likely. Moreover, some TCB information is about dominant path properties rather than the specific host. IP addresses may differ, yet the relevant part of the path may be the same.¶

9.1. Layering

TCB interdependence pushes some of the TCP implementation from its typical placement solely within the transport layer (in the ISO model) to the network layer. This acknowledges that some components of state are, in fact, per-host-pair or can be per-path as indicated solely by that host-pair. Transport protocols typically manage per-application-pair associations (per stream), and network protocols manage per-host-pair and path associations (routing). Round-trip time, MSS, and congestion information could be more appropriately handled at the network layer, aggregated among concurrent connections, and shared across connection instances [RFC3124].¶

An earlier version of RTT sharing suggested implementing RTT state at the IP layer rather than at the TCP layer. Our observations describe sharing state among TCP connections, which avoids some of the difficulties in an IP-layer solution. One such problem of an IP-layer solution is determining the correspondence between packet exchanges using IP header information alone, where such correspondence is needed to compute RTT. Because TCB sharing computes RTTs inside the TCP layer using TCP header information, it can be implemented more directly and simply than at the IP layer. This is a case where information should be computed at the transport layer but could be shared at the network layer.¶

9.2. Other Possibilities

Per-host-pair associations are not the limit of these techniques. It is possible that TCBs could be similarly shared between hosts on a subnet or within a cluster, because the predominant path can be subnet-subnet rather than host-host. Additionally, TCB interdependence can be applied to any protocol with congestion state, including SCTP [RFC4960] and DCCP [RFC4340], as well as to individual subflows in Multipath TCP [RFC8684].¶

There may be other information that can be shared between concurrent connections. For example, knowing that another connection has just tried to expand its window size and failed, a connection may not attempt to do the same for some period. The idea is that existing TCP implementations infer the behavior of all competing connections, including those within the same host or subnet. One possible optimization is to make that implicit feedback explicit, via extended information associated with the endpoint IP address and its TCP implementation, rather than per-connection state in the TCB.¶

This document focuses on sharing TCB information at connection initialization. Subsequent to RFC 2140, there have been numerous approaches that attempt to coordinate ongoing state across concurrent connections, both within TCP and other congestion-reactive protocols, which are summarized in [Is18]. These approaches are more complex to implement, and their comparison to steady-state TCP equivalence can be more difficult to establish, sometimes intentionally (i.e., they sometimes intend to provide a different kind of "fairness" than emerges from TCP operation).¶

C.1. Introduction

Temporal sharing, as described earlier in this document, builds on the assumption that multiple consecutive connections between the same host-pair are somewhat likely to be exposed to similar environment characteristics. The stored information can become less accurate over time and suitable precautions should take this aging into consideration (this is discussed further in Section 8.1). However, there are also cases where it can make sense to track these values over longer periods, observing properties of TCP connections to gradually influence evolving trends in TCP parameters. This appendix describes an example of such a case.¶

TCP's congestion control algorithm uses an initial window value (IW) both as a starting point for new connections and as an upper limit for restarting after an idle period [RFC5681] [RFC7661]. This value has evolved over time; it was originally 1 maximum segment size (MSS) and increased to the lesser of 4 MSSs or 4,380 bytes [RFC3390] [RFC5681]. For a typical Internet connection with a maximum transmission unit (MTU) of 1500 bytes, this permits 3 segments of 1,460 bytes each.¶

The IW value was originally implied in the original TCP congestion control description and documented as a standard in 1997 [RFC2001] [Ja88]. The value was updated in 1998 experimentally and moved to the Standards Track in 2002 [RFC2414] [RFC3390]. In 2013, it was experimentally increased to 10 [RFC6928].¶

This appendix discusses how TCP can objectively measure when an IW is too large and that such feedback should be used over long timescales to adjust the IW automatically. The result should be safer to deploy and might avoid the need to repeatedly revisit IW over time.¶

Note that this mechanism attempts to make the IW more adaptive over time. It can increase the IW beyond that which is currently recommended for wide-scale deployment, so its use should be carefully monitored.¶

C.2. Design Considerations

TCP's IW value has existed statically for over two decades, so any solution to adjusting the IW dynamically should have similarly stable, non-invasive effects on the performance and complexity of TCP. In order to be fair, the IW should be similar for most machines on the public Internet. Finally, a desirable goal is to develop a self-correcting algorithm so that IW values that cause network problems can be avoided. To that end, we propose the following design goals:¶

• Impart little to no impact to TCP in the absence of loss, i.e., it should not increase the complexity of default packet processing in the normal case.¶
• Adapt to network feedback over long timescales, avoiding values that persistently cause network problems.¶
• Decrease the IW in the presence of sustained loss of IW segments, as determined over a number of different connections.¶
• Increase the IW in the absence of sustained loss of IW segments, as determined over a number of different connections.¶
• Operate conservatively, i.e., tend towards leaving the IW the same in the absence of sufficient information, and give greater consideration to IW segment loss than IW segment success.¶

We expect that, without other context, a good IW algorithm will converge to a single value, but this is not required. An endpoint with additional context or information, or deployed in a constrained environment, can always use a different value. In particular, information from previous connections, or sets of connections with a similar path, can already be used as context for such decisions (as noted in the core of this document).¶

However, if a given IW value persistently causes packet loss during the initial burst of packets, it is clearly inappropriate and could be inducing unnecessary loss in other competing connections. This might happen for sites behind very slow boxes with small buffers, which may or may not be the first hop.¶

C.3. Proposed IW Algorithm

Below is a simple description of the proposed IW algorithm. It relies on the following parameters:¶

• MinIW = 3 MSS or 4,380 bytes (as per [RFC3390])¶
• MaxIW = 10 MSS (as per [RFC6928])¶
• MulDecr = 0.5¶
• AddIncr = 2 MSS¶
• Threshold = 0.05¶

We assume that the minimum IW (MinIW) should be as currently specified as standard [RFC3390]. The maximum IW (MaxIW) can be set to a fixed value (we suggest using the experimental and now somewhat de facto standard in [RFC6928]) or set based on a schedule if trusted time references are available [Al10]; here, we prefer a fixed value. We also propose to use an Additive Increase Multiplicative Decrease (AIMD) algorithm, with increase and decreases as noted.¶

Although these parameters are somewhat arbitrary, their initial values are not important except that the algorithm is AIMD and the MaxIW should not exceed that recommended for other systems on the Internet (here, we selected the current de facto standard rather than the actual standard). Current proposals, including default current operation, are degenerate cases of the algorithm below for given parameters, notably MulDec = 1.0 and AddIncr = 0 MSS, thus disabling the automatic part of the algorithm.¶

The proposed algorithm is as follows:¶

1. On boot:¶

      IW = MaxIW; # assume this is in bytes and indicates an integer
                  # multiple of 2 MSS (an even number to support
                  # ACK compression)
   

   ¶

2. Upon starting a new connection:¶

      CWND = IW;
      conncount++;
      IWnotchecked = 1; # true
   

   ¶

3. During a connection's SYN-ACK processing, if SYN-ACK includes ECN (as similarly addressed in Section 5 of ECN++ for TCP [Ba20]), treat as if the IW is too large:¶

      if (IWnotchecked && (synackecn == 1)) {
         losscount++;
         IWnotchecked = 0; # never check again
      }
   

   ¶

4. During a connection, if retransmission occurs, check the seqno of the outgoing packet (in bytes) to see if the re-sent segment fixes an IW loss:¶

      if (Retransmitting && IWnotchecked && ((seqno - ISN) < IW))) {
         losscount++;
         IWnotchecked = 0; # never do this entire "if" again
      } else {
         IWnotchecked = 0; # you're beyond the IW so stop checking
      }
   

   ¶

5. Once every 1000 connections, as a separate process (i.e., not as part of processing a given connection):¶

      if (conncount > 1000) {
         if (losscount/conncount > threshold) {
            # the number of connections with errors is too high
            IW = IW * MulDecr;
         } else {
            IW = IW + AddIncr;
         }
      }
   

   ¶

As presented, this algorithm can yield a false positive when the sequence number wraps around, e.g., the code might increment losscount in step 4 when no loss occurred or fail to increment losscount when a loss did occur. This can be avoided using either Protection Against Wrapped Sequences (PAWS) [RFC7323] context or internal extended sequence number representations (as in TCP Authentication Option (TCP-AO) [RFC5925]). Alternately, false positives can be tolerated because they are expected to be infrequent and thus will not significantly impact the algorithm.¶

A number of additional constraints need to be imposed if this mechanism is implemented to ensure that it defaults to values that comply with current Internet standards, is conservative in how it extends those values, and returns to those values in the absence of positive feedback (i.e., success). To that end, we recommend the following list of example constraints:¶

• The automatic IW algorithm MUST initialize MaxIW a value no larger than the currently recommended Internet default in the absence of other context information.¶

  Thus, if there are too few connections to make a decision or if there is otherwise insufficient information to increase the IW, then the MaxIW defaults to the current recommended value.¶

• An implementation MAY allow the MaxIW to grow beyond the currently recommended Internet default but not more than 2 segments per calendar year.¶

  Thus, if an endpoint has a persistent history of successfully transmitting IW segments without loss, then it is allowed to probe the Internet to determine if larger IW values have similar success. This probing is limited and requires a trusted time source; otherwise, the MaxIW remains constant.¶

• An implementation MUST adjust the IW based on loss statistics at least once every 1000 connections.¶

  An endpoint needs to be sufficiently reactive to IW loss.¶

• An implementation MUST decrease the IW by at least 1 MSS when indicated during an evaluation interval.¶

  An endpoint that detects loss needs to decrease its IW by at least 1 MSS; otherwise, it is not participating in an automatic reactive algorithm.¶

• An implementation MUST increase by no more than 2 MSSs per evaluation interval.¶

  An endpoint that does not experience IW loss needs to probe the network incrementally.¶

• An implementation SHOULD use an IW that is an integer multiple of 2 MSSs.¶

  The IW should remain a multiple of 2 MSS segments to enable efficient ACK compression without incurring unnecessary timeouts.¶

• An implementation MUST decrease the IW if more than 95% of connections have IW losses.¶

  Again, this is to ensure an implementation is sufficiently reactive.¶

• An implementation MAY group IW values and statistics within subsets of connections. Such grouping MAY use any information about connections to form groups except loss statistics.¶

There are some TCP connections that might not be counted at all, such as those to/from loopback addresses or those within the same subnet as that of a local interface (for which congestion control is sometimes disabled anyway). This may also include connections that terminate before the IW is full, i.e., as a separate check at the time of the connection closing.¶

The period over which the IW is updated is intended to be a long timescale, e.g., a month or so, or 1,000 connections, whichever is longer. An implementation might check the IW once a month and simply not update the IW or clear the connection counts in months where the number of connections is too small.¶

C.4. Discussion

There are numerous parameters to the above algorithm that are compliant with the given requirements; this is intended to allow variation in configuration and implementation while ensuring that all such algorithms are reactive and safe.¶

This algorithm continues to assume segments because that is the basis of most TCP implementations. It might be useful to consider revising the specifications to allow byte-based congestion given sufficient experience.¶

The algorithm checks for IW losses only during the first IW after a connection start; it does not check for IW losses elsewhere the IW is used, e.g., during slow-start restarts.¶

• An implementation MAY detect IW losses during slow-start restarts in addition to losses during the first IW of a connection. In this case, the implementation MUST count each restart as a "connection" for the purposes of connection counts and periodic rechecking of the IW value.¶

False positives can occur during some kinds of segment reordering, e.g., that might trigger spurious retransmissions even without a true segment loss. These are not expected to be sufficiently common to dominate the algorithm and its conclusions.¶

This mechanism does require additional per-connection state, which is currently common in some implementations and is useful for other reasons (e.g., the ISN is used in TCP-AO [RFC5925]). The mechanism in this appendix also benefits from persistent state kept across reboots, which would also be useful to other state sharing mechanisms (e.g., TCP Control Block Sharing per the main body of this document).¶

The receive window (rwnd) is not involved in this calculation. The size of rwnd is determined by receiver resources and provides space to accommodate segment reordering. Also, rwnd is not involved with congestion control, which is the focus of the way this appendix manages the IW.¶

C.5. Observations

The IW may not converge to a single global value. It also may not converge at all but rather may oscillate by a few MSSs as it repeatedly probes the Internet for larger IWs and fails. Both properties are consistent with TCP behavior during each individual connection.¶

This mechanism assumes that losses during the IW are due to IW size. Persistent errors that drop packets for other reasons, e.g., OS bugs, can cause false positives. Again, this is consistent with TCP's basic assumption that loss is caused by congestion and requires backoff. This algorithm treats the IW of new connections as a long-timescale backoff system.¶