]> Private Octopus Inc.

huitema@huitema.net

Cisco

snandaku@cisco.com

Cisco

fluffy@iii.ca

Web and Internet Transport C4 Congestion Control Realtime Communication Media over QUIC Christian's Congestion Control Code is a new congestion control algorithm designed to support Real-Time applications such as Media over QUIC. It is designed to drive towards low delays, with good support for the "application limited" behavior frequently found when using variable rate encoding, and with fast reaction to congestion to avoid the "priority inversion" happening when congestion control overestimates the available capacity. It pays special attention to the high jitter conditions encountered in Wi-Fi networks. The design emphasizes simplicity and avoids making too many assumption about the "model" of the network. The main control variables are the estimate of the data rate and of the maximum path delay in the absence of queues.

Introduction Christian's Congestion Control Code (C4) is a new congestion control algorithm designed to support Real-Time multimedia applications, specifically multimedia applications using QUIC and the Media over QUIC transport . These applications require low delays, and often exhibit a variable data rate as they alternate high bandwidth requirements when sending reference frames and lower bandwith requirements when sending differential frames. We translate that into 3 main goals: Drive towards low delays (see ), Support "application limited" behavior (see ), React quickly to changing network conditions (see ). The design of C4 is inspired by our experience using different congestion control algorithms for QUIC, notably Cubic , Hystart , and BBR , as well as the study of delay-oriented algorithms such as TCP Vegas and LEDBAT . In addition, we wanted to keep the algorithm simple and easy to implement. C4 assumes that the transport stack is capable of signaling to the congestion algorithms events such as acknowledgements, RTT measurements, ECN signals or the detection of packet losses. It also assumes that the congestion algorithm controls the transport stack by setting the congestion window (CWND) and the pacing rate. C4 tracks the state of the network by keeping a small set of variables, the main ones being the "nominal rate", the "nominal max RTT", and the current state of the algorithm. The details on using and tracking the min RTT are discussed in . The nominal rate is the pacing rate corresponding to the most recent estimate of the bandwidth available to the connection. The nominal max RTT is the best estimate of the maximum RTT that can occur on the network in the absence of queues. When we do not observe delay jitter, this coincides with the min RTT. In the presence of jitter, it should be the sum of the min RTT and the maximum jitter. C4 will compute a pacing rate as the nominal rate multiplied by a coefficient that depends on the state of the protocol, and set the CWND for the path to the product of that pacing rate by the max RTT. The design of these mechanisms is discussed in . Over time, we updated C4 to: Focus on the pacing rate as the control variable, and use an estimation of the largest path RTT to set the CWND, as explained in Improve the handling of congestion response, growth cascade and careful resume, as explained in

Studying the reaction to delays The current design of C4 is the result of a series of experiments. Our initial design was to monitor delays and react to dellay increases in much the same way as congestion control algorithms like TCP Vegas or LEDBAT: monitor the current RTT and the min RTT if the current RTT sample exceed the min RTT by more than a preset margin, treat that as a congestion signal. The "preset margin" is set by default to 10 ms in TCP Vegas and LEDBAT. That was adequate when these algorithms were designed, but it can be considered excessive in high speed low latency networks. For the initial C4 design, we set it to the lowest of 1/8th of the min RTT and 25ms. The min RTT itself is measured over time. The detection of congestion by comparing delays to min RTT plus margin works well, except in two conditions: if the C4 connection is competing with a another connection that does not react to delay variations, such as a connection using Cubic, if the network exhibits a lot of latency jitter, as happens on some Wi-Fi networks. We also know that if several connection using delay-based algorithms compete, the competition is only fair if they all have the same estimate of the min RTT. We handle that by using a "periodic slow down" mechanism.

Managing Competition with Loss Based Algorithms Competition between Cubic and a delay based algorithm leads to Cubic consuming all the bandwidth and the delay based connection starving. This phenomenon force TCP Vegas to only be deployed in controlled environments, in which it does not have to compete with TCP Reno or Cubic. We handled this competition issue by using a simple detection algorithm. If C4 detected competition with a loss based algorithm, it switched to a "pig war" mode and stopped reacting to changes in delays -- it would instead only react to packet losses and ECN signals. In that mode, we used another algorithm to detect when the competition has ceased, and switch back to the delay responsive mode. In our initial deployments, we detected competition when delay based congestion notifications leads to CWND and rate reduction for more than 3 consecutive RTT. The assumption is that if the competition reacted to delays variations, it would have reacted to the delay increases before 3 RTT. However, that simple test caused many "false positive" detections. We refined this test to start the pig war if we observed 4 consecutive delay-based rate reductions and the nominal CWND was less than half the max nominal CWND observed since the last "initial" phase, or if we observed at least 5 reductions and the nominal CWND is less than 4/5th of the max nominal CWND. We validated this test by comparing the ratio CWND/MAX_CWND for "valid" decisions, when we are simulating a competition scenario, and "spurious" decisions, when the "more than 3 consecutive reductions" test fires but we are not simulating any competition: Ratio CWND/Max valid spurious Average 30% 75% Max 49% 100% Top 25% 37% 91% Media 35% 83% Bottom 25% 20% 52% Min 12% 25% <50% 100% 20% Note that this validation was based on simulations, and that we cannot claim that our simulations perfectly reflect the real world. We will discuss in how this imperfections lead us to use change our overall design. Our initial exit competition algorithm was simple. C4 will exit the "pig war" mode if the available bandwidth increases.

Handling Chaotic Delays Some Wi-Fi networks exhibit spikes in latency. These spikes are probably what caused the delay jitter discussed in . We discussed them in more details in . We are not sure about the mechanism behind these spikes, but we have noticed that they mostly happen when several adjacent Wi-Fi networks are configured to use the same frequencies and channels. In these configurations, we expect the hidden node problem to result in some collisions. The Wi-Fi layer 2 retransmission algorithm takes care of these losses, but apparently uses an exponential back off algorithm to space retransmission delays in case of repeated collisions. When repeated collisions occur, the exponential backoff mechanism can cause large delays. The Wi-Fi layer 2 algorithm will also try to maintain delivery order, and subsequent packets will be queued behind the packet that caused the collisions. In our initial design, we detected the advent of such "chaotic delay jitter" by computing a running estimate of the max RTT. We measured the max RTT observed in each round trip, to obtain the "era max RTT". We then computed an exponentially averaged "nominal max RTT":

If the nominal max RTT was more than twice the min RTT, we set the "chaotic jitter" condition. When that condition was set, we stopped considering excess delay as an indication of congestion, and we changed the way we computed the "current CWND" used for the controlled path. Instead of simply setting it to "nominal CWND", we set it to a larger value:

In this formula, alpha is the amplification coefficient corresponding to the current state, such as for example 1 if "cruising" or 1.25 if "pushing" (see ), and max_bytes_acked is the largest amount of bytes in flight that was succesfully acknowledged since the last initial phase. The increased target_cwnd enabled C4 to keep sending data through most jitter events. There is of course a risk that this increased value will cause congestion. We limit that risk by only using half the value of max_bytes_ack, and by the setting a conservative pacing rate:

Using the pacing rate that way prevents the larger window to cause big spikes in traffic. The network conditions can evolve over time. C4 will keep monitoring the nominal max RTT, and will reset the "chaotic jitter" condition if nominal max RTT decreases below a threshold of 1.5 times the min RTT.

Monitor min RTT Delay based algorithm rely on a correct estimate of the min RTT. They will naturally discover a reduction in the min RTT, but detecting an increase in the max RTT is difficult. There are known failure modes when multiple delay based algorithms compete, in particular the "late comer advantage". In our initial design, the connections ensured that their min RTT is valid by occasionally entering a "slowdown" period, during which they set CWND to half the nominal value. This is similar to the "Probe RTT" mechanism implemented in BBR, or the "initial and periodic slowdown" proposed as extension to LEDBAT in . In our implementation, the slowdown occurs if more than 5 seconds have elapsed since the previous slowdown, or since the last time the min RTT was set. The measurement of min RTT in the period that follows the slowdown is considered a "clean" measurement. If two consecutive slowdown periods were followed by clean measurements larger than the current min RTT, we detect an RTT change and reset the connection. If the measurement results in the same value as the previous min RTT, C4 continue normal operation. Some applications exhibit periods of natural slow down. This is the case for example of multimedia applications, when they only send differentially encoded frames. Natural slowdown was detected if an application sent less than half the nominal CWND during a period, and more than 4 seconds had elapsed since the previous slowdown or the previous min RTT update. The measurement that follows a natural slowdown was also considered a clean measurement. A slowdown period corresponds to a reduction in offered traffic. If multiple connections are competing for the same bottleneck, each of these connections may experience cleaner RTT measurements, leading to equalization of the min RTT observed by these connections.

Simplifying the initial design After extensive testing of our initial design, we felt we had drifted away from our initial "simplicity" tenet. The algorithms used to detect "pig war" and "chaotic jitter" were difficult to tune, and despite our efforts they resulted in many false positive or false negative. The "slowdown" algorithm made C4 less friendly to "real time" applications that prefer using stable estimated rates. These algorithms interacted with each other in ways that were sometimes hard to predict.

Chaotic jitter and rate control As we observed the chaotic jitter behavior, we came to the conclusion that only controlling the CWND did not work well. we had a dilemma: either use a small CWND to guarantee that RTTs remain small, or use a large CWND so that transmission would not stall during peaks in jitter. But if we use a large CWND, we need some form of pacing to prevent senders from sending a large amount of packets too quickly. And then we realized that if we do have to set a pacing rate, we can simplify the algorithm. Suppose that we compute a pacing rate that matches the network capacity, just like BBR does. Then, in first approximation, the setting the CWND too high does not matter too much. The number of bytes in flight will be limited by the product of the pacing rate by the actual RTT. We are thus free to set the CWND to a large value.

Monitoring the nominal max RTT The observation on chaotic jitter leads to the idea of monitoring the maximum RTT. There is some difficulty here, because the observed RTT has three components: The minimum RTT in the absence of jitter The jitter caused by access networks such as Wi-Fi The delays caused by queues in the network We cannot merely use the maximum value of the observed RTT, because of the queing delay component. In pushing periods, we are going to use data rate slightly higher than the measured value. This will create a bit of queuing, pushing the queing delay component ever higher -- and eventually resulting in "buffer bloat". To avoid that, we can have periodic periods in which the endpoint sends data at deliberately slower than the rate estimate. This would enable a "clean" measurement of the Max RTT. However, tests showed that only measuring the Max RTT during recovery periods is not reactive enough. For example, if the underlying RTT changes, we would need to wait up to 6 RTT before registering the change. In practice, we can measure the Max RTT in both the "recovery" and "cruising" periods, i.e., all the periods in which data is sent at most at the "nominal data rate". If we are dealing with jitter, the clean Max RTT measurements will include whatever jitter was happening at the time of the measurement. It is not sufficient to measure the Max RTT once, we must keep the maximum value of a long enough series of measurement to capture the maximum jitter than the network can cause. But we are also aware that jitter conditions change over time, so we have to make sure that if the jitter diminished, the Max RTT also diminishes. We solved that by measuring the Max RTT during the "recovery" periods that follow every "push". These periods occur about every 6 RTT, giving us reasonably frequent measurements. During these periods, we try to ensure clean measurements by setting the pacing rate a bit lower than the nominal rate -- 6.25% slower in our initial trials. We apply the following algorithm: compute the max_rtt_sample as the maximum RTT observed for packets sent during the recovery period. if the max_rtt_sample is more than max_jitter above running_min_rtt, reset it to running_min_rtt + max_jitter (by default, max_jitter is set to 250ms). if max_rtt_sample is larger than nominal_max_rtt, set nominal_max_rtt to that value. else, set nominal_max_rtt to:

The gamma coefficient is set to 1/8 in our initial trials.

Preventing Runaway Max RTT Computing Max RTT the way we do bears the risk of "run away increase" of Max RTT: C4 notices high jitter, increases Nominal Max RTT accordingly, set CWND to the product of the increased Nominal Max RTT and Nominal Rate If Nominal rate is above the actual link rate, C4 will fill the pipe, and create a queue. On the next measurement, C4 finds that the max RTT has increased because of the queue, interprets that as "more jitter", increases Max RTT and fills the queue some more. Repeat until the queue become so large that packets are dropped and cause a congestion event. Our proposed algorithm limits the Max RTT to at most running_min_rtt + max_jitter, but that is still risky. If congestion causes queues, the running measurements of min RTT will increase, causing the algorithm to allow for corresponding increases in max RTT. This would not happen as fast as without the capping to running_min_rtt + max_jitter, but it would still increase.

Initial Phase and Max RTT During the initial phase, the nominal max RTT and the running min RTT are set to the first RTT value that is measured. This is not great in presence of high jitter, which causes C4 to exit the Initial phase early, leaving the nominal rate way too low. If C4 is competing on the Wi-Fi link against another connection, it might remain stalled at this low data rate. We considered updating the Max RTT during the Initial phase, but that prevents any detection of delay based congestion. The Initial phase would continue until path buffers are full, a classic case of buffer bloat. Instead, we adopted a simple workaround: Maintain a flag "initial_after_jitter", initialized to 0. Get a measure of the max RTT after exit from initial. If C4 detects a "high jitter" condition and the "initial_after_jitter" flag is still 0, set the flag to 1 and re-enter the "initial" state. Empirically, we detect high jitter in that case if the "running min RTT" is less that 2/5th of the "nominal max RTT".

Monitoring the nominal rate The nominal rate is measured on each acknowledgement by dividing the number of bytes acknowledged since the packet was sent by the RTT measured with the acknowledgement of the packet, protecting against delay jitter as explained in , without additional filtering as discussed in . We only use the measurements to increase the nominal rate, replacing the current value if we observe a greater filtered measurement. This is a deliberate choice, as decreases in measurement are ambiguous. They can result from the application being rate limited, or from measurement noises. Following those causes random decrease over time, which can be detrimental for rate limited applications. If the network conditions have changed, the rate will be reduced if congestion signals are received, as explained in .

Rate measurement The simple algorithm protects from underestimation of the delay by observing that delivery rates cannot be larger than the rate at which the packets were sent, thus keeping the lower of the estimated receive rate and the send rate. The algorithm uses four input variables: current_time: the time when the acknowledment is received. send_time: the time at which the highest acknowledged packet was sent. bytes_acknowledged: the number of bytes acknowledged by the receiver between send_time and current_time first_sent: the time at which the packet containing the first acknowledged bytes was sent. The computation goes as follow:

This is in line with the specification of rate measurement in . We use the data rate measurement to update the nominal rate, but only if not congested (see )

nominal_rate and not congested: nominal_rate = measured_rate ]]>

Avoiding Congestion Bounce In our early experiments, we observed a "congestion bounce" that happened as follow: congestion is detected, the nominal rate is reduced, and C4 enters recovery. packets sent at the data rate that caused the congestion continue to be acknowledged during recovery. if enough packets are acknowledged, they will cause a rate measurement close to the previous nominal rate. if C4 accepts this new nominal rate, the flow will bounce back to the previous transmission rate, erasing the effects of the congestion signal. Since we do not want that to happen, we specify that the nominal rate cannot be updated during congested periods, defined as: C4 is in "recovery" state, The recovery state was entered following a congestion signal, or a congestion signal was received since the beginning of the recovery era.

Not filtering the measurements There is some noise in the measurements of the data rate, and we protect against that noise by retaining the maximum of the ack_delay and the send_delay. During early experiments, we considered smoothing the measurements for eliminating that noise. The best filter that we could defined operated by smoothing the inverse of the data rate, the "time per byte sent". This works better because the data rate measurements are the quotient of the number of bytes received by the delay. The number of bytes received is easy to assert, but the measurement of the delays are very noisy. Instead of trying to average the data rates, we can average their inverse, i.e., the quotients of the delay by the bytes received, the times per byte. Then we can obtain smoothed data rates as the inverse of these times per byte, effectively computing an harmonic average of measurements over time. We could for example compute an exponentially weighted moving average of the time per byte, and use the inverse of that as a filtered measurement of the data rate. We do not specify any such filter in C4, because while filtering will reduce the noise, we will also delay any observation, resulting into a somewhat sluggish response to change in network conditions. Experience shows that the precaution of using the max of the ack delay and the send delay as a divider is sufficient for stable operation, and does not cause the response delays that filtering would.

Early Congestion Modification We want C4 to handle Early Congestion Notification in a manner compatible with the L4S design. For that, we monitor the evolving ratio of CE marks that the L4S specification designates as alpha (we use ecn_alpha here to avoid confusion), and we detect congestion if the ratio grows over a threshold. We did not find a recommended algorithm for computing ecn_alpha in either or , but we could get some concrete suggestions in . That draft, now obsolete, suggests updating the ratio once per RTT, as the exponential weighted average of the fraction of CE marks per packet:

This kind of averaging introduces a reaction delay. The draft suggests mitigating that delay by preempting the averaging if the fraction is large:

0.5: ecn_alpha = frac ]]>

We followed that design, but decided to update the coefficient after each acknowledgement, instead of after each RTT. This is in line with our implementation of "delayed acknowledgements" in QUIC, which results in a small number of acknowledgements per RTT. The reaction of C4 to an excess of CE marks is similar to the reaction to excess delays or to packet losses, see .

Competition with other algorithms We saw in that delay based algorithms required a special "escape mode" when facing competition from algorithms like Cubic. Relying on pacing rate and max RTT instead of CWND and min RTT makes this problem much simpler. The measured max RTT will naturally increase as algorithms like Cubic cause buffer bloat and increased queues. Instead of being shut down, C4 will just keep increasing its max RTT and thus its running CWND, automatically matching the other algorithm's values. We verified that behavior in a number of simulations. We also verified that when the competition ceases, C4 will progressively drop its nominal max RTT, returning to situations with very low queuing delays.

No need for slowdowns The fairness of delay based algorithm depends on all competing flows having similar estimates of the min RTT. As discussed in , this ends up creating variants of the latecomer advantage issue, requiring a periodic slowdown mechanism to ensure that all competing flow have chance to update the RTT value. This problem is caused by the default algorithm of setting min RTT to the minimum of all RTT sample values since the beginning of the connection. Flows that started more recently compute that minimum over a longer period, and thus discover a larger min RTT than older flows. This problem does not exist with max RTT, because all competing flows see the same max RTT value. The slowdown mechanism is thus not necessary. Removing the need for a slowdown mechanism allows for a simpler protocol, better suited to real time communications.

React quickly to changing network conditions Our focus is on maintaining low delays, and thus reacting quickly to changes in network conditions. We can detect some of these changes by monitoring the RTT and the data rate, but experience with the early version of BBR showed that completely ignoring packet losses can lead to very unfair competition with Cubic. The L4S effort is promoting the use of ECN feedback by network elements (see ), which could well end up detecting congestion and queues more precisely than the monitoring of end-to-end delays. C4 will thus detect changing network conditions by monitoring 3 congestion control signals: Excessive increase of measured RTT (above the nominal Max RTT), Excessive rate of packet losses (but not mere Probe Time Out, see ), Excessive rate of ECN/CE marks If any of these signals is detected, C4 enters a "recovery" state. On entering recovery, C4 reduces the nominal_rate by a factor "beta":

The cofficient beta differs depending on the nature of the congestion signal. For packet losses, it is set to 1/4, similar to the value used in Cubic. For delay based losses, it is proportional to the difference between the measured RTT and the target RTT divided by the acceptable margin, capped to 1/4. If the signal is an ECN/CE rate, we may use a proportional reduction coefficient in line with , again capped to 1/4. During the recovery period, target CWND and pacing rate are set to a fraction of the "nominal rate" multiplied by the "nominal max RTT". The recovery period ends when the first packet sent after entering recovery is acknowledged. Congestion signals are processed when entering recovery; further signals are ignored until the end of recovery. Network conditions may change for the better or for the worse. Worsening is detected through congestion signals, but increases can only be detected by trying to send more data and checking whether the network accepts it. Different algorithms have done two ways: pursuing regular increases of CWND until congestion finally occurs, like for example the "congestion avoidance" phase of TCP RENO; or periodically probe the network by sending at a higher rate, like the Probe Bandwidth mechanism of BBR. C4 adopt the periodic probing approach, in particular because it is a better fit for variable rate multimedia applications (see details in ).

Do not react to Probe Time Out QUIC normally detect losses by observing gaps in the sequences of acknowledged packet. That's a robust signal. QUIC will also inject "Probe time out" packets if the PTO timeout elapses before the last sent packet has not been acknowledged. This is not a robust congestion signal, because delay jitter may also cause PTO timeouts. When testing in "high jitter" conditions, we realized that we should not change the state of C4 for losses detected solely based on timer, and only react to those losses that are detected by gaps in acknowledgements.

Update the Nominal Rate after Pushing C4 configures the transport with a larger rate and CWND than the nominal CWND during "pushing" periods. The peer will acknowledge the data sent during these periods in the round trip that followed. When we receive an ACK for a newly acknowledged packet, we update the nominal rate as explained in . This strategy is effectively a form of "make before break". The pushing only increase the rate by a fraction of the nominal values, and only lasts for one round trip. That limited increase is not expected to increase the size of queues by more than a small fraction of the bandwidth*delay product. It might cause a slight increase of the measured RTT for a short period, or perhaps cause some ECN signaling, but it should not cause packet losses -- unless competing connections have caused large queues. If there was no extra capacity available, C4 does not increase the nominal CWND and the connection continues with the previous value.

Driving for fairness Many protocols enforce fairness by tuning their behavior so that large flows become less aggressive than smaller ones, either by trying less hard to increase their bandwidth or by reacting more to congestion events. We considered adopting a similar strategy for C4. The aggressiveness of C4 is driven by several considerations: the frequency of the "pushing" periods, the coefficient alpha used during pushing, the coefficient beta used during response to congestion events, the delay threshold above a nominal value to detect congestion, the ratio of packet losses considered excessive, the ratio of ECN marks considered excessive. We clearly want to have some or all of these parameters depend on how much resource the flow is using. There are know limits to these strategies. For example, consider TCP Reno, in which the growth rate of CWND during the "congestion avoidance" phase" is inversely proportional to its size. This drives very good long term fairness, but in practice it prevents TCP Reno from operating well on high speed or high delay connections, as discussed in the "problem description" section of . In that RFC, Sally Floyd was proposing using a growth rate inversely proportional to the logarithm of the CWND, which would not be so drastic. In the initial design, we proposed making the frequency of the pushing periods inversely proportional to the logarithm of the CWND, but that gets in tension with our estimation of the max RTT, which requires frequent "recovery" periods. We would not want the Max RTT estimate to work less well for high speed connections! We solved the tension in favor of reliable max RTT estimates, and fixed to 4 the number of Cruising periods between Recovery and Pushing. The whole cycle takes about 6 RTT. We also reduced the default rate increase during Pushing to 6.25%, which means that the default cycle is more on less on par with the aggressiveness of RENO when operating at low bandwidth (lower than 34 Mbps).

Absence of constraints is unfair Once we fixed the push frequency and the default increase rate, we were left with responses that were mostly proportional to the amount of resource used by a connection. Such design makes the resource sharing very dependent on initial conditions. We saw simulations where after some initial period, one of two competing connections on a 20 Mbps path might settle at a 15 Mbps rate and the other at 5 Mbps. Both connections would react to a congestion event by dropping their bandwidth by 25%, to 15 or 3.75 Mbps. And then once the condition eased, both would increase their data rate by the same amount. If everything went well the two connection will share the bandwidth without exceeding it, and the situation would be very stable -- but also very much unfair. We also had some simulations in which a first connection will grab all the available bandwidth, and a late comer connection would struggle to get any bandwidth at all. The analysis showed that the second connection was exiting the initial phase early, after encountering either excess delay or excess packet loss. The first connection was saturating the path, any additional traffic did cause queuing or losses, and the second connection had no chance to grow. This "second comer shut down" effect happened particularly often on high jitter links. The established connections had tuned their timers or congestion window to account for the high jitter. The second connection was basing their timers on their first measurements, before any of the big jitter events had occured. This caused an imbalance between the first connection, which expected large RTT variations, and the second, which did not expect them yet. These shutdown effects happened in simulations with the first connection using either Cubic, BBR or C4. We had to design a response, and we first turned to making the response to excess delay or packet loss a function of the data rate of the flow.

Introducing a sensitivity curve In our second design, we attempted to fix the unfairness and shutdowns effect by introducing a sensitivity curve, computing a "sensitivity" as a function of the flow data rate. Our first implementation is simple: set sensitivity to 0 if data rate is lower than 50000B/s linear interpolation between 0 and 0.92 for values between 50,000 and 1,000,000B/s. linear interpolation between 0.92 and 1 for values between 1,000,000 and 10,000,000B/s. set sensitivity to 1 if data rate is higher than 10,000,000B/s The sensitivity index is then used to set the value of delay and loss thresholds. For the delay threshold, the rule is:

For the loss threshold, the rule is:

For the CE mark threshold, the rule is:

This very simple change allowed us to stabilize the results. In our competition tests we see sharing of resource almost equitably between C4 connections, and reasonably between C4 and Cubic or C4 and BBR. We do not observe the shutdown effects that we saw before. There is no doubt that the current curve will have to be refined. We have a couple of such tests in our test suite with total capacity higher than 20Mbps, and for those tests the dependency on initial conditions remain. We will revisit the definition of the curve, probably to have the sensitivity follow the logarithm of data rate.

Cascade of Increases We sometimes encounter networks in which the available bandwidth changes rapidly. For example, when a competing connection stops, the available capacity may double. With low Earth orbit satellite constellations (LEO), it appears that ground stations constantly check availability of nearby satellites, and switch to a different satellite every 10 or 15 seconds depending on the constellation (see ), with the bandwidth jumping from 10Mbps to 65Mbps. Because we aim for fairness with RENO or Cubic, the cycle of recovery, cruising and pushing will only result in slow increases increases, maybe 6.25% after 6 RTT. This means we would only double the bandwidth after about 68 RTT, or increase from 10 to 65 Mbps after 185 RTT -- by which time the LEO station might have connected to a different orbiting satellite. To go faster, we implement a "cascade": if the previous pushes at 6.25% was successful, the next pushing will use 25% (see ), or an intermediate value if the observed ratio of ECN marks is greater than 0. If three successive pushes all result in increases of the nominal rate, C4 will reenter the "startup" mode, during which each RTT can result in a 100% increase of rate and CWND.

Supporting Application Limited Connections C4 is specially designed to support multimedia applications, which very often operate in application limited mode. After testing and simulations of application limited applications, we incorporated a number of features. The first feature is the design decision to only lower the nominal rate if congestion is detected. This is in contrast with the BBR design, in which the estimate of bottleneck bandwidth is also lowered if the bandwidth measured after a "probe bandwidth" attempt is lower than the current estimate while the connection was not "application limited". We found that detection of the application limited state was somewhat error prone. Occasional errors end up with a spurious reduction of the estimate of the bottleneck bandwidth. These errors can accumulate over time, causing the bandwidth estimate to "drift down", and the multimedia experience to suffer. Our strategy of only reducing the nominal values in reaction to congestion notifications much reduces that risk. The second feature is the "make before break" nature of the rate updates discussed in . This reduces the risk of using rates that are too large and would cause queues or losses, and thus makes C4 a good choice for multimedia applications. C4 adds two more features to handle multimedia applications well: coordinated pushing (see ), and variable pushing rate (see ).

Coordinated Pushing As stated in , the connection will remain in "cruising" state for a specified interval, and then move to "pushing". This works well when the connection is almost saturating the network path, but not so well for a media application that uses little bandwidth most of the time, and only needs more bandwidth when it is refreshing the state of the media encoders and sending new "reference" frames. If that happens, pushing will only be effective if the pushing interval coincides with the sending of these reference frames. If pushing happens during an application limited period, there will be no data to push with and thus no chance of increasing the nominal rate and CWND. If the reference frames are sent outside of a pushing interval, the rate and CWND will be kept at the nominal value. To break that issue, one could imagine sending "filler" traffic during the pushing periods. We tried that in simulations, and the drawback became obvious. The filler traffic would sometimes cause queues and packet losses, which degrade the quality of the multimedia experience. We could reduce this risk of packet losses by sending redundant traffic, for example creating the additional traffic using a forward error correction (FEC) algorithm, so that individual packet losses are immediately corrected. However, this is complicated, and FEC does not always protect against long batches of losses. C4 uses a simpler solution. If the time has come to enter pushing, it will check whether the connection is "application limited", which is simply defined as testing whether the application send a "nominal CWND" worth of data during the previous interval. If it is, C4 will remain in cruising state until the application finally sends more data, and will only enter the the pushing state when the last period was not application limited.

Variable Pushing Rate C4 tests for available bandwidth at regular pushing intervals (see ), during which the rate and CWND is set at 25% more than the nominal values. This mimics what BBR is doing, but may be less than ideal for real time applications. When in pushing state, the application is allowed to send more data than the nominal CWND, which causes temporary queues and degrades the experience somewhat. On the other hand, not pushing at all would not be a good option, because the connection could end up stuck using only a fraction of the available capacity. We thus have to find a compromise between operating at low capacity and risking building queues. We manage that compromise by adopting a variable pushing rate: If pushing at 25% did not result in a significant increase of the nominal rate, the next pushing will happen at 6.25% If pushing at 6.25% did result in some increase of the nominal CWIN, the next pushing will happen at 25%, otherwise it will remain at 6.25% If the observed ratio of ECN-CE marks is greater than zero, we will use it to modulate the amount of pushing. We leave the pushing rate at 6.25% if the previous pushing attempt was not successful, but otherwise we pick a value intermediate between 25% (if 0 ECN marks) and 6.25% (if the ratio of ECN marks approaches the threshold). As explained in , if three consecutive pushing attempts result in significant increases, C4 detects that the underlying network conditions have changed, and will reenter the startup state. The "significant increase" mentioned above is a matter of debate. Even if capacity is available, increasing the send rate by 25% does not always result in a 25% increase of the acknowledged rate. Delay jitter, for example, may result in lower measurement. We initially computed the threshold for detecting "significant" increase as 1/2 of the increase in the sending rate, but multiple simulation shows that was too high and caused lower performance. We now set that threshold to 1/4 of the increase in the sending rate.

Pushing rate and Cascades The choice of a 25% push rate was motivated by discussions of BBR design. Pushing has two parallel functions: discover the available capacity, if any; and also, push back against other connections in case of competition. Consider for example competition with Cubic. The Cubic connection will only back off if it observes packet losses, which typically happen when the bottleneck buffers are full. Pushing at a high rate increases the chance of building queues, overfilling the buffers, causing losses, and thus causing Cubic to back off. Pushing at a lower rate like 6.25% would not have that effect, and C4 would keep using a lower share of the network. This is why we will always pushed at 25% in the "pig war" mode. The computation of the interval between pushes is tied to the need to compete nicely, and follows the general idea that the average growth rate should mimic that of RENO or Cubic in the same circumstances. If we pick a lower push rate, such as 6.25% or maybe 12.5%, we might be able to use shorter intervals. This could be a nice compromise: in normal operation, push frequently, but at a low rate. This would not create large queues or disturb competing connections, but it will let C4 discover capacity more quickly. Then, we could use the "cascade" algorithm to push at a higher rate, and then maybe switch to startup mode if a lot of capacity is available. This is something that we intend to test, but have not implemented yet.

Adaptation to ECN/L4S Tests with L4S active queue management showed the tension between the periodic updates and L4S goal to minimize queue sizes. Typical L4S deployment start marking packets with ECN/CE when the queue size is about 1.5ms, and increase the mark rate progressively as the queue size increases, reaching 100% when the queue size is about 2ms. If C4 pushes at 25% every 6 RTT, and if the bandwidth estimate is accurate, the queue size will increase by 25% of the RTT during the first roundtrip, before any correction signal can be applied. The increased marking rate will affect all connections sharing the bottleneck, which is not desirable. L4S is tuned for the "Prague" algorithm, which increases CWIN by one packet every RTT. In a typical trial with a 20ms RTT and a 100 Mbps data rate, it takes 0.12ms to send a packet, and thus 12.5 RTT before building a queue of 1.5ms. In the same conditions, C4 would have increased the rate by 25% after 6 RTT in the aggressive scenario, thus triggering a high rate of marking. The cascade process made the problem even worse. If a push at 6.25% does increase the nominal rate, the next push will be at 25%. If that push and the next one did increase the nominal rate, C4 will reenter the initial phase, even if some of the pushes did cause ECN/CE marks. The initial phase will then cause a lot of packet losses, which will degrade performance. To mitigate this issue, we had to add a "very low" pushing mode, setting the pushing rate to only 3.125% if the previous push resulted in a high rate of ECN/CE marks. We also replaced the somewhat adhoc "count of successive probes" by the management of a "probe level", defining 4 levels: level 0: pushing at 3.125%, spend 1 cycle in cruising before pushing. level 1: pushing at 6.25%, spend 4 cycles in cruising before pushing. level 2: pushing at 25%, spend at most 1 cycle in cruising before pushing. level 3: pushing at 25%, spend at most 1 cycle in cruising before pushing. The "probe level" is updated after the recovery phase as follow: if the previous probe was successful and did not result in a high rate of ECN/CE marks, increase the probe level by 1. If the probe level was already at 3, reenter the startup phase. if the previous probe was successful but did result in a high rate of ECN/CE marks, remain at the same probe level. if the previous probe was not successful but did not result in a high rate of ECN/CE marks, stay at probe level 0 if already at that level, otherwise move back to probe level 1. if the previous probe was not successful and did result in a high rate of ECN/CE marks, move to probe level 0. This logic treats the CE marking differently from other congestion signals, because the CE marks are an intentional indication of congestion by the network, and is thus less ambiguous than delay increases or packet losses, which can be caused by other factors such as delay jitter or random transmission issues. Simulations show that this logic allows to quickly discover the available capacity in L4S networks, whithout spuriously reentering the startup phase and causing packet losses. It is equivalent to the previous logic when the network does not support L4S.

Revisiting the Initial Phase Our November 2025 design of C4 included a "rate based" initial phase, during which C4 will send at twice the "nominal rate", monitor acknowledgments and increase the nominal rate if measurements increase, and exit if congestion is detected or if the measurements do not increase for 3 consecutive RTT. That algorithm works well in most scenario, but we were observing early exits in "high delay jitter" scenarios, such as Wi-Fi networks with lots of packet collisions. After observing that phenomenon, we realized that the rate based algorithm was failing in case of high delay jitter because it was setting the CWND to the product of pacing rate and the "nominal" max RTT. The nominal Max RTT was set to a fixed value, observed either before the initial phase or on the first roundtrip in that phase. It would work if the initial phase started during a high jitter event and the initial RTT was large enough, but in many case it was not and became a limiting factor.

Why not increasing Max RTT during Initial phase? In the initial phase, the algorithm tries to discover the bandwidth and does not yet have a good estimate of delay jitter, which typically requires a series of measurements. In these conditions, it is easy to underestimate the max RTT. On the other hand, the flow is deliberately probing at a high data rate. If the algorithm allows updates of max RTT during that phase, the risks of spiraling into buffer boat are very high, but if the CWND remains too low, the risk of exiting startup with a severely underestimated data rate is also very high. We tried to develop simple rules to classify the delay measurements between caused by jitter, and caused by congestion. If we could do that, we would be able to increase the max RTT safely, when appropriate. However, we could not find variables that were both easy to monitor and well correlated with the actual cause of the delay.

Building a robust initial estimator The "rate based" initial estimator requires estimating both the data rate and the max RTT simultaneously. In contrast, the "CWND based" initial estimator use in algorithms like Reno or Cubic only requires estimating the CWND, plus a possibly loose estimate of the data rate. The Reno algorithm is remarkably simple: just increase the CWND by the number of bytes acknowledged, without any explicit dependency on the measured latency. The Reno algorithm terminates when packet losses are observed, leading to bufferbloat. Hystart improves that by terminating when the measured delays start increasing, but this can lead to early exit in case of delay jitter. The rate based algorithm terminate when the measured bandwidth stops growing, which provides good results. Our proposal is to combine a Reno like growth of the CWND with a rate-control like exit condition. Of course, things are not that simple. The "rate" test only stops the growth of the CWND after the third "non growing" round. If CWND doubles after each round it becomes excessive, buffers fill up, and lots of packets are lost. We dealt with that problem by essentially freezing the increases of the CWND after the first "non growing" round. If a larger measurement happens before 3 RTT, the increases resume, otherwise, C4 exits the initial phase. When the initial phase completes, we retain as estimate of the data rate the highest value measured so far. We also want to obtain a reasonable estimate of the "max RTT". In the Reno logic, the "ssthresh" is set to half the CWND value before congestion is detected. C4 will not use the ssthresh variable after exiting the Initial phase, but it can set the max RTT to the quotient of ssthresh by the final rate estimate.

Revision after the second design The first design, presented to the IETF in March 2026, was showing some good promises, but it needed some extra work. we have been able to develop and test three significant improvements: better reaction to a drop in path bandwidth, resolving the issues with the cascade design, and improve the handling of the "careful resume" scenario in the initial phase.

Reaction to a drop in path bandwidth In some scenarios, the bandwidth of a path may suddenly drop. This will normally cause congestion events such as packet losses or ECN marks, and C4 will react by reducing the nominal rate by the specified "beta" coefficient. However, while these reductions in bandwidth are adequate for the small decreases due to increased competition, they are not sufficient for the large drops. Such large drops can happen after routing events. In satellite based network, this happens if the connection switches to a different satellite. Simulation showed that in the large drop scenario C4 reacted by a series of small reductions until finally matching the available bandwidth. This means operating for a long period in conditions of congestion, with large queues and packet losses. We solved that problem by detecting the persistence of condition and reacting forcefully to sustained congestion. We improved C4 by monitoring the number of "recent congestions". If C4 observes more than 2 successive congestion events it will drop the nominal rate to the value of the "recent maximum rate", which hopefull reflects the new path bandwidth. This change resulted in a significant improvement in the performance measured in simulations of path bandwidth reductions.

Two steps cascade or continuous push When C4 engages in pushing, we see a clear difference in the amount of data in flight, between the consecutive cruising, pushing and recovery phase, as shown on the conceptual figure below:

time ]]>

Before entering the pushing phase, C4 is sending at the nominal rate, and the bytes in transit are proportional to the product of the nominal rate by the RTT. The sending rate increases during the pushing phase, until reaching the product of the pushing rate by the RTT. During the recovery period, the bytes in transit decrease, and by the end of that period they should be equal to the nominal rate times the RTT. The pushing period ends when the first packet sent during "pushing" is acknowledged. The recovery period ends when the last packet sent during the pushing period is acknowledged. The rate measurements during the recovery period correspond to the transmission during the the pushing period.

Two steps cascade If the highest rate measured during the recovery period is higher than the nominal rate, C4 assesses that the pushing was successful, which indicates that the previous nominal rate was smaller than the path capacity, possibly because the path capacity increased. We use these measurements to tune the rate at which C4 is pushing. By default, C4 tries small rate increases during pushes, 6.25% by default or 3.25% if the path is providing ECN feedback. If the pushing detects available capacity, C4 may try larger increases, such as 25%, 50%, 100% or even 200%, inorder to quickly adapt to the new path capacity. However, trying larger increases has a risk: if the increased rate exceeds the path capacity, queues will build, and packets may well be lost. C4 attempts to find the right push level by assessing the result of the previous push at the end of the recovery phase: if the new rate measurement grew by less than 25% of the rate increase, C4 will reset the pushing rate to its default value. if the rate measurement grew by at least 75% of the rate increase, C4 will start pushing at the next level. if the rate measurement is intermediate, C4 might keep the increase rate unchanged, or might decide to only push at a lower level. In short, the two step process allows C4 to perform a "cautious cascade", increasing the push level fast enough to adapt quickly, but backing off as soon as the path appears saturated.

Finding out why the cascade does not work. We need to program the different levels of the cascade. We know that we should not set the first level too large, because adopting a large value causes the queue size and the RTT to periodically increase. Our philosophy is to use a small increase, like 6.25%, that causes minimal disruption. This allows C4 to start pushing frequently, and thus reduce the delay needed to detect a change in bandwidth. We have also observed that increasing too fast may cause queues and losses. The whole point of the two step approach is to proceed cautiously and always test the result of the previous increase before increasing the push rate. In practice, this means limiting the largest levels to 100% increase, on par with slow start. This leaves a choice for the second level of the cascade. We tried two values: 25%, similar to the BBR push rate; 50%, twice that rate, since we are twice more cautious. (We could have tried 56.25%, equivalent to 2 BBR rounds at 25%). The following table compares the increments applied by BBR after each successive RTT and the multiplier from the original rate. BBR increases by 25% after each RTT, but that C4 only increases the data rate every other RTT. The cascade helps, and once C4 starts increasing at 100% per period, it does catch up. However, BBR will be more aggressive for the first rounds. Nb rounds BBR-INC C4-25-INC C4-50-INC BBR-BW C4-25-BW C4-50-BW 0 0 0 0 1 1 1 1 25% 6.25% 6.25% 1.25 1.06 1.06 2 25% 0% 0% 1.56 1.00 1 3 25% 25% 50% 1.95 1.33 1.59 4 25% 0% 0% 2.44 1.06 1.06 5 25% 100% 100% 3.05 2.66 3.19 6 25% 0% 0% 3.81 1.33 1.59 7 25% 100% 100% 4.77 5.31 6.38 8 25% 0% 0% 5.96 2.66 3.19 The BBR probe BW or the BBR cascade stop when the measured rate stops increasing. The main advantage of the two steps approach is to reduce the amount of queuing when that happens. However, if the cascade starts using large increments, the benefits over BBR's continous approach are not obvious. The BBR algorithm is continuous, and when it stops, the probe bandwidth always exceed the nominal bandwidth by just 25%. The C4 algorithm is not continuous. If it stops during the first RTT, the excess bandwidth will be at most 6.125% of the available bandwidth, but up to 25% if it stops in the second RTT and up to 100% later. In short, the cautious approach pays off during the initial "6%" probing stage, grows slower than BBr after that, and also produces larger queues. In short, we have a bug.

Addind a Pushing state We can capture the lessons of the previous analysis by splitting the "pushing" state in two: A "probing" state executed frequently, with a low 6.125% probing margin. A "pushing" state similar to the "probe-BW" state of BBR, pushing at 25%, remaining in the same state until the measurements stop increasing.

Keeping Probing and Pushing as separate state The design recomended in this section separates the Probing and Pushing states. We considered simplifying the design and merging the two states, and tried that in an experiment: just enter the pushing state and use the 6.125% increase for the first RTT, and then switch to 25% as soon as a measurement shows an increase in rate. The results were mixed. One test out of 22 did improve significantly, i.e., competition with a BBR connection that starts after the C4 connection, showing performance more than 20% better than the previous version. This is probably because the merging of the two states makes C4 more aggressive. However, several other tests showed regressions, including the "drop and back" test, which showed a 6% regression. This is probably due to the reduced agility of the merged state: each cycle of cruising-pushing-recovery lasts 5 RTT instead of 4, causing the change in bandwidth to be detected later. The merged state also caused regressions in the testing of competiton between C4 connections, and in the Wi-Fi scenarios, probably due to the excessive aggressivity of the merged solution. The separation of Probing and Pushing states appear to be a good compromise between agility and aggressivity.

Support for careful resume The performance measurements of the previous version of C4 showed worse performance in the "geo stationary satellite" scenario. The typical slow start algorithm doubles the congestion window after every RTT, which requires a number of RTT proportional to the logarithm of the bandwidth delay product. Transmission via geostationary satellite has a 600ms RTT, which means large bandwith delay product, and a long time to reach the available bandwidth. The "careful resume" algorithm specified in was designed to address that issue, and we had implemented it in C4. However, the previous implementation of careful resume was inefficient, because it was too tightly enmeshed with the "slow start" logic. It is simpler to treat "resuming" as a state by itself, rather than a complication of startup. The resuming state is entered when the stack sets the seed CWIN, from which we deduce the the seed rate. During that state, pacing rate and CWIN are pegged to the seed values. The state lasts for 2 eras, giving enough time for the rate measurement to stabilize. The eras are expanded if the connection is app limited, to avoid exiting too early. After two eras, or at any time if congestion is detected, the state transitions to recovery. The performance of the satellite test are significantly improved. They are better than BBR, almost on par with Cubic, and within the 7 second performance target set for that test: average time for simple tests c4 bbr cubic c4_pushing this PR satellite 7660817 7431946 6704246 7660783 6807146 top 90% time for simple tests c4 bbr cubic c4_pushing This PR satellite 7661569 7432276 6704247 7661405 6807183 It might be possible to improve a bit more by some further tuning, for example exiting directly to cruising instead of recovery if the nominal rate matches the seed rate. This might be something for further work.

Revision after the third design After presentation to the IETF during the meeting in Vienna in July 2026 (IETF 126), we received feedback about a number of potential issue, including the need to test C4 in presence of buffer bloat, and the need to analyse competition issues. To address this feedback, we added "buffer bloat" tests to the list of simulations used for validating the design, and we also decided to improve the reporting of tests by also reporting the evolution of the RTT, and, for tests involving competition, the share of network capacity claimed by the C4 connection. These tests showed that improvements were needed to avoid persisting higher-than-capacity pacing rates in some scenarios, and to quickly reduce network queues if they were built. We discuss these improvements in and . These improvements had the expected result, but they also triggered a performance regression in the "bad WiFi" scenario, which we discuss in . With these three modification, we obtain a C4 protocol that meets our requirement, providing high performance and low delays with a simple design.

Trimming excess capacity The new set of measurements showed that in some scenarios, the "third design" version of C4 was driving the RTT towards large values. The large delays that we observe are caused by increased queues, which are caused by sending data too fast for too long. The design included precautions to avoid keeping excessive max RTT values (see ). These precautions solve the "too long" part of the issue, but there are limits because the long nominal max RTTs are necessary in high jitter situations. In fact, we do see cases where delay jitter causes the data rate estimate to be slightly higher than the actual available bandwidth. To reduce the queues, we probably need C4 to send a bit slower. We tried a simple change: when the data rate does not increase for a whole cycle, and the transmission in the probing stage was not application limited, reduce the nominal data rate to track the measurements -- something that C4 only did before when noticing persistent congestion. We experimented with different rates of decrease, and found that a 1/2th decrease provided the most interesting results. The change in specification would be, "if the data rate does not increase for a whole cycle, and the transmission in the probing stage was not application limited, reduce the nominal data rate to the average of the previous value and the highest measurement in the last cycle." top 90% of RTT + standard deviation for network events tests c4 bbr cubic Trimming 1/2 rate alone 145798 107604 167571 141307 alone_200 88677 57168 70772 88272 alone_1_5M 96662 64683 90867 73264 alone_512k 105400 104226 106116 90963 low_and_up 127913 134293 147216 129420 drop_and_back 148289 150347 161333 149161 blackhole 479783 500638 485330 432236 short_long 230005 232484 409884 229986 satellite 602366 830684 621278 602326 The top RTT values are lower in all the network events tests but two, the "low and up" test, with a difference of 1.4 ms, and the "drop and back" test, with a difference of 1.1 ms. We see significant improvement in the 1.5 Mbps test, with a difference of 23 ms, and in the 512 Kbps test, with a difference of 14 ms. We seem to see a compromise: trimming the nominal rate quickly does reduce the queues in general, but having a slightly lower nominal rate means that queues will persist a bit longer after a rate increase. The tables of "time spent" do not show any preformance decrease in these tests -- some are faster, some are slower, but the differences are tiny. top 90% of RTT + standard deviation for buffer bloat tests c4 bbr cubic Trimming 1/2 bbloat 158066 95924 322642 156178 bbloat_c4 361542 118202 939834 345468 bbloat_bbr 160655 159931 408664 159903 bbloat_cubic 859970 137306 1039625 859395 top 90% load for buffer bloat tests c4 bbr cubic Trimming 1/2 bbloat 98% 95% 98% 98% bbloat_c4 59% 58% 60% 59% bbloat_bbr 82% 58% 86% 82% bbloat_cubic 60% 58% 60% 60% Trimming has very little effect on the buffer bloat tests. We do see some small reductions in the top RTT for some tests, but these are too small to matter. We also do not see any big change in the fairness of competing under buffer bloat. One possibility is that, while trimming prevents the queues from increasing too much, it does not by itself drain them. We will complement trimming by some explicit form of draining. top 90% of RTT + standard deviation for compete tests c4 bbr cubic Trimming 1/2 vs_bbr 127906 156408 163495 125994 vs_c4 147578 133587 154995 162418 vs_cubic 161612 151990 165428 160008 after_c4 150671 129600 139090 150557 before_c4 109443 96827 110039 108681 vs_c4_lg 147437 112710 153852 149633 vs_c4_lg2 158254 154662 141550 157719 vs_bbr_lg 141010 160019 162743 137970 vs_bbr_lg2 147211 155612 149919 154808 vs_cubic_lg 151434 120204 162934 150852 vs_cubic_lg2 146843 149529 159656 146035 top 90% load for compete tests c4 bbr cubic Trimming 1/2 vs_bbr 78% 47% 79% 78% vs_c4 52% 31% 33% 54% vs_cubic 66% 31% 46% 69% after_c4 40% 34% 32% 38% before_c4 72% 50% 67% 69% vs_c4_lg 61% 58% 58% 61% vs_c4_lg2 64% 61% 60% 63% vs_bbr_lg 81% 59% 81% 81% vs_bbr_lg2 78% 69% 61% 80% vs_cubic_lg 78% 58% 60% 77% vs_cubic_lg2 76% 81% 63% 75% Trimming the nominal rate quickly does not seem to improve the fairness of C4 in the compete tests, but it does improve the top RTT measurements in the competition between C4 and BBR, and between C4 and Cubic, except for the "vs_bbr_lg2" test, where the top RTT is 7.6 ms higher than before. We also see a slight degradation in the "internal competition" tests, where C4 competes with itself. top 90% time for wifi tests c4 bbr cubic Trimming 1/2 wifi_bad 4841982 7522796 4449644 5249372 wifi_fade 5351680 5649668 5550500 5244862 wifi_suspension 4574262 4616911 4602102 4572601 wifi_bad_bbr 11014365 10730711 13129404 10454091 wifi_bad_c4 11665740 12481659 12241907 11615184 wifi_bad_cubic 11883994 12246412 13881720 11388974 top 90% of RTT + standard deviation for wifi tests c4 bbr cubic Trimming 1/2 wifi_bad 236990 146354 149225 260841 wifi_fade 210684 177334 187572 210766 wifi_suspension 32878 34220 45312 28783 wifi_bad_bbr 335207 339723 327846 332603 wifi_bad_c4 331313 333343 333270 331021 wifi_bad_cubic 339101 357546 327358 335187 We avoided making similar changes before because of the effect on high jitter situations, in particular "bad wifi" environments. These are discussed in .

Draining the backlog The results of tests in buffer bloat scenarios show C4 creating shorter queues than Cubic but longer queues than BBR, even after implementing the "trimming" changes. average RTT for buffer bloat tests c4 bbr cubic trimming 1/2 bbloat 113475 84750 229647 111173 bbloat_c4 229418 92740 660442 234925 bbloat_bbr 131826 124558 266277 131812 bbloat_cubic 551053 97505 647602 551157 We analysed "qlog" traces of the BBR and C4 traces in the "buffer bloat" trace, and found that most of the difference in average RTT measurements was due to the more efficient "draining" of the excess queues created in the Initial phase. BBR enters a "drain" phase after exiting start-up, during which it operates at a small fraction of the estimated bandwidth, and exits that phase when the number of bytes in flight is below the bandwidth-delay product of the path, defined as the product of the estimated bandwidth by the min RTT. We don't want to copy BBR's algorithm, because its dependency on the min RTT of the path conflicts with the inherent variation of the RTT in high delay jitter scenarios. We also want to limit the size of the bandwidth drops when draining, because such drops would add delays to the transmission of real time media. Instead, we first tried a simple change, adding a "draining" option to the recovery phase. If the draining option is set, the pacing rate during recovery is set to only 7/8th of the nominal rate, instead of the default 15/16th. We would set this option on exiting the Initial or the pushing phase, and perhaps also if some other condition detected the presence of a backlog. This provided promising initial results, and also reduced the aggressiveness of C4 during the compete tests, which improved its fairness. However, we noticed that we could improve the results with a simple change: instead of maintaing just one flag, maintain an estimate of the backlog. The estimate of the backlog is set to 1 full RTT of data after the Initial phase, because during that phase the pacing rate is twice the nominal rate. It is set to a quarter of the RTT of data after the pushing phase, because during that phase the pacing rate is 5/4 of the nominal rate. Upon entering recovery, the pacing rate is set as a function of the backlog: if the backlog estimate is less than 1/16, set the pacing rate to 15/16th of the nominal rate; else if the backlog estimate is less than 1/8, set the pacing rate to 7/8th of the nominal rate; else set the pacing rate to 3/4th of the nominal rate. At the end of the recovery phase, the backlog will be 1/16th, 1/8th, or 1/4th, depending of the reduction in the pacing rate -- or set to zero if the reduction would result in a negative value. average RTT for buffer bloat tests c4 bbr cubic draft-04 bbloat 91540 84752 231784 113442 bbloat_c4 181516 99182 661710 232427 bbloat_bbr 159868 124409 269224 132447 bbloat_cubic 667539 97432 647872 551412 This simple change has a very positive effect on the average RTT for the buffer bloat test. The average RTT is only 8% larger than what we get with BBR, but BBR uses the min RTT of the path to determine the congestion window, while C4 avoids building a dependency on the min RTT. When competing in the buffer bloat scenario, C4 is more aggressive than BBR, and the average RTT observed when competing with C4 or with BBR is larger that what BBR achieves. On the other hand, when competing with Cubic, BBR is shut down, while C4 keeps competing. The backlog algorithm is promising, but it may need additional tuning.

Revisiting support for Wi-Fi Each of our test runs performs 100 simulations of the "bad Wi-Fi" scenario. We observe that there is a high variability between the simulations. Some of that is expected. The simulation of Wi-Fi jitter is a random process, and we can expect that different simulations will encounter different patterns of jitter, and thus different performances. The graph of durations shows that, but it also shows that the worse 20 or so simulations are significantly worse "after trim and drain" than with the unmodified "draft-04" implementation. Analysis of the logs of the worst simulation results let us see an obvious pattern. In the "good" case, we see an event about 1.5 second into the connection that leads C4 to explore higher RTT and higher data rates, while in the "bad" case the RTT is constrained into a much narrower band and the pacing rate converges on a much lower value. The growth pattern of the "good" connection between 1.5 and 2 seconds after the starts corresponds to C4 re-entering the Initial state, following the rule specified in section 4.4.1 of the version 04 of the specification: C4 will reenter the "initial" phase on the first time high jitter is detected for the flow. The high jitter is detected after updating the "nominal max RTT" at the end of the recovery era, if running_min_rtt < nominal_max_rtt*2/5. This will be done at most once per flow. We discovered that in the worst cases, C4 did re-enter the Initial phase, be reentered it very early. Draining kept the running_min_rtt low, and thus even a limited jitter would trigger the reinitialization. The "only once" rule was preventing adaptation when the larger jitter event occured later. The solution was to remember the time the last initial phase completed, or more precisely the time the recovery that followed the last initial phase completed. We added to the state a variable last_initial_time, initialized at zero when entering the "initial" state, and set to the current time if the first exit of the recovery state after that initial state. Then, we changed the specification to: C4 will reenter the "initial" phase on the first time high jitter is detected for the flow. The high jitter is detected after updating the "nominal max RTT" at the end of the recovery era, if running_min_rtt < nominal_max_rtt*2/5, and if the variable last_initial_time is not zero and if current_time > last_initial_time + 1 second. The performance for the Wi-Fi test improved immediately after this fix, with the average execution time of Wi-Fi tests falling below the value observe in the previous design, and without causing any significant regression in the other scenarios.

Revising the trimming pattern In the discussion of trimming (see ), we recommended evaluating the maximum bandwidth measurement over a cycle of cruising, probing and recovery, and trimming the nominal rate to the midpoint between the nominal rate and that measurement. Once all the previous changes were made, we reevaluated that decision and considered three variants: Drop to the middle of the nominal rate and the measured rate, Drop to the measured rate, Drop to the maximum of the current measured rate and the rate measured in the previous cycle. The second option was quickly ruled out because the "top 90%" media latency in the "bad wifi" scenario was 100ms higher than in the "middle" scenario. When we observe heavy jitter as in the "bad wifi" scenario, rate measurements become noisy. The "middle" and "max of 2" methods will filter out that noise, but the immediate drop will not, and the performance can suffer. The "max of two" option provides performance roughly on par with the "middle" scenario, generally with lower measured RTT. It does provide significantly better performance in the ECN scenario, and slghtly more fair results in competition scenarios. We adopted it in the specification.

State Machine The state machine for C4 has the following states: "startup": the initial state, during which the CWND is set to twice the "nominal_CWND". The connection exits startup if the "nominal_cwnd" does not increase for 3 consecutive round trips. When the connection exits startup, it enters "recovery". "resuming": management of careful resume, during which the CWND and pacing rate are pegged to the seed values. The state lasts for 2 eras, giving enough time for the rate measurement to stabilize. The eras are expanded if the connection is app limited, to avoid exiting too early. After two eras, or at any time if congestion is detected, the state transitions to recovery. "recovery": the connection enters that state after "startup", "pushing", or a congestion detection in a "cruising" state. It remains in that state for at least one roundtrip, until the first packet sent in "recovery" is acknowledged. Once that happens, the connection goes back to "startup" if the last 3 pushing attemps have resulted in increases of "nominal rate", or enters "cruising" otherwise. "cruising": the connection is sending using the "nominal_rate" and "nominal_max_rtt" value. If congestion is detected, the connection exits cruising and enters "recovery" after lowering the value of "nominal_cwnd". Otherwise, the connection will remain in "cruising" state until at least 4 RTT and the connection is not "app limited". At that point, it enters "pushing". "probing": the connection is using a rate and CWND 6.15% larger than "nominal_rate" and "nominal_CWND". After 1 RTT, it moves back to "recovery" in order to assess the results. If the data rate appears to have increased, the connection moves to the "pushing" state. "pushing": the connection is using a rate and CWND 25% larger than "nominal_rate" and "nominal_CWND". It remains in that state for at least one round trip, and until the measured rate stops growing. If the pushing lasts more than 3 RTT, C4 re-enters the initial state. These transitions are summarized in the following state diagram.

| Recovery | | ^ ^ +----|---|---+ | | | | | Rate increase | | | | +---------+ | | | | | | | | v | | | | +----------+ | | | | | Cruising | | | | | +-|--|-----+ v | | | Congestion | | +---------+ | | +-------------+ | | Pushing | | | | +----|--|-+ | | v | | | | +----------+ | +-------+ | | Probing | | Rapid | +----|-----+ | increase | | | +<------------------+ | ^ | | | +---------------------------------+ ]]>

C4 exits the pushing state after one era, or if a congestion signal is received before that. In an exception to standard congestion processing, the reduction in nominal_rate and nominal_max_RTT are not applied if the congestion signal is tied to a packet sent during the probing or pushing state.

Security Considerations We do not believe that C4 introduce new security issues. Or maybe there are, such as what happen if applications can be fooled in going to fast and overwhelming the network, or going to slow and underwhelming the application. Discuss!

IANA Considerations This document has no IANA actions.

&RFC9000; &I-D.ietf-moq-transport; &RFC9438; &I-D.ietf-ccwg-bbr; &RFC6817; &RFC6582; &RFC3649; &I-D.irtf-iccrg-ledbat-plus-plus; &RFC9330; &RFC9331; &RFC9959; &I-D.briscoe-iccrg-prague-congestion-control;

Acknowledgments TODO acknowledge.