TCP: Reliability, Windows, and Congestion

IP delivers packets that may be lost, duplicated, reordered, or corrupted. TCP, whose scattered specifications were finally consolidated into RFC 9293 in 2022, offers four guarantees on top of that:

• A byte stream, not messages. Write 100 bytes then 200 bytes, and the reader may see one 300-byte read. TCP has no record boundaries.
• Ordered delivery. Bytes arrive in the order they were sent, or not at all.
• Reliability. Anything lost is retransmitted until acknowledged or the connection dies.
• Rate control. The sender is throttled both by what the receiver can absorb and by what the network can carry.

Every guarantee has a price. Ordering means a lost packet stalls everything queued behind it even if that data already arrived. Reliability means waiting for timers. Rate control means the sender starts slow and probes upward rather than sending at line rate. Almost every mystery about internet performance is one of these guarantees doing its job.

Three packets, one round trip before any data moves. Each side must learn the other's initial sequence number and confirm it was received.

The handshake is not a greeting. It is a mutual agreement on starting state.

Each side chooses an initial sequence number and needs proof the other side received it. That takes two exchanges in each direction, and the middle two collapse into one packet, hence three. Along the way both sides advertise their receive window, maximum segment size, whether they support selective acknowledgement, and the window scale factor. Those options appear only in the SYN, so anything lost there is lost for the life of the connection.

Initial sequence numbers are randomised, not zero. A predictable ISN lets an off-path attacker inject data into a connection they cannot see.

The cost is one full round trip before a single byte of application data moves. On a 200 ms intercontinental path that is 200 ms spent before anything useful happens, and TLS historically added more on top. Half-open connections waiting in the accept backlog are also a denial-of-service vector, which is why SYN cookies exist: the server encodes its state into the sequence number and keeps nothing until the final ACK arrives.

TCP numbers bytes, not packets. A segment header carries the sequence number of its first byte, so a 1460-byte segment starting at 5000 covers bytes 5000 to 6459 and the next segment starts at 6460.

Acknowledgements are cumulative and forward-only. An ACK of 6460 means "I have every byte below 6460" and says nothing about anything above it. That is compact and robust: a lost ACK is harmless because the next one supersedes it.

It is also lossy as information. Suppose segments 1, 2, 4 and 5 arrive but 3 is lost. The receiver can only keep repeating ACK 3, so the sender learns that something is missing but not that 4 and 5 are safe in the receive buffer. Classic TCP responded by retransmitting everything from 3 onward.

Selective acknowledgement, defined in RFC 2018, fixes this with a TCP option listing the blocks actually received. The sender then retransmits only the hole. SACK is negotiated in the SYN and is effectively universal today, but it is still an option, not part of the base header.

The last-resort recovery mechanism is a timer. RFC 6298 specifies it precisely.

The sender maintains a smoothed round-trip time and a round-trip variance, updated on every measurement:

RTTVAR = (1 - 1/4) * RTTVAR + (1/4) * |SRTT - R|
SRTT   = (1 - 1/8) * SRTT   + (1/8) * R
RTO    = SRTT + max(G, 4 * RTTVAR)

The variance term is the interesting part. A path with steady latency gets a tight timeout; a jittery path gets a generous one, because on a jittery path an aggressive timer would fire on packets that were merely late.

Three constants matter in practice. Before any measurement exists the RTO is 1 second. Whenever the computed RTO falls below 1 second it is rounded up to 1 second, so even on a 5 ms datacentre link the first timeout costs a full second. And each consecutive timeout doubles the RTO, with implementations permitted to cap it no lower than 60 seconds.

That is why a genuinely stalled connection takes minutes to fail rather than milliseconds.

Waiting a full second for a timer is unacceptable when the actual round trip is 30 ms, so TCP uses the acknowledgement stream itself as a loss signal.

When a receiver gets an out-of-order segment it immediately repeats its last cumulative ACK. RFC 5681 fixes the rule: the arrival of three duplicate ACKs is treated as evidence of loss, and the sender retransmits the missing segment at once without waiting for the RTO.

Why three and not one? Reordering is normal on the internet, and a single duplicate ACK usually means a packet took a different path and arrived late. Three is an empirical compromise between reacting quickly and reacting to noise.

Fast retransmit is paired with fast recovery, which halves the sending rate rather than collapsing it. The distinction is deliberate: duplicate ACKs prove packets are still flowing, so the path is congested but working. A timeout proves nothing is getting through, and there the sender restarts from the beginning.

Gotcha: fast retransmit needs a continuing stream of ACKs. Lose the last segment of a response and there is no following segment to trigger duplicates, so recovery falls back to the timer.

Flow control protects the receiver. Every ACK advertises a receive window: the number of additional bytes the receiver has buffer space for. The sender may never have more than that many unacknowledged bytes in flight.

The window field is 16 bits, so the raw maximum is 65,535 bytes. Work out what that costs on a fast, long path.

link       = 100 Mbit/s
RTT        = 80 ms
BDP        = 100e6 * 0.080 / 8 = 1,000,000 bytes in flight to fill the pipe
max rate with a 64 KB window
           = 65,535 * 8 / 0.080 = 6.55 Mbit/s

Six and a half megabits on a hundred-megabit link, and no amount of bandwidth fixes it, because the limit is the window divided by the round trip.

RFC 7323 window scaling solves this with a shift factor negotiated in the SYN, allowing windows up to about 1 GB. Since the factor only appears in the SYN, a middlebox that strips unknown TCP options silently caps a connection at 64 KB, and the symptom is a transfer that runs at exactly the same disappointing speed no matter what you upgrade.

A receiver whose application stops reading advertises a zero window, and the sender falls back to periodic window probes.

The single most common confusion in TCP is treating flow control and congestion control as one mechanism. They are separate, they protect different things, and they are computed from different signals.

The sender is bound by both at once: bytes in flight must not exceed min(cwnd, rwnd).

The practical value of separating them is diagnostic. If a transfer is slow and the advertised window is small, the receiving application is not reading fast enough and the network is innocent. If the advertised window is large but throughput stays low, the sender's congestion window is the constraint and you should look for loss, queueing, or a mis-tuned algorithm.

A new connection knows nothing about the path, so it probes. RFC 5681 defines two phases separated by a threshold called ssthresh.

In slow start the sender increases cwnd by one segment for every ACK of new data, which doubles the window every round trip. It is exponential growth with a modest name. RFC 6928, based on large-scale experiments across Google's front ends, raised the standard starting point from 2 to 4 segments up to 10.

Once cwnd passes ssthresh the sender enters congestion avoidance and grows by roughly one segment per round trip: additive increase. On loss it cuts the window multiplicatively, classically by half. That is AIMD, and the resulting throughput graph is the familiar sawtooth.

Count the cost on the 1 MB pipe from the previous step. Filling it needs about 700 segments of 1460 bytes. Starting at 10 and doubling, the sequence runs 10, 20, 40, 80, 160, 320, 640, 1280: seven round trips, and at 80 ms each that is 560 ms of ramp-up before the link is busy.

This is why short connections never reach full speed, and why connection reuse matters more than raw bandwidth.

Classic AIMD scales badly. On a fast, long path the window needed is enormous, and adding one segment per round trip takes an impractically long time to recover after every loss. Worse, the recovery rate depends on the round-trip time, so a short-RTT flow grabs bandwidth far faster than a long-RTT flow sharing the same bottleneck.

CUBIC, standardised as RFC 9438 in August 2023, replaces the linear increase with a cubic function of the time elapsed since the last congestion event. The shape matters more than the formula: growth is fast immediately after the cut, flattens as the window approaches the size that previously caused loss, pauses there to probe cautiously, then accelerates again if nothing breaks.

Because the curve is driven by wall-clock time rather than by ACK arrivals, flows with different round-trip times converge on more similar shares of a bottleneck. RFC 9438 notes CUBIC is the default congestion control in the Linux, Windows and Apple stacks, which makes it the algorithm behind most of the traffic you will ever measure.

It is still loss-based. CUBIC learns the network is full only once a queue has overflowed.

Loss-based control has a hidden assumption: that packet loss means congestion. On a modern network that is often false. Wireless links lose packets to interference while the path is empty, and deep buffers absorb megabytes without dropping anything, so a loss-based sender happily fills them.

BBR, introduced by Cardwell and colleagues at Google in 2016, drops the assumption. It continuously estimates two quantities: the bottleneck bandwidth, from the maximum delivery rate observed, and the round-trip propagation time, from the minimum RTT observed. The optimal operating point is sending at exactly the bottleneck rate with only bandwidth times minimum RTT in flight. Above that, extra packets add queue and latency but no throughput.

BBR therefore paces packets at its bandwidth estimate rather than sending bursts, and periodically probes up and down to refresh both estimates. BBRv3, released in 2023, tuned fairness against CUBIC; Google's IETF presentation on its public deployment reported a 12 percent reduction in the packet retransmit rate.

The honest caveat is that BBR and loss-based flows do not always share a bottleneck evenly, and the balance depends heavily on buffer depth. This is an area of active measurement, not a settled result.

Jim Gettys and Kathleen Nichols named this problem in "Bufferbloat: Dark Buffers in the Internet", published in ACM Queue and Communications of the ACM in 2011. Memory became cheap, so equipment vendors added generous buffers on the theory that dropping packets is bad. The result was the opposite of what they intended.

A loss-based sender expands until it sees loss. If the bottleneck holds a second of traffic, the sender fills a second of traffic before getting any feedback, and every packet behind it waits that second. Throughput is fine. Latency is destroyed. That is why a large upload can make a video call unusable on the same connection while the speed test still reads full bandwidth.

Gettys and Nichols also pointed out that the damage compounds: TCP's reaction time worsens quadratically with over-buffering, so a link buffered ten times too deeply not only adds ten times the latency but takes a hundred times as long to respond to congestion.

The fix is not smaller buffers, which hurt bursty traffic, but active queue management that drops or marks early. CoDel and FQ-CoDel, specified in RFC 8290, target queue delay directly rather than queue length.

UDP gives you addressing, ports, and a checksum. Nothing else: no handshake, no ordering, no retransmission, no congestion control. That is a feature when TCP's guarantees are actively harmful.

Reach for UDP when:

• Late data is worthless. In a voice or video call, a frame that arrives 400 ms late is useless, and TCP would have stalled every later frame waiting for it. Concealing the gap beats waiting.
• The exchange is a single request and reply. A DNS query fits in one packet, and a three-packet handshake to carry one packet is pure overhead.
• One loss must not stall unrelated work. Independent streams over one TCP connection all stall together; over UDP they do not.
• You want to control the transport yourself. QUIC runs on UDP precisely so it can implement handshake, ordering, and congestion control in userspace and ship changes without waiting for kernels.

The warning attached to that last point: if you send bulk data over UDP you must implement congestion control yourself. An application that blasts UDP at line rate is not clever, it is the thing congestion control was invented to prevent. And because UDP is a common reflection and amplification vector, plenty of networks rate-limit or block it, so a UDP-based protocol needs a TCP fallback.