▲ 224 points
back
209 comments
Most of TCP woes comes from high-bandwith latency-sensitive stuff like HFT and video, but TCP isn't particularly good for low-bandwidth high-latency networks either (e. g. NB-IoT with 10 seconds worst case RTT):
- TCP will waste roundtrips on handshakes. And then some extra on MTU discovery.
- TCP will keep trying to transmit data even if it's no longer useful (same issue as with real-time multimedia).
- If you move into a location with worse coverage, your latency increases, but TCP will assume packet loss due to congestion and reduce bandwidth. And in general, loss-based congestion control just doesn't work at this point.
- Load balancers and middleboxes (and HTTP servers, but that's another story) may disconnect you randomly because hey, you haven't responded for four seconds, you are probably no longer there anyway, right?
- You can't interpret the data you've got until you have all of it - because TCP will split packets with no regards to data structure. Which is twice as sad when all of your data would actually fit in 1200 bytes.
And TCP wants to pretend that all your data arrives as a linear stream one after the other; and keeps you from seeing package n+1, if you haven't received package n, yet.
That's useful for some things, but often you can make use of package n+1, even when package n hasn't arrived, yet.
For example, when you are transferring a large file, you could use erasure encoding to just automatically deal with 5% package loss. (Or you could use a fountain code to deal with variable packet loss, and the sender just keeps sending until the receiver says "I'm done" for the whole file, instead of ack-ing individual packages.
Fountain codes are how deep space probes send their data back. Latency is pretty terrible out to Jupiter or Mars.)
> TCP will waste roundtrips on handshakes. And then some extra on MTU discovery.
This is especially true when using TLS (which is at least a major use case nowadays if not the default assumption); the TCP handshake has to complete before the TLS handshake can occur, whereas QUIC has protocol-level support for doing TLS negotiation as part of the initial handshake. Having the network protocol and the encryption be composable rather than tightly coupled feels more elegant, it's hard for me to convince myself that being able to save a round trip per connection isn't a more practical benefit in a world where we now expect pretty much everything to be encrypted in transit.
> If you move into a location with worse coverage, your latency increases, but TCP will assume packet loss due to congestion and reduce bandwidth. And in general, loss-based congestion control just doesn't work at this point.
Don't lower levels of wifi below TCP now do often their own retransmit, even if it is just from weak/noisy signal and not a sensed collision?
Is NB-IoT actually being used these days? I remember it was hyped up but then the buzz disappeared.
We use straight UDP datagrams for streaming high-frequency sensor data. One of our R&D people built a new system that uses quic and solves most of our problems with out-of-order delivery. We still use datagrams over UDP for everything because we have to support some 3rd party sensors out of the box without adapters, and UDP is all they can do.
Also pretty much every media art system in the world just uses OSC over UDP
Yeah, high frequency sensor data was on my mind too. Do you have any figures on the power saved wrt TCP?
what language was the system build with? thats super cool!
You could have used RoCE for reliability over UDP
"solves most of our problems"
Indeed, until the problems it creates begin to feature.
Have a great day =3
This may seem like a minor nit, but I think there is a problem with using the term "unreliable" to describe UDP. The more commonly used term, and IMHO better term, is "best-effort" [1]. UDP makes its best effort to deliver the datagrams, but the datagrams may be dropped anyway. But it does not make UDP inherently unreliable.
IMO "best-effort" is euphemistic and confusing to people outside of this space, even (especially?) to native English speakers. (I would probably have described UDP as "reasonable effort" and TCP as "best effort", if not for the existing terminology.)
I recall my computer networking professor describing this as "Best-effort means never having to say you're sorry". In practice, best-effort does not mean you try your hardest to make sure the message gets from A to B, it means that you made an effort. Router in the path was congested? Link flap leading to blackholing on the order of 50ms before fast reroute kicks in? Oh well, we tried.
Meanwhile, TCP's reliable delivery will retry several times and will present an in-order data stream to the application.
Reliable vs unreliable might be bad terminology, but I don't think best-effort is any better.
My experience with unreliable systems is that they're great something like 95% of the time, and they're great for raw throughput, but there are many cases where that last 5% makes a huge difference.
The term "best effort delivery" in networking is a weasel term that is no better than "unreliable". It should probably be burned.
"Effort" generally refers to some sort of persistence in the face of difficulty. Dropping a packet upon encountering a resource problem isn't effort, let alone best effort.
The way "best effort" is used in networking is quite at odds with the "best efforts" legal/business term, which denotes something short of a firm commitment, but not outright flaking off.
Separately from the delivery question, the checksums in UDP (and TCP!) also poorly assure integrity when datagrams are delivered. They only somewhat improve on the hardware.
> but the datagrams may be dropped anyway
That's what I thought "unreliable" meant? I can't really tell what misconception you are trying to avoid.
i agree that "unreliable" isn't a good term for the transport. reliability is a property of the system, not of the transport. and you can make an unreliable system with TCP or a reliable one with UDP
but, "best-effort" implies that it's doing some effort to ensure delivery, when it's really dropping any packet that looks funny or is unlucky enough to hit a full buffer
i like "lossy", but this is definitely one of the two hard problems
We’ve been calling it “unreliable” transport since the 80’s, and that’s what it is. Want your packets to get there? TCP. Don’t care much? UDP.
Oversimplified.
Best effort is a dumb term. There’s no effort.
This sounds like the problem is the term "best-effort" (hand wavy, what's the measure of effort? What's "the best" effort?).
In the end, best-effort is just saying "unreliable" in a fussier way.
>But it does not make UDP inherently unreliable.
Isn't that exactly what it does make it?
If that's not it, then what woud an actual "inherently unreliable" design for such a protocol be? Calling an RNG to randomly decide whether to send the next packet?
Fire-and-forget?
There’s nothing intuitive about what “best effort” means, but if you know anything about udp and tcp you know which is “reliable.”
> UDP makes its best effort
TCP tries really hard, too, you know.
IMO stream abstractions make it too convenient to write fragile programs which are slow to recover from disconnections (if they do at all) and generally place too many restrictions on the transport layer. Congestion control is definitely needed but everything else seems questionable.
In a datagram-first world we would have no issue bonding any number of data links with very high efficiency or seamlessly roaming across network boundaries without dropping connections. Many types of applications can handle out-of-order frames with zero overhead and would work much faster if written for the UDP model.
> In a datagram-first world we would have no issue bonding any number of data links with very high efficiency or seamlessly roaming across network boundaries without dropping connections. Many types of applications can handle out-of-order frames with zero overhead and would work much faster if written for the UDP model.
So your argument is that software isn’t written well because TCP is too convenient, but we’re supposed to believe that a substantially more complicated datagram-first world would have perfectly robust and efficient software?
In practice, moving to less reliable transports doesn’t make software automatically more reliable or more efficient. It actually introduces a huge number of failure modes and complexities that teams would have to deal with.
> Congestion control is definitely needed but everything else seems questionable.
Even congestion control can be optional for some applications with the right error correcting code. (Though if you have a narrow bottleneck somewhere in your connection, I guess it doesn't make too much sense to produce lots and lots more packages that will just be discarded at the bottleneck.)
One thing not mentioned often is that a lot of networks will drop UDP packets first when encountering congestion. The thinking is that those packets will not re-transmit, so it’s an effective means to shed excess traffic. Given we now have protocols that aggressively re-transmit on UDP, I wonder how that has changed things. I do seem to remember QUIC having re-transmit issues (vs HTTP1/2) years ago because of this.
I've attempted to replace the clickbait title* using terms from the article itself, but if someone can suggest a more representative phrase from the article, we can change it again.
(* in keeping with the HN's title guideline: "Please use the original title, unless it is misleading or linkbait" - https://news.ycombinator.com/newsguidelines.html)
>> The common wisdom is:
>> >> use TCP if you want reliable delivery
>> >> use UDP if you want unreliable delivery
>> What the *(& does that mean? Who wants unreliability?
I dont agree with the premise of this article, UDP isnt for unreliability, it provides a tradeoff which trades speed and efficiency and provides best-efforts instead of guarantees.
It makes sense depending on your application. For example, if I have a real-time multi-player video game, and things fall behind, the items which fell behind no longer matter because the state of the game changed. Same thing for a high-speed trading application -- I only care about the most recent market data in some circumstances, not what happened 100ms ago.
What should be based on datagrams:
- Local discovery (DHCP, slaac, UPnP, mDNS, tinc, bittorrent)
- Broadcasts (Local network streaming)
- Package encapsulation (wireguard, IPSec, OpenVPN, vlan)
Silly clickbait title, which the author even admits up front.
UDP and TCP have different behaviour and different tradeoffs, you have to understand them before choosing one for your use case. That's basically it. No need for "Never do X" gatekeeping.
Nahhh. While most of the applications/cases will be using session-based connections, there are uses for using datagrams directly — don’t be afraid. Yes, you will have to take care of many more details yourself. And as a side line, it is a great way of learning the low level aspects of networking.
Quic is implemented over UDP. It's literally running over datagrams.
> The bytes within each stream are ordered, reliable, and can be any size; it’s nice and convenient. Each stream could be a video frame […] But you can tell the QUIC stack to focus on delivering important streams first. The low priority streams will be starved, and can be closed to avoid wasting bandwidth.
Is the author saying that with QUIC I can send a "score update" for my game (periodic update) on a short-lived stream, and prevent retransmissions? I'll send an updated "score update" in a few seconds, so if the first one got lost, then I don't want it to waste bandwidth retransmitting. Especially I don't want it retransmitted after I've sent a newer update.
It mentions that the video game industry uses UDP but then fails to further address that use case. So, should competitive shooter video games switch to QUIC? Is that even supported across all the various gaming platforms?
I wonder if this guy thinks video GOPs should all be hundreds of frames long because P frames are so much smaller than I frames and since in his world we NEVER use an unreliable network you might as well.
I wonder if the DiffServ[0] bits in IPv6 could be another way to prevent bufferbloat from affecting real-time datagrams? Or are they like IPv4's ToS[1] bits, which I think were never implemented widely (or properly) enough for any software to bother with?
Great in depth article from what seems really a person who knows that stuff.
This is a narrow way of looking at UDP applications. The whole HFT, low-latency fintech world is built on top of datagrams. Using TCP would be pure the worst choice possible.
Imagine Ethernet was designed like this and you had to implement mandatory congestion control and other cruft. The layer of the stack that has knowledge of the content should be implementing the congestion control.
Maybe ... never use on a congested network.
or Never use on a network where congestion is above a certain level
or Never us on a network where this parameter is above a certain level - like network latency
or only use on a LAN not a WAN ....?
I've seen UDP used for great effect in video streaming. Especially timely video streaming such as cloud gaming. When waiting a late packet is no longer useful.
Hope this one gets captured by quackernews - can’t wait to see its description.
The choice is not UDP vs TCP.
UDP adds minimum over raw sockets so that you don't need root privileges. Other protocols are build on top of UDP.
It's better to use existing not-TCP protocols instead of UDP when the need arises instead of making your own. Especially for streaming.
Used high numbers of UDP packets over intercontinental internet links for mission critical applications for 15 years. 20mbit of UDP carrying RTP. Loss on a given flow is quite rare, and the application helps (via duplication, or retransmits)
As time has progressed increased from nothing to fec to dual-streaming and offset-streaming to RIST and SRT depending on the criticality.
On the other hand I've seen people try to use TCP (with rtmp) and fail miserably. Never* use TCP.
Or you know, use the right tool for the right job.
Network is nothing but datagrams.
I feel like this article misses why people avoid TCP like the plague, and why people use UDP for many applications
1. Routers do all kinds of terrible things with TCP, causing high latency, and poor performance. Routers do not do this to nearly the same extent with UDP
2. Operating systems have a tendency to buffer for high lengths of time, resulting in very poor performance due to high latency. TCP is often seriously unusable for deployment on a random clients default setup. Getting caught out by Nagle is a classic mistake, its one of the first things to look for in a project suffering from tcp issues
3. TCP is stream based, which I don't think has ever been what I want. You have to reimplement your own protocol on top of TCP anyway to introduce message frames
4. The model of network failures that TCP works well for is a bit naive, network failures tend to cluster together making the reliability guarantees not that useful a lot of the time. Failures don't tend to be statistically independent, and your connection will drop requiring you to start again anyway
5. TCP's backoff model on packet failures is both incredibly aggressive, and mismatched for a flaky physical layer. Even a tiny % of packet loss can make your performance unusable, to the point where the concept of using TCP is completely unworkable
Its also worth noting that people use "unreliable" to mean UDP for its promptness guarantees, because reliable = TCP, and unreliable = UDP
QUIC and TCP actively don't meet the needs of certain applications - its worth examining a use case that's kind of glossed over in the article: Videogames
I think this article misses the point strongly here by ignoring this kind of use case, because in many domains you have a performance and fault model that are simply not well matched by a protocol like TCP or QUIC. None of the features on the protocol list are things that you especially need or even can implement for videogames (you really want to encrypt player positions?). In a game, your update rate might be 1KB/s - absolutely tiny. If more than N packets get dropped - under TCP or UDP (or quic) - because games are a hard realtime system you're screwed, and there's nothing you can do about it no matter what protocol you're using. If you use QUIC, the server will attempt to send the packet again which.... is completely pointless, and now you're stuck waiting for potentially a whole queue of packets to send if your network hiccups for a second, with presumably whatever congestion control QUIC implements, so your game lags even more once your network recovers. Ick! Should we have a separate queue for every packet?
Videogame networking protocols are built to tolerate the loss of a certain number of packets within a certain timeframe (eg 1 every 200ms), and this system has to be extremely tightly integrated into the game architecture to maintain your hard realtime guarantees. Adding quic is just overhead, because the reliability that QUIC provides, and the reliability that games need, are not the same kind of reliability
Congestion in a videogame with low bandwidths is extremely unlikely. The issue is that network protocols have no way to know if a dropped packet is because of congestion, or because of a flaky underlying connection. Videogames assume a priori that you do not have congestion (otherwise your game is unplayable), so all recoverable networking failures are 1 off transient network failures of less than a handful of packets by definition. When you drop a packet in a videogame, the server may increase its update rate to catch you up via time dilation, rather than in a protocol like TCP/QUIC which will reduce its update rate. A well designed game built on UDP tolerates a slightly flakey connection. If you use TCP or QUIC, you'll run into problems. QUIC isn't terrible, but its not good for this kind of application, and we shouldn't pretend its fine
For more information about a good game networking system, see this video: https://www.youtube.com/watch?v=odSBJ49rzDo, and it goes over pretty in detail why you shouldn't use something like QUIC
IMO one of the worst mistakes made in IP development was not having made a standard protocol for the one thing that pretty much everyone seems to want:
A reliable, unlimited-length, message-based protocol.
With TCP there's a million users that throw out the stream aspect and implement messages on top of it. And with UDP people implement reliability and the ability to transmit >1 MTU.
So much time wasted reinventing the wheel.
TL;DR - use QUIC (should have just looked at the domain name)
I use UDP for my desktop game like software with clients all over the world to propagate anonymized state. It needs no encryption as the transmitted data has zero value to any third party. It needs no reliability since dropped packets would be handled by predictive filter with practical reliability that is way more than needed.
So why the F.. would I bother with anything else?