back
126 comments
Impressive that he managed to say so much without mentioning head-of-line blocking, which is the main TCP problem with realtime apps.

This is not due to congestion control, but because of the ordered byte stream semantics of TCP. You can't deliver a packet until the retransmission dance is done for any previous lost packets.

As an aside, these days it's worth noting that a 3G, WLAN or 4G link layer will try very hard not to drop any packets. So when you're doing UDP, watch out for those 40000 millisecond old packets that will burst to your lap once the rain cloud moves away from the cell tower or your co-worker's cocoa blings in the microwave.

SCTP is sometimes promoted as an alternative to solve that problem. You can have multiple stream per connection so if one blocks the other ones can keep going.
I was under the impression that packets behind the blocked TCP packet still sent through the network, relying on the client-side or last-hop to maintain the sequential stream illusion and thus reducing the delay of the subsequent frames to the delay blocked/resent frame. Could with a deeper knowledge confirm or deny this?
He didn't mention the term head-of-line blocking, but the issue was apparent to me as I read it. Perhaps because I have a bit of previous experience with tcp vs udp, but still.
We use TCP for some things, UDP for some things, and our own variant on Reliable UDP for others.

We use TCP for sending real-time vessel track data to the central correlation servers, because they need to know if the connection has dropped and change their behavior accordingly. Traffic volume is low, so congestion throttling and re-tries are not a problem. The system is able to cope with reports being delayed and then coming in much later (and catching up) because reports are time-stamped and the correlator works in fuzzy 6D phase-space.

We use UDP for sending live radar data because, frankly, if you've missed one sector of data you really, really don't want to delay subsequent sectors while waiting for the data that will be irrelevant in 2 seconds anyway. The compression and encryption schemes are specially crafted to allow for lost packets while still using solid implementations of off-the-shelf algorithms where appropriate, and lovingly hand-crafted algorithms for the non-security critical aspects.

And I can't tell you what we use our locally defined reliable UDP for.

Usually that's for state messages. The kind of message that defines the meaning of subsequent messages. Like codec parameters, or framing data or something.
The article emphasizes that TCP congestion control can make TCP underperform with respect to the actual available bandwidth. So True and So Frustrating!

But it doesn't seem to talk about the virtues of congestion control - the primary one being it prevents the collapse of the Internet. Reacting to packet loss by just trying harder results in nothing but a network filled with doomed packets. This is not theoretical - before Van Jacobson that was the Internet.

I'm not defending the status quo nor TCP - TCP has deep problems figuring out what is really loss. and it speeds up too slowly but still doesn't manage to slow down when it should resulting in induced delay. But the mere fact that is worried about congestion (not just unreliability - one result of congestion) and sharing the channel isn't its problem.

have a look at QUIC, ledbat, Minion and the recent IETF TAPS BoF for work going on improving transport options in these areas. Its just the wrong takeaway to read that article and say "UDP is better because it doesn't have congestion control" - you need to consider CC in any reliable UDP based transport you roll too.

rfc 5405 provides some advice (http://tools.ietf.org/html/rfc5405) - its a little dated but still useful.

Excellent point, awesome RFC! It's worth reproducing this point from the abstract, which makes the same point you do:

> Because congestion control is critical to the stable operation of the Internet, applications and upper-layer protocols that choose to use UDP as an Internet transport must employ mechanisms to prevent congestion collapse and to establish some degree of fairness with concurrent traffic.

It's a shame that the internet requires this kind of good behaviour from individual hosts, because it leaves it vulnerable to buggy or malicious nodes. But as long as it does, it's essential that users of UDP know about this.

> TCP has deep problems figuring out what is really loss. and it speeds up too slowly but still doesn't manage to slow down when it should resulting in induced delay

TCP is really pretty reasonable. Fast retransmit/fast recovery lets TCP eat up small packet losses without entering congestion control. Plus everybody recently upped their intial congestion window to 10.

Also, there was an attempt to deploy ECN (explicit congestion notification) for TCP, but net equipment vendors and providers resisted deploying it. It was specced as an official standards track TCP feature in 2001.

If you absolutely positively need to have that lowest lag time between points then UDP is the way to go. However 1) UDP will drop the occasional packet. 2) UDP will NOT work behind many corporate firewalls. 3) UDP connections between users on separate networks will fail due to random configuration issues (I am sure there is a reason, just don't know what it is after years of looking). 4) UDP connections between users on the same DHCP servers requires a different addresses then the global address to make connections. 5) UDP connections can time out if you don't send anything for a few minutes. Easy to fix, hard to find.

In the virtual world platform my team wrote, we started using UDP exclusively, then moved to mixed TCP and UDP, then moved to using TCP for everything except P2P voice connections. with UDP I had always dreaded going into see a potential corporate client, and not knowing if my networking was going to make it through the firewall.

This is a good discussion of the subject. http://stackoverflow.com/questions/992069/ace-vs-boost-vs-po...

There are no such thing as "UDP connections". UDP is datagram based : you send a packet to an address, either it reaches its destination either it not (regardless of the order). A packet can be dropped in case of network congestion, TTL that reaches 0, etc.

That is : One can easily rewrite a TCP-like on top of UDP.

All true, if you read 'connection' to mean 'path'. Yes firewalls are an issue. Often HTTP tunneling is tried because 'HTTP will always work' but in practice HTTP has more pitfalls than UDP. There are 100 firewall settings about HTTP; you have to get them all right to stream HTTP thru a firewall (and proxies and ISP servers in the same path). Whereas UDP usually has exactly one setting: enable UDP.

We've been poking UDP holes in firewalls for 5 years; its the best (most frequently successful) option for enterprise by far.

Failures to communicate on the same subnet with UDP are the wellknown 'hairpin' issue, where some routers will not recognize their own outside IP address and short-circuit a packet back to another subnet inside their domain. Not a lot you can do about it, except use TURN or another UDP proxy.

Years ago, i read about some application protocol (for a game, i think) that sat on top of UDP. It packed small messages into datagrams for transmission, and handled acking and redelivery.

One of the cute things it did was improve reliability for critical messages by preemptively sending them twice, in successive datagrams. That allowed the protocol to avoid an ack-miss-retransmit cycle for those packets, as long as it was only a single datagram that was dropped.

It's the Quake 3 network protocol. Carmack invented it, and that protocol was the first one to solve the UDP ack problem in a way that didn't impact gameplay. It was a huge deal and was probably one of the reasons the Quake 3 engine generated over a billion dollars in revenue.
That's a variation of FEC and its good stuff http://en.wikipedia.org/wiki/Forward_error_correction

we're seeing it come back into vogue in TCP and modern protocols. QUIC has a FEC component and TCP Tail Loss Probe (http://www.ietf.org/proceedings/84/slides/slides-84-tcpm-14....) is a subtle variation on the theme too

the million dollar question around FEC is how correlated are different loss events which has a lot to do with how well it can work.

X-Wing vs Tie Fighter, a classic read on networking over the internet: http://www.gamasutra.com/view/feature/131781/the_internet_su...
gsfgf says:

"Hi, I'd like to hear a TCP joke."

"Hello, would you like to hear a TCP joke?"

"Yes, I'd like to hear a TCP joke."

"OK, I'll tell you a TCP joke."

"Ok, I will hear a TCP joke."

"Are you ready to hear a TCP joke?"

"Yes, I am ready to hear a TCP joke."

"Ok, I am about to send the TCP joke. It will last 10 seconds, it has two characters, it does not have a setting, it ends with a punchline."

"Ok, I am ready to get your TCP joke that will last 10 seconds, has two characters, does not have an explicit setting, and ends with a punchline."

"I'm sorry, your connection has timed out. Hello, would you like to hear a TCP joke?"

I'd reply with a UDP joke, but you might not get it
I used to use UDP. Its nice for some things. Things I liked: When you read a UDP message you got the whole message or nothing (its not a stream). Fast. Multicast!

Things that challenged us: 64K message size limit (varied slightly from linux/solaris/hpux) you can't tell if the other side is listening or got the message. multicast requires use of certain ip ranges which people seem to forget frequently.

I had the UNIX Network Programming by W. Richard Stevens tome I borrowed from my boss. Good for networking.

Yeah you have to recapitulate some TCP semantics to use UDP effectively. Packet numbering (so you can catch out-of-order packets). Dice-and-reassemble. Congestion control.

But the good news is, you CAN do those things yourself. And avoid the TCP lockups, slowdowns and endless retransmissions.

the wrs book wears very well. he was a good man.

as an aside, its generally a mistake to use IP-Size > PMTU (commonly ~1500).. the result is IP fragmentation and IP stacks commonly have very limited resources devoted to IP reassembly (otherwise its a DoS attack) - so its very easy for your fragmented UDP message to get "lost" even when the network transport does not have an error.

When it seems to work ok its basically because your app is the only one doing it.. if it were common practice you would all fight for the same reassembly buffer space and the OS would have to start dropping things. Your application also becomes trivially dosable at low bandwidths by an attacker intentionally using up the reassembly buffer.

Biggest problem with UDP: No support at all in modern browsers without plugin. So when doing web games, you can't really get around using Websockets which are TCP based.
Or, the biggest problem with web games: no UDP.
This will no longer be the case once you can use WebRTC data channels. Its just a matter of making sure your server can support the protocol.
The reality is that with broadband connections these days there is generally little or no packet loss unless your ISP is having problems. In the case where you are getting packet loss it usually tends to be around 50%, so UDP is going to have trouble as well.

Also, these days with fast retransmit TCP isn't as bad as it used to be when there are network problems.

I've been using TCP for over 20 years for multi-player games, VOIP and web conferencing, and it works incredibly well. Even if you do all the legwork to get UDP working reliably (which TCP give you for free), you still have the problem of firewalls.

A very good recap on how networking in Quake3 works without reliable packet delivery:

http://trac.bookofhook.com/bookofhook/trac.cgi/wiki/Quake3Ne...

Simple, yet robust.

I've been hoping someone makes a decent server-side WebRTC peer, meaning that we can use UDP in the browser and really push HTML5 gaming forward. It seems to go completely under the radar with all this talk of audio/video P2P streaming.
from the_skys_kid:

I'll defend UDP.

UDP is the honey badger of the internet protocol suite.

UDP is all about the transaction. UDP is standing on a cliff yelling, "Come at me, bro", whether you're there or not.

UDP is a man's protocol doing real shit like bootstrapping your ass and slapping an IP on you. Get up, motha fucka!

UDP will talk shit to one of you or all of you. UDP ain't scared. UDP brought the fear.

UDP understands that you may be slow sometimes. So UDP will wait for your sorry ass. UDP grew up without a father, too.

UDP sends a message and couldn't give a fuck if you got it or not.

UDP got a message from you saying that you got his messages and guess what? UDP didn't even open it! Not one fuck given.

Don't try to shake UDP's hand! You crazy?

And, when UDP dies because you weren't available, UDP doesn't shed a tear. UDP is hardcore. He's going out even if he knows you ain't there. UDP is a goddam one-man slaughter house. Why?

Because UDP doesn't give a fuck.

In re the sections on "hiding the lag" - Bungie (the guys behind the original Halo games) have a pretty cool presentation on how they hid lag in Halo Reach.

http://downloads.bungie.net/presentations/David_Aldridge_Pro...

Just use RakNet http://www.jenkinssoftware.com

I mean if you want to ship a product. If your goal is to learn networking, then by all means, roll your own.

One side note - what the protocol is on the wire and how you're using it are not quite the same thing. With regard to head-of-line, in principle you can look at later TCP packets before a dropped packet comes in. Of course, correcting framing may be expensive or impossible, and (more importantly, for most applications) if the OS is handling TCP for you you'd have to find some way to cajole it into passing those packets to you.
I have used UDP for two major projects: a game engine while working at Angel Studios (networking was tightly coupled with camera and vehicle AI) and for the NMRD project in the late 1980s (NMRD could detect nuclear tests based on seismic waves).

I am surprised that UDP is not used more often when the business cost of loss data is small.

Some RTC implementations (VoIP and video) are using both in order to get through firewalls - try UDP first, resort to TCP (including even HTTP over TCP) if UDP can't connect.

The TCP fallback only consistently works well (in terms of QoE) in cases in which the TCP legs are short.

One thing I've always wondered - has anyone written something that works over UDP but offers fairly basic TCP-like reliability functionality? So you'd have the speed advantage of UDP, however you could test the validity of your data, for instance.
Someone should write a series of articles about this issue from the POV of Javascript/browser clients. There, UDP isn't available at all, and we also have to deal with quite a bit of pain in terms of differences between platform implementations.
Lately, you can just use things like http://www.pubnub.com/ now a days and do both and not have to worry.

Socket servers are so 80's.

TCP can get you quite far, even in a fast paced multiplayer game, as long as the game is suited.

For example, the multiplayer version of asteroids in this article uses TCP sockets:

http://www.wildbunny.co.uk/blog/2012/11/20/how-to-make-a-mul...

Cheers, Paul.

The problem with using UDP is that many ISPs block UDP packets because it doesn't have features like congestion control. So using it for an internet game is out of the question, because there's a chance that a bunch of your users won't be able to play online.
I've heard before that TCP is a really awful fit for mobile. So shouldn't there be a general replacement for TCP on mobile? Everybody inventing their own custom fix does not sound like the most efficient way to solve this problem.
Back when i did this kind of stuff i used both of them udp for nonesential stuff like player position.

Where a lost packet did not matter, there would still be a new player position packet a half a second later.

Then for chat and score and stuff like that tcp was used.

enet is a networking library which uses UDP but generate redundancy and other things so it has the advantages of udp with some advantages of TCP.
I don't get the bottom line part: how can you send something via "HTTP"? What method of transport layer is being used?
You might as well paint "DDoS the hell out of me" on your forehead if you rely on UDP. Use TCP and block all UDP at your edge. Backhaul your own UDP traffic through GRE tunnels to offsite VPS's.

If you have cool transit providers, you can have them place upstream filters to block UDP. That might be a pipedream though, I don't know of any large providers who will do that for you these days.

Could anyone point me to books and ressources for beginners on the topic of multiplayer game development?
Be weary of buffer overruns when using a language like C :).
This article is boss. TCP vs. UDP holywar is not going to end any time soon, it's nice that some people can actually explain the differences in simple manner.